x-token.c File Reference

Tokenizer: analysis, reading, writing, and display of expressions. More...

#include "x-eval.h"
#include "x-heap.h"
#include "x-obj.h"
#include "x-token.h"
#include "x-type.h"
#include "x-type/buffer.h"
#include "x-type/char.h"
#include "x-type/int.h"
#include "x-type/iter.h"
#include "x-type/prim.h"
#include "x-type/str.h"
#include "x-type/list.h"
#include "x-token/sexp/atom.h"
#include "x-token/sexp/pair.h"

Macros
#define	prim_arg_prim   x_0((x_obj_t *)prim_args)

Functions
x_obj_t *	x_token_delimit (x_obj_t *p_base, x_obj_t *p_args)
x_obj_t *	x_token_analyse (x_obj_t *p_base, x_obj_t *p_args, x_int_t *p_variant)
x_obj_t *	x_token_read (x_obj_t *p_base, x_obj_t *p_args)

Variables
x_satom_t	x_type_list_iter_prim
x_satom_t	x_token_eof_prim = x_obj_set(x_type_atom_obj, X_OBJ_FLAG_NONE, { .v = (void *)x_token_eof_prim })

Detailed Description

Tokenizer: analysis, reading, writing, and display of expressions.

Author

Jon Ruttan (jonru.nosp@m.ttan.nosp@m.@gmai.nosp@m.l.co.nosp@m.m)

Copyright

2021 Jon Ruttan @license MIT No Attribution (MIT-0)

Macro Definition Documentation

prim_arg_prim

#define prim_arg_prim   x_0((x_obj_t *)prim_args)

Function Documentation

x_token_analyse()

x_obj_t * x_token_analyse	(	x_obj_t *	p_base,
		x_obj_t *	p_args,
		x_int_t *	p_variant
	)

Determine which type best matches the next token in the buffer.

Iterates all registered types, calling each type's analyse hook character by character. Tracks the winning type by score (number of buffer characters consumed). Uses first-char hint strings on x-lang closures to skip non-matching types early. The >= comparison means later types (C built-ins) win ties against earlier (custom) types.

After finding the winner, advances the buffer read pointer by the winning score.

Parameters

p_base	x_obj_t* – Base (execution context)
p_args	x_obj_t* – (buffer . base) pair
p_variant	x_int_t* – Out: the variant the winning handler declared through the variant cell (0 when none)

Returns

x_obj_t* – Winning type alist entry (name . type-struct), or NULL if no type matched

Note

Negative scores indicate inverse-priority matches (the symbol fallback scores negative, sexp/symbol.c, so any positive match wins over it). The absolute value determines advancement.

x_token_delimit()

x_obj_t * x_token_delimit	(	x_obj_t *	p_base,
		x_obj_t *	p_args
	)

Check whether any type's delimiter matches at the current buffer position.

Iterates all registered types (except the given type itself) and calls each type's delimit hook. Returns the buffer if a delimiter matched, NULL otherwise.

Parameters

p_base	x_obj_t* – Base (execution context)
p_args	x_obj_t* – Read-args whose first element is the buffer

Returns

x_obj_t* – The buffer if a delimiter matched, or NULL

x_token_read()

x_obj_t * x_token_read	(	x_obj_t *	p_base,
		x_obj_t *	p_args
	)

Read the next token from the buffer and return its parsed object.

Calls x_token_analyse to identify the token type, counts newlines in the consumed region for line tracking, then invokes the winning type's read hook. Types with no read hook (e.g. whitespace) are discarded and the loop retries. Stamps source line numbers on created objects when meta tracking is enabled.

Parameters

p_base	x_obj_t* – Base (execution context)
p_args	x_obj_t* – (buffer . base) pair

Returns

x_obj_t* – Parsed object; NULL when the token read a nil VALUE (()) or every reader declined; x_token_eof_prim at clean end of input (zero consumption).

Variable Documentation

x_token_eof_prim

x_satom_t x_token_eof_prim = x_obj_set(x_type_atom_obj, X_OBJ_FLAG_NONE, { .v = (void *)x_token_eof_prim })

Clean-EOF sentinel (see x-token.h). The value word is its own address so eq?'s value-word compare cannot conflate it with a small integer; identity tests use the ADDRESS (C ==, x-lang (obj same?)).

x_type_list_iter_prim

x_satom_t x_type_list_iter_prim

extern