x-token.h File Reference

Tokenization interface. More...

#include "x-obj.h"

Go to the source code of this file.

Macros
Reader Argument Access Macros
Decompose the argument list passed to token reader callbacks. Layout: ((prim/buffer . (score . (... (char . ()) ...)))).
#define	x_token_read_arg_prim(X)   x_0((X))
#define	x_token_read_arg_buffer(X)   x_0((X))
#define	x_token_read_arg_score(X)   x_01((X))
#define	x_token_read_arg_char(X)   x_0(x_11((X)))
#define	x_token_read_arg_variant(X)   x_01((X))

Tokenization Pipeline
x_satom_t	x_token_eof_prim
x_obj_t *	x_token_delimit (x_obj_t *p_base, x_obj_t *p_obj)
x_obj_t *	x_token_analyse (x_obj_t *p_base, x_obj_t *p_obj, x_int_t *p_variant)
x_obj_t *	x_token_read (x_obj_t *p_base, x_obj_t *p_obj)

Detailed Description

Tokenization interface.

Declares the type-dispatched tokenization pipeline: delimiting, analysing, reading, writing, and displaying. Each stage iterates the registered type alist and delegates to per-type handlers.

THE VARIANT CHANNEL. An analyser knows things about the token it accepts that the text alone does not say – which of its states ran, whether a numeric literal passed through a fraction or an exponent – and used to throw them away, leaving the reader to rescan the text. The score cell an analyser is handed now carries a VARIANT cell on its rest: the state writes an integer there as it accepts (x-lang's score-variant!, a plain set-cell-int! on the cell; compiled states call jit_score_variant), the analyse loop records the winning handler's variant, and x_token_read hands it to the type's reader as the reader's SECOND argument – a raw ATOM CELL whose value word is the integer (x_atomint in C, cell-int in x-lang; nil when no state declared one, which is every type that never heard of the channel). A cell rather than an int because an int object only exists relative to a base that registered the int type, and a tokenizer base has none by design. Readers already received (buffer ()) so nothing about their arity changes.

Author

Jon Ruttan (jonru.nosp@m.ttan.nosp@m.@gmai.nosp@m.l.co.nosp@m.m)

Copyright

2021 Jon Ruttan @license MIT No Attribution (MIT-0)

Macro Definition Documentation

x_token_read_arg_buffer

#define x_token_read_arg_buffer

(

X

)

x_0((X))

Alias – buffer object for the reader.

x_token_read_arg_char

#define x_token_read_arg_char

(

X

)

x_0(x_11((X)))

Lookahead character.

x_token_read_arg_prim

#define x_token_read_arg_prim

(

X

)

x_0((X))

Reader primitive / buffer object.

x_token_read_arg_score

#define x_token_read_arg_score

(

X

)

x_01((X))

Current best score (integer).

x_token_read_arg_variant

#define x_token_read_arg_variant

(

X

)

x_01((X))

In a READER's argument list, (buffer variant): the variant the accepting analyser declared, a raw atom cell (read it with x_atomint), or NULL when none did.

Function Documentation

x_token_analyse()

x_obj_t * x_token_analyse	(	x_obj_t *	p_base,
		x_obj_t *	p_args,
		x_int_t *	p_variant
	)

Run per-type analysis on a completed token buffer. Writes the winning handler's declared variant (0 when none) through p_variant.

Determine which type best matches the next token in the buffer.

Iterates all registered types, calling each type's analyse hook character by character. Tracks the winning type by score (number of buffer characters consumed). Uses first-char hint strings on x-lang closures to skip non-matching types early. The >= comparison means later types (C built-ins) win ties against earlier (custom) types.

After finding the winner, advances the buffer read pointer by the winning score.

Parameters

p_base	x_obj_t* – Base (execution context)
p_args	x_obj_t* – (buffer . base) pair
p_variant	x_int_t* – Out: the variant the winning handler declared through the variant cell (0 when none)

Returns

x_obj_t* – Winning type alist entry (name . type-struct), or NULL if no type matched

Note

Negative scores indicate inverse-priority matches (the symbol fallback scores negative, sexp/symbol.c, so any positive match wins over it). The absolute value determines advancement.

x_token_delimit()

x_obj_t * x_token_delimit	(	x_obj_t *	p_base,
		x_obj_t *	p_args
	)

Check whether p_obj delimits the current token for any type.

Check whether any type's delimiter matches at the current buffer position.

Iterates all registered types (except the given type itself) and calls each type's delimit hook. Returns the buffer if a delimiter matched, NULL otherwise.

Parameters

p_base	x_obj_t* – Base (execution context)
p_args	x_obj_t* – Read-args whose first element is the buffer

Returns

x_obj_t* – The buffer if a delimiter matched, or NULL

x_token_read()

x_obj_t * x_token_read	(	x_obj_t *	p_base,
		x_obj_t *	p_args
	)

Read a single token from the input stream.

Read the next token from the buffer and return its parsed object.

Calls x_token_analyse to identify the token type, counts newlines in the consumed region for line tracking, then invokes the winning type's read hook. Types with no read hook (e.g. whitespace) are discarded and the loop retries. Stamps source line numbers on created objects when meta tracking is enabled.

Parameters

p_base	x_obj_t* – Base (execution context)
p_args	x_obj_t* – (buffer . base) pair

Returns

x_obj_t* – Parsed object; NULL when the token read a nil VALUE (()) or every reader declined; x_token_eof_prim at clean end of input (zero consumption).

Variable Documentation

x_token_eof_prim

x_satom_t x_token_eof_prim

extern

Clean end-of-input sentinel, returned by x_token_read when analysis finds no token with zero consumption. Distinguishes true EOF from a read NIL VALUE (() reads as NULL by design). Compared by ADDRESS in C; bound as token-eof for x-lang, where identity is (obj same?) – never eq?, which compares value words.

Clean-EOF sentinel (see x-token.h). The value word is its own address so eq?'s value-word compare cannot conflate it with a small integer; identity tests use the ADDRESS (C ==, x-lang (obj same?)).