← Index

x/type/buf

Tokenizer buffers (Buf) and the token stream (Tok): the low-level reader surface for custom types.

ns buf/tok are de-registered; reader-hot callers fetch-and-cache the prims, never class-dispatch per token.

Class Buf

(Buf tok buffer)

The buffer’s current token as a string.

Parameters:

• buffer : ANY — A tokenizer buffer

Returns: STRING — The token text

(Buf last-char buffer)

The last character read into the buffer, as its integer code.

Parameters:

• buffer : ANY — A tokenizer buffer

Returns: INT — The character code

(Buf reset buffer)

Empty a tokenizer buffer: reset its read and write cursors to the base.

Parameters:

• buffer : ANY — A tokenizer buffer

Returns: ANY — The buffer

(Buf retain buffer)

Compact a buffer’s unread data to the front, freeing consumed space.

Parameters:

• buffer : ANY — A tokenizer buffer

Returns: ANY — The buffer

(Buf append buffer ch)

Append one character at a buffer’s write cursor.

Parameters:

• buffer : ANY — A tokenizer buffer
• ch : CHAR — Character to write

Returns: ANY — The buffer

(Buf read buffer)

Read one character into a buffer, extending from the input channel if it is exhausted.

Parameters:

• buffer : ANY — A tokenizer buffer

Returns: ANY — The buffer, or () at EOF

(Buf read-text buffer)

Like (Buf read), but a NUL character counts as end-of-input.

Parameters:

• buffer : ANY — A tokenizer buffer

Returns: ANY — The buffer, or () at EOF/NUL

Class Tok

(Tok read buffer)

Read one token from a buffer through the type system’s analysers.

Parameters:

• buffer : ANY — A tokenizer buffer

Returns: ANY — The parsed token

(Tok read-str base s)

Tokenize a string using a base’s type system.

Parameters:

• base : ANY — Base object with type alist (Base instance or raw)
• s : STRING — Source string to tokenize

Returns: LIST — List of parsed tokens