x-lang

← Index

x/tool/syntax

Syntax: lexical grammars and the renderers that draw their spans.

One vocabulary of classes, one span shape, three renderers: a grammar never writes markup, and a renderer never reads text.

Theme says how a terminal draws each class; the user’s theme file chooses one.

x/repl/paint colours the REPL line and x/tool/highlight the documentation from (Syntax x).

Class Theme

How a terminal draws each class: a Style (x/tui/style) a class, a class the theme leaves out drawn plain. The theme in use, (Theme current), is what Syntax ansi, sgr and code draw with.

The user’s theme file is $X_THEME, else $XDG_CONFIG_HOME/x/theme.x, else ~/.config/x/theme.x, and an empty X_THEME reads none. It is data, read and never evaluated: each (theme NAME ROW …) is a theme, its rows over the theme of that name already known, else over the default, and (use NAME) the theme to use. A row is (CLASS ATTRIBUTES FOREGROUND BACKGROUND) as Style make takes them, so (theme default (comment (italic) 8 ())) changes only comments.

The file is read on the first use of a theme, and again after forget!; one that does not read is one line on stderr and the default theme.

name

The theme’s name, a symbol

Field: data carried by a Theme instance.

styles

Each (CLASS . STYLE): STYLE a Style integer

Field: data carried by a Theme instance.

codes

Each (CLASS . SEQUENCE): the SGR sequence that draws the class, as sgr answers it

Field: data carried by a Theme instance.

(style cls)

The Style the theme draws a class with, 0 (plain) for one it leaves out.

Instance method: called on a Theme instance.

Parameters:

Returns: INTEGER — A Style

Examples:

((Theme default) style 'punct) => 0

(sgr cls)

The SGR sequence that draws a class over the plain style, empty for a class drawn plain. A renderer resets after each span, so the sequence does not reset first.

Instance method: called on a Theme instance.

Parameters:

Returns: STRING — The escape sequence

Examples:

((Theme default) sgr 'keyword) => "\x1b[1;35m"

(with rows . name)

A theme with the rows over this one’s: a class in rows is drawn as they say, every other as before.

Instance method: called on a Theme instance.

Parameters:

Returns: Theme — The theme

Examples:

(((Theme default) with '((number (bold) 3 ()))) sgr 'number) => "\x1b[1;33m"

(Theme make name rows)

A theme drawing each class in rows as it says, and every other plain. Register it to have named and the theme file find it.

Parameters:

Returns: Theme — The theme

Examples:

((Theme make 'mono '((keyword (bold) () ()))) sgr 'string) => ""

(Theme default)

The default theme: comments and privates dim; strings green, numbers yellow, names blue, characters and escapes magenta; keywords, constants and classes bold in magenta, red and cyan; errors bold red; punctuation and operators plain.

Returns: Theme — The theme

Examples:

((Theme default) sgr 'string) => "\x1b[32m"

(Theme register! t)

Add a theme to the ones named finds; one of the same name is replaced. A theme file’s themes are found before registered ones.

Parameters:

Returns: Theme — The theme

Examples:

((Theme register! (Theme make 'mono ())) name) => 'mono

(Theme named name)

The theme of that name: the user’s theme file’s, else a registered one, else the default for default; nil for none.

Parameters:

Returns: Theme — The theme, or nil

Examples:

(eq? (Theme named 'default) (Theme default)) => #t

(Theme names)

The names named answers a theme for, the default’s last.

Returns: LIST — Theme names

Examples:

(List last (Theme names)) => 'default

(Theme current)

The theme in use: the one use! set, else the one the user’s theme file names with (use NAME), else the default.

Returns: Theme — The theme

(Theme use! t)

Draw with a theme from now on.

Parameters:

Returns: Theme — The theme

Examples:

((Theme use! (Theme default)) name) => 'default

(Theme file)

The user’s theme file: $X_THEME, else $XDG_CONFIG_HOME/x/theme.x, else ~/.config/x/theme.x; nil for an empty X_THEME or no home. The file need not exist.

Returns: STRING — The path, or nil

(Theme load! path)

Read a theme file: its themes join the ones named finds, ahead of any of the same name, and its (use NAME) becomes the theme in use. Nothing changes when the file does not read.

Parameters:

Returns: LIST — The file’s themes

(Theme forget!)

Drop the theme in use and the theme file’s themes, read again on next use; registered themes stay.

Returns: NIL — Nothing

Examples:

(null? (Theme forget!)) => #t

Class Syntax

A lexical grammar for highlighting: a Lexer whose rules read every byte, and a table from each token’s tag to its class. spans answers (offset length class) for a stretch of text; ansi, html and cells render spans for a terminal, a web page and a screen editor. (Syntax x) is x-lang’s own grammar.

The classes are one vocabulary, (Syntax classes), so a renderer knows every grammar’s output: comment string char number keyword constant class private name punct operator escape error.

A grammar’s lexer is made on its first read, not when the grammar is: making a Lexer compiles its states, which a process that never highlights should not pay for.

name

The grammar’s name, a symbol

Field: data carried by a Syntax instance.

rules

The Lexer rules, each token’s tag looked up in classes

Field: data carried by a Syntax instance.

end

The Lexer’s end text: see Lexer’s end field

Field: data carried by a Syntax instance.

classes

Each (TAG . CLASS): CLASS a class symbol, nil to leave the token plain, or a function of the token’s text answering one, or answering the token’s pieces, each (offset length class) inside the text, in order, a piece of class nil left plain

Field: data carried by a Syntax instance.

sync

A test (fn (_ text offset)) of a line start, true where the lexer can start outside any string or comment; nil to read from the text’s start

Field: data carried by a Syntax instance.

lines

How many lines spans walks back looking for a line the sync test passes

Field: data carried by a Syntax instance.

files

The file name endings the grammar is for, as “.x”

Field: data carried by a Syntax instance.

names

The whole file names the grammar is for, as “Makefile”

Field: data carried by a Syntax instance.

interpreters

The programs a #! line names for the grammar’s files, as “sh”; a version after the name is ignored, so “python” is for python3.12

Field: data carried by a Syntax instance.

lexer

The Lexer, made from the rules on the first read

Field: data carried by a Syntax instance.

(spans text . range)

The spans of text from start for length bytes, each (offset length class) in order, offsets into text and clipped to the range. The lexer reads from a line the sync test passes, so a string or comment opened before start colours the range’s start, and to the end of the line the range ends in, never past the end given. Tokens with no class are left out.

Instance method: called on a Syntax instance.

Parameters:

Returns: LIST — ((offset length class) …)

Examples:

((Syntax x) spans "\"a b\" c" 3 4) => ((3 2 'string) (6 1 'name))

(class-of tag text)

The class the grammar gives a token, or nil for one it leaves plain; a class function may answer the token’s pieces instead, each (offset length class) inside the text.

Instance method: called on a Syntax instance.

Parameters:

Returns: SYMBOL — A class, or the pieces

Examples:

((Syntax x) class-of 'atom "Str8") => 'class

(%lexer)

Instance method: called on a Syntax instance.

(Syntax make name rules classes)

A grammar. Set sync, lines and files on it after, and register! it to have for-file find it.

Parameters:

Returns: Syntax — The grammar

(Syntax over l classes)

A grammar over a Lexer already made, reading from the text’s start.

Parameters:

Returns: Syntax — The grammar

Examples:

((Syntax over (Lexer make (list (Lexer run 'w "ab" "ab")) " ") (list (pair 'w 'name))) spans "ab") => ((0 2 'name))

(Syntax x)

x-lang’s grammar: comments, strings, character literals and parens by their syntax, and every other atom by what the reader reads it as – a keyword when lib/x/constructs.x lists it, a number, string, character or constant by its value, and otherwise a name, a private (a leading %) or a class (a leading capital).

Returns: Syntax — The grammar

Examples:

((Syntax x) class-of 'atom "3") => 'number

(Syntax register! g)

Add a grammar to the ones named and for-file finds; one of the same name is replaced. A grammar module a bundle’s lang.xon names registers its grammar this way when it loads.

Parameters:

Returns: Syntax — The grammar

Examples:

(eq? (Syntax register! (Syntax x)) (Syntax named 'x)) => #t

(Syntax named name)

The grammar of that name: a registered one, else one a bundle’s lang.xon declares, its module loaded now; nil for none.

Parameters:

Returns: Syntax — The grammar, or nil

Examples:

(eq? (Syntax named 'x) (Syntax x)) => #t

(Syntax for-file path . text)

The grammar for a file: with its text, the one a modeline in the first or last five lines names (vi: set ft=NAME, or filetype, syn or syntax), else the one whose interpreters hold the program the #! line runs; else the one whose files the name ends with or whose names hold its base name. Registered grammars are asked before bundles’ and loaded as named loads them; nil for none.

Parameters:

Returns: Syntax — The grammar, or nil

Examples:

((Syntax for-file "lib/x/tool/syntax.x") name) => 'x
((Syntax for-file "build" "#!/usr/bin/env x\n") name) => 'x

(Syntax catalogue)

Every grammar named and for-file can find, as (NAME MODULE ROOT FILES NAMES INTERPRETERS): the registered ones with no module or root, then each bundle’s (grammar …) forms in its lang.xon, read once.

Returns: LIST — The rows

Examples:

(first (Syntax catalogue)) => ('x () () (".x" ".xon") () ("x"))

(Syntax classes)

Every class a grammar’s table may answer, in index order.

Returns: LIST — Class symbols

Examples:

(Syntax classes) => ('comment 'string 'char 'number 'keyword 'constant 'class 'private 'name 'punct 'operator 'escape 'error)

(Syntax index cls)

The class’s index, from 1 in classes order; 0 for nil or a symbol not among them.

Parameters:

Returns: INTEGER — The index

Examples:

(Syntax index 'string) => 2

(Syntax rouge cls)

The Rouge token class a class is written with in HTML, nil for none.

Parameters:

Returns: STRING — The token class

Examples:

(Syntax rouge 'keyword) => "k"

(Syntax sgr cls)

The SGR sequence a class is drawn with in the theme in use, whether or not standard output takes colour: for a screen editor that writes to its own terminal. Empty for a class drawn plain.

Parameters:

Returns: STRING — The escape sequence

(Syntax code cls)

The SGR sequence a class is drawn with in the theme in use when standard output takes colour (Ansi enabled?), else empty.

Parameters:

Returns: STRING — The escape sequence, or empty

(Syntax ansi text spans start len . code-of)

The bytes of text from start, each span’s wrapped in its class’s sequence and a reset, the rest plain. The sequences are code’s, the theme in use’s and empty when colour is off, unless code-of answers one.

Parameters:

Returns: STRING — The text with escape sequences inserted

(Syntax html text spans start len)

The bytes of text from start as HTML: each span’s in a span element whose class is its Rouge token class, so a Rouge or Pygments stylesheet styles it; every byte escaped.

Parameters:

Returns: STRING — The markup

Examples:

(let ((s "(f 1)")) (Syntax html s ((Syntax x) spans s) 0 5)) => "<span class=\"p\">(</span><span class=\"n\">f</span> <span class=\"m\">1</span><span class=\"p\">)</span>"

(Syntax cells spans start len)

A byte a text byte from start: 64 plus the index of the byte’s class, a space for a byte of none. A screen editor that lays out its own rows – tabs widened, control bytes shown as ^X – carries a row of these beside its text and draws each cell in its class’s sgr.

Parameters:

Returns: STRING — len bytes

Examples:

(let ((s "(f 1)")) (Syntax cells ((Syntax x) spans s) 0 5)) => "JI DJ"

(Syntax cell-class b)

The class a cell byte stands for, nil for a space.

Parameters:

Returns: SYMBOL — The class, or nil

Examples:

(Syntax cell-class 66) => 'string

(Syntax classify text)

The class x-lang’s grammar gives an atom: the reader’s answer for its bytes, a construct a keyword. Memoised per distinct atom; forget! drops the memo.

Parameters:

Returns: SYMBOL — The class

Examples:

(List map (fn (_ s) (Syntax classify s)) (list "def" "-" "#t" "%x")) => ('keyword 'name 'constant 'private)

(Syntax keywords)

x-lang’s constructs, as a Dict used as a set.

Returns: Dict — Construct name -> #t

Examples:

((Syntax keywords) get "def") => #t

(Syntax forget!)

Drop the construct set, the memo of atom classes and the bundles’ catalogue rows, read again on next use: a lang that registers literal syntax changes what the reader answers, and an image runs where other bundles are installed.

Returns: NIL — Nothing

Examples:

(null? (Syntax forget!)) => #t