Crafting a lang

docs/lang-contract.md says what a lang bundle is — the files, the declarations, what the platform ships and what a bundle must never vendor. This document says how you actually build one, distilled from building x-python: a Python 3 surface taken from a stub that answered #<python: not implemented> to everything, to a language with containers, classes, exceptions, comprehensions, slicing, and a working REPL.

Almost every rule below was learned by getting it wrong first. Where that happened, the mistake is stated with its mechanism, because the mistakes are the transferable part: the next lang author will be tempted by the same wrong turns, and a rule without its failure mode reads as style advice and gets ignored.

1. The layout of a bundle

A lang is a bundle: lang.xon declares it, run.x is the entry, the implementation lives in one directory named for the lang, and tests/spec-runner.sh sources the platform’s runner and vendors nothing.

lang.xon                (lang "python") (dialect xe) (requires-release "…") (entry "run.x")
run.x                   imports the implementation, wires the REPL
python/tokens.x         the reader: token types on an isolated base
python/indent.x         (historical) line structure — now a name, see §4
python/types.x          the lang's values, as x types
python/runtime.x        what the lang's operators and rules MEAN
python/parse.x          grammar → emitted x forms
tests/spec-runner.sh    sources the platform runner; sets policy knobs
tests/specs/*.spec.md   the hand-written suite — the real construction record

Two rules from the contract bear repeating because they bite in practice:

• No path into the x-lang source tree. x.sh arms the bundle’s import root and answers --share-dir; a bundle that reaches for ../../../lib works in one checkout and dangles everywhere else.
• The platform ships the runner; bundles do not vendor it. Your tests/spec-runner.sh is a dozen lines of policy (which files, which knobs) that sources the shared one.

2. Build against a scoreboard, construct against specs

Before x-python had a tokenizer it had a conformance suite: the upstream MicroPython test programs, each pinned by commit and sha256, each with its expected output taken from a real CPython run on the same machine. A stub that answers everything the same way scores 0 — and 0 against a suite that runs is worth more than green against six hand-picked cases.

But learn what that scoreboard is for. Each conformance case compares a whole program’s whole stdout, so one missing feature zeroes a sixty-line program: a dict section scoring 0/19 says nothing about dicts when every case also needs str(), while, += and er.args. Whole-program comparison forbids partial credit by design. So:

• The conformance suite is a regression detector and a shopping list, never a progress meter.
• Construction happens against hand-written specs, added with each feature, each one a .spec.md case with prose saying why the behavior is what it is. x-python’s suite grew this way, and the spec prose is the best documentation the bundle has.
• When choosing what to build next, count the corpus instead of trusting the per-section scores: how many corpus files use Python’s type chooses better than the class section’s percentage.

3. Where the seams go

The five-module split is not aesthetic; each boundary is a fact about the system:

• tokens.x owns everything that must be decided while reading characters — a column can only be measured before the whitespace is gone.
• types.x owns representation: what the lang’s values are.
• runtime.x owns meaning: what the lang’s operators and rules do. The parser emits calls to named functions (%py-add, not +), and every one is a place where a lang rule can be stated. A parser that emits the host’s operators is writing a different language wearing the same clothes.
• parse.x owns grammar only. It should know nothing about representation — x-python’s parser never sees a label or a type handle.

4. The reader is the engine’s loop — use it

The single largest structural lesson. x-python first hand-wrote a recursive-descent pass over a flat token list, in interpreted x, scanning for closing brackets at 26 call sites and carrying a bracket-depth counter through its line-structure pass. None of the other bundles has such a pass, and the reason is that the engine already provides the loop.

The analyse protocol

A token type registers analyse and (optionally) read handlers on an isolated base — (Base make-tok) — via (Base make-type …). The analyse handler is called once per character and returns one of three things: another state function (keep consuming), a score (accept), or nil (reject).

• (%score-set score 1 buffer) accepts including the current character.
• (%buffer-unread buffer) first accepts excluding it — how a token that ends at a delimiter gives the delimiter back.
• (%score-label! score K) declares which variant the accepting state saw — an integer the type’s reader recovers with (%read-label args) (its second argument carries it as a raw cell; nil when no state declared one). The analyser already knows whether a literal ran through the fraction or the exponent state; this is how it says so, instead of the reader rescanning the text.
• A negative score is “matched and discarded” — whitespace, comments.
• The score’s magnitude is the match length, so contests between types are settled by longest match.
• Discarding is “matched, with no read handler.” A negative score does not suppress a type that has a reader; if you need a line-layout that vanishes (blank lines, comment-only lines), give it its own reader-less type whose analyser claims exactly those layouts.

Idioms that matter: the accept is the return value of %score-set — wrap it in a sequencing form that returns nil and you have written a reject. And state builders that close over the current character must copy it ((+ chr 0)), never capture the callback’s own binding.

Rules instead of states: the Lexer

Most bundles want the same half-dozen token shapes, and lib/x/reader/lexer.x builds the base from them. Each rule is data — a run of characters, a run to drop, a table of literals matched longest-first, a quoted literal with an escape byte, a span from one literal to another, a number with its label, a pattern of steps each a class and how many bytes of it (a printf directive, a numeric escape) — and (Lexer make rules) registers one type a rule on a (Base make-tok) child, in list order, so the first rule wins an equal-length tie:

(import x/reader/lexer)
(def c-lexer
  (Lexer make (list
    (Lexer skip " \t\n")
    (Lexer until () "/*" "*/")
    (Lexer table 'kw (list "int" "return" "if"))
    (Lexer run 'id (list (pair 97 122) (pair 65 90) 95) (list (pair 97 122) (pair 65 90) (pair 48 57) 95))
    (Lexer number 'num "uUlL")
    (Lexer quoted 'str 34 34 92)
    (Lexer table 'op (list "(" ")" "{" "}" ";" "=" "==" "+" "+=")))))
(c-lexer read-str "int x = 0x10;")
; => ((kw "int") (id "x") (op "=") (num "0x10" 3) (op ";"))

Every state is one (fn (me buffer score chr) …) form in the assembler lane’s dialect: it is compiled through compile-asm when the lane is open, and the same form is evaluated as the interpreted twin when it is not, so the token stream is the contract and the realization is reported ((l compiled)). The base and its states are process state — they are dropped before a state image is written and made again after a load — so a bundle holds the Lexer and nothing else. (l end "\n") sets the text appended before every read, which is how the last token meets a delimiter; a space unless set. Binary input has a rule of its own: (Lexer record tag shapes) reads a record whose first byte picks a shape and whose length the shape’s steps read from its own bytes – a machine instruction, a tagged field – and (l read-span s start len) reads a span of bytes, NULs included, with nothing appended, since a record ends on its own last byte. tests/x/specs/lib/lexer.spec.md is the executable reference, and a type the rules cannot express – typed instances, a read that recurses, a live stream – is still written by hand on the protocol above, each of its analyser states realized from the same kind of form through (Lexer state form fvars), so it compiles all the same.

Nesting is free: regions and blocks

(prim-ref 'tok 'read) reads the next expression from the same buffer, and a read handler may call it. So a delimited region collects its own contents by recursing through the engine’s reader:

• Brackets: an opener’s read handler loops tok read until it sees its closer token, and the bracketed run becomes ONE token with its contents nested inside. Strings are consumed by the string types before the region’s handler ever asks, so ["]"] needs no quote tracking — the problem is not solved, it never exists. Implicit line joining falls out: a newline inside a region never reaches the line-structure machinery.
• Indentation is a delimitation question too — a column opens and closes a region the way a bracket does — so blocks work the same way: the newline type measures the column (through the shared x/reader/indent stack, so tab policy has one answer across langs), and on an open recurses to collect a (tok-block …). One read returns one token and a single dedent can close several blocks, so the surplus lives in an “owed” counter that each enclosing block loop collects.

After this conversion, x-python’s line-structure pass became a name, four closing-bracket scanners were deleted rather than moved, and comma-splitting needed no depth count — an inner region is a single token.

What the reader cannot do

Precedence. Scoring answers one question — where does this token end — which is longest-match on boundaries. A bracket or an indented run fits because it is self-delimiting. Precedence is a ranking between tokens already read, and there is no place in analyse/read to say “I built this wrong, re-parent it”: analyse returns a state, a score, or nil; read returns a value. Nesting is not ranking. x-sweet’s curly reader confirms this by refusing — it folds {a + b + c} only when every operator is identical and otherwise hands $nfx$ to the program. Keep operator precedence in a recursive-descent ladder over the (now nested) token stream, as x-ash does.

Read handlers and errors

A read handler cannot raise. The C reader loop is driving, and an error unwinding out of a handler through it takes the interpreter down rather than reaching any guard. Worse, a value caught across that boundary arrives as nil — the payload does not survive the trip. So a reader-detected error (x-python’s IndentationError) is carried out as data: a flag the tokenize entry point re-raises once reading is over and x is driving again.

5. Values go on the type system, not the class system

x’s class system is for types written in x, resolved when a file loads. A lang’s values are built at run time and follow the lang’s own rules — one level lower is the level that fits. The door is the two-argument make-type (catalogue type/make), which registers on the base it is called in — the running base, where the numeric tower already lives. This is about the lang’s values. The bundle’s own tables are classes, and its files are modules (lang-contract.md, “Modules, names and classes”).

Two prims, and the difference decided a failed design:

prim	registers onto	use for
base-make-type	the base you name	token types on the isolated reader base
make-type	the base it is called in	the lang’s VALUE types

x-python first tried a child base per lang (Base make + base-make-type + Base eval) and had most of its specs green before discovering that a child base has no numeric tower — float and bigint are library types registered on whichever base loaded them. The wrong conclusion (“this design cannot work”) was written into a doc and survived a day; the capability it asked for already existed under a name in another catalogue namespace. Before concluding the engine cannot do something, grep all of src/x-prim/*.c — the namespace/method pair is often not what the C name suggests — and look for a library type that already does the thing (x/num/rational.x and x/type/vector.x were working models of everything x-python needed).

What the handlers buy, mapped from Python:

lang feature	type slot
repr / str	write / display, calling back into the lang’s repr per element
len(x)	length
x[i], and calling a class Foo()	call
iteration	iter
a + b on your types	ops, through push-op on the wrapped type (((Type wrap t) push-op (lit +) f))

Details with teeth:

• Handler arity is (fn (_ self) …) — first parameter is the TYPE. Getting it wrong reads a non-pair and segfaults rather than raising.
• Mutation needs a cell. A mutable container’s instance holds a cell whose rest is the payload, so append mutates the cell and every reference sees it. An immutable type (tuples) skips the cell — the instance IS the value.
• The iterator contract: exhaustion rides the STATE. A step answers (value . next-state) and only a nil pair ends the walk, so a nil value (Python’s None) is an ordinary element. x-python registered value-terminated steppers, nothing consumed the slot, and the bug sat invisible until specs exercised it: a registered handler nobody steps is untested code that looks done.
• Push ops only onto types you invented. A type’s ops fire when either operand carries the type, so pushing * onto the host’s string type changes what * means for every string in the process. The lang’s rules for host types stay in runtime functions behind a predicate.
• write-to-str closes the str/repr gap. (prim-ref 'io 'write-to-str) runs the writer with its sink redirected into a string, so the same type handlers that print also render — nested containers come out right with no second rendering path.
• Classes are callable values: Foo() needs no parser special case, just a call handler. A constructor entry keyed by a name no lang identifier can spell ("%ctor") lets builtin type objects convert instead of allocate. A qualname slot travels with the class because the display prefix is a fact the constructor’s caller knows.

6. The rules that bite

Every one of these cost a debugging cycle, most of them with the symptom far from the cause.

The %-globals share one flat namespace across the bundle. x-python collided four times (%py-len, %py-elems, %py-names-of, %py-block-of), and one collision presented as a syntax error from a runtime-only edit. Grep the bundle before defining a name. The platform’s check-percent-globals gate holds per-file budgets, shrink-only — when it rejects your new globals, the intended fix is fewer names, not a bigger budget.

def binds in the current environment. A call makes a child environment, so a def in a function body, or in a guard handler’s frame, is private to that activation and goes with it; (eval (list (lit def) name value) e) binds in e, and eval! evaluates in the root (engine law 10). Use let for function locals. When something evaluated at depth (a REPL loop, a guard handler) must define for the session, evaluate the def with eval!, the value under lit so def does not evaluate it a second time.

A raise skips your restore. Any save/restore around a parse or eval leaks when the body raises — x-python’s lexical-class cell leaked out of a failed parse and a later super() error reported the wrong context. Per-run state gets reset at the entry point, not restored at exits.

Nothing collects unless you ask. x has no automatic collection: every sweep in the tree is a hand-placed (Heap collect) — one at the end of the boot amalgams, one at the top of each REPL turn (lib/x/repl/loop.x), and lib/x/codec/sha256.x schedules its own inside the digest loop. So the interactive session is fine and everything else accumulates until the process exits. A lang that replaces the REPL loop (§7) inherits that turn sweep as a duty: forget it and a long session, or a -f script with a loop in it, grows without bound. The platform’s own note calls the per-turn sweep “the seat is quiet” — the previous turn’s eval has finished and no reader is mid-flight, which is what makes everything unreachable there genuinely dead.

Whole-file paren balance can lie. Two miscounted closers in different functions cancel to a clean total. Check each edited definition closes at depth zero, not the file sum. And bound a text replacement by the text being replaced, never by “up to the next definition” — that once deleted most of a parser.

7. Interactive is a different loop, not a different prompt

The platform REPL’s customization surface is the prompt string and the printer. The read is the ambient sexp reader, and no banner changes what a reader is — a “Python” prompt over a sexp reader evaluates 1 + 2 as three forms across three prompts. A lang REPL replaces the loop: read a line (a block, when it opens one), parse with the lang’s parser, evaluate, echo by the lang’s rules. Wire it by set!-ing the launcher’s globals (%banner, repl) from run.x.

What the loop must know:

• User stdin lives on fd 3 during boot (the boot stream occupies fd 0). Reclaim it on first call — (Sys dup2 3 0) — or you will read the exhausted boot pipe and every line arrives as EOF.
• Each line is its own parse, so any per-parse emissions (hoisted declarations, undefined-name shims) must be conditional — a guard that evaluates the name and only defines on the unbound raise — or line two clobbers line one’s bindings.
• Echo by emitted layout. The parser’s output says which forms are statements (silent) and which are expressions (echo their repr) — but inspect carefully: the same head (let) can serve both, distinguished by what it binds.
• The loop owns the collect. The platform loop sweeps at the top of every iteration, and that is what keeps a long session’s heap at its live set rather than at its history. Replacing the loop means replacing the sweep — unless your lang owns a tokenizer base, in which case you cannot have it (see §6), and a long session will grow. Know which case you are in.
• The line editor reads your lines; its colouring and its completion do not know your syntax. A lang that has not replaced repl inherits the editor; the buffer, the cursor, the history and the raw-mode bracketing carry no grammar. Three things do: set %repl-eval-line to a function from the finished line’s text that parses your syntax, evaluates and prints — asking for more lines itself with (Line read %repl-prompt-more) when the entry is not complete — %repl-paint to a function from the line’s text to the text to display for it, and %repl-complete to a function from the Edit buffer to (typed . names). The last two take () for none — but note that a nil %repl-paint means no painter installed rather than no colour, which --no-color, NO_COLOR and TERM=dumb already answer. Tab’s default prefix-searches the doc registry, so a lang that parses its own syntax completes x-lang’s names until it installs its own. Replacing repl instead gives all of this up.
• Register the bundle as a lang. (Lang register! "NAME" alist) with the seams above, then (Lang use! "NAME") if this lang owns the session. (lang NAME) at x-lang’s prompt switches to it, and your own spelling of the switch is a call to Lang use!; a seam the bundle does not name takes x-lang’s value, so say () for no painter rather than leaving it out.
• Take the prompt only when you lead. -l is repeatable, and a bundle named by a second or later -l is loaded beside the first lang. Its entry runs either way; %lang-lead is the first name, so compare it with your own and register without use! (and without replacing repl or %banner) when it is not yours. Guard the read: a wrapper older than the seam binds nothing, and the answer then is that you lead.
• The banner should identify the whole stack. %param-release (engine) and %platform-release (x-lang) arrive as boot data; printing them plus the resolved root makes every which-install-am-I-running mystery self-answering. A -dirty in the banner is a feature.

8. Making it fast

Boot cost decomposes before it optimizes. Measure with (quit) piped through each dialect: most of x-python’s boot turned out to be the xenon tower amalgam, which every tower dialect pays, plus the bundle loading under xenon’s heavier reader; the bundle’s own code was a small share under a light dialect. Know whose cost you are looking at before “fixing” your share of it.

Interpreted analysers are the hot path — one call per character per contesting type — and the platform can compile them:

• There are two JIT lanes. compile-asm (the assembler lane) emits machine code directly and needs no toolchain; compile (the cc lane) shells out to a PATH cc at runtime. Analysers use the assembler lane.
• Declare the calling mode. The compiler has two: integer functions (called from x, prim ABI, args evaluated and unboxed, result boxed) and analysers (called from C with live stack values — nothing evaluated, objects stay pointers, result returned unboxed). compile-asm’s third argument says which — #t for an analyser, #f for an integer function — and a compile that carries fvars refuses without it: an fvar is a handoff target in one world and a named callee in the other, so both are (fn (self a b c) …) over one vocabulary with nothing in the expression to separate them. A compile with no fvars and no third argument is an integer function. An analyser is for the tokenizer, not for you: direct-calling it is outside the contract.
• A body compiled for the wrong world refuses rather than answering. In analyser mode the leading one or two params are objects, and arithmetic, a shift or an ordered comparison on one of those is not something an analyser means, so it refuses at generation, names the parameter, and names the declaration that makes the compile an integer function. Without that, an integer function read as an analyser compiled and then handed back one of its own arguments. =, not, and and or stay allowed on an object param — testing a pointer for equality or for truth is meaningful, and (= buffer ()) asks a real question. See tests/x/specs/ext/jit-fvar-mode.spec.md.
• A compiled function can call something other than itself. A name bound to an fvar that holds a prim compiles to a call to that prim, and (%call HEAD arg ...) calls a prim the code computes — a callback parameter, a pointer out of a dispatch table. Both build the same argument list the self-call builds, and both check at run time that the head really is a callable prim rather than branching into its first word. Up to four arguments, and the callee gets every parameter it declares (passing fewer segfaults). A head the emitter cannot resolve still refuses at generation: silently compiling one as a self-call is what once produced an infinite recursion and a crash far from the cause.
• States that hand to each other go through a cell. An fvar is baked as its object’s address, so a state is given the states it hands to when it is made, and of two states that hand to each other one compiles before the other exists. (first CELL) and (rest CELL) read a pair when the code runs: bind the fvar to a pair, compile every state, then put the states in their cells with %set-first!. The operand is an fvar, an object parameter, or another first or rest; analyser mode only. The cell is an object of the heap, so rooting the cell roots the state it holds, which an address baked into code does not. The interpreted twin reads the same form. See tests/x/specs/ext/jit-cell.spec.md.
• %score-set’s sign folds (- 0 1) and raises loudly on other non-literals; any other non-trivial constant belongs in an fvar. %score-label!’s label is a literal integer the same way.
• Measure the tiers before writing a byte walk. A byte walk in x through the raw string prims is slow and through Str8 ref slower; a base with interpreted analysers sits between them; the same states compiled run at the engine’s own reader’s speed. The Lexer (section 4) emits the compiled form from a rule, so a bundle gets the last tier without writing a state.
• Adopt with sha256.x’s pattern: lazy, threshold-triggered, the whole attempt in a guard that pins failed and carries on pure-x. Compiling costs seconds once; never per-call, and never unconditionally at load.

9. Testing is the construction method

• Probes are specs. Never run the engine binary directly, and never probe by piping forms at it — the runner’s process timeout and alloc ceiling are the machine’s protection, and a probe written as a .spec.md becomes a regression test the moment it teaches you something. Prototype risky mechanisms (a new reader trick, a compiled callback) on a scratch base in a throwaway spec before touching working code.
• The wrapper is the contract. Every runner expects its lang wrapper (which sets LANG_LIB, X_BIN, policy knobs). Invoking a core directly produces failures that misattribute themselves — both halves of a two-session debugging saga were exactly this mistake, and the resulting guards now fail loudly. Corollaries: after editing anything that is amalgamated (tool/compile.x), make boot, or your edit silently does not load; suites resolve modules against the root the harness baked in, so point X= at the tree you mean.
• One process per spec file once a suite grows: a batch accumulates parse garbage against the alloc ceiling, and the failure (“interpreter died mid-batch”) names innocent cases. Respect the unit timeout’s margin — a file that reaches it under CI load fails from the tail backward; split files before they graze it. And never force PARALLEL over many tower-booting files, or run two heavy suites concurrently: the per-process guards cannot bound total memory.
• Size the alloc ceiling to the smallest machine that runs the suite, not the biggest. With the per-snippet collect off (SPEC_SEAM_COLLECT=0, the accumulate-then-exit regime x-ash ran in until x-engine-c v0.2.7) a spec job accumulates a whole file’s garbage, so the alloc-limit! guard is bounding a sum. The platform default is calibrated for a dev box: on a smaller CI runner a process approaching it exhausts the machine before the guard trips, and the job dies with spec-gate: killed by SIGTERM and no output at all to say why. Lower it until the guard fires first — a failed spec is legible, a killed job is not. Measure on the small machine; a workstation that passes proves nothing about the runner, and “it is green here” is how this gets shipped twice.
• Controls before conclusions. In two days, seven confident diagnoses died under control runs — a wrong bisect, two wrong suspects, a misattributed environment difference, a self-defective probe among them. The discipline that survived: reproduce on a pristine tree, run the interpreted twin of every compiled failure, exchange exact command lines before comparing results across sessions, and when an error names an operation your code never performs, check the arguments you handed the machinery first. An error that changes wording across versions of the callee while your code stands still is near-proof the defect is in what you handed it.
• State divergences as pending specs, not code comments. A divergence with a spec (return inside try skipping finally) stays visible and testable; a comment rots. And when a case’s example graduates — the feature it relied on being absent gets built — keep the old expectation as a new case asserting the new answer: the change of answer IS the feature.

10. The failure mode to fear is the silent wrong number

Across the whole build, the expensive bugs shared one pattern: no error, plausible output, wrong value. [1] + [2] printing a pointer as an integer; 1e10 reading as 1; float('abc') answering 0.0; a float literal silently truncated to its integer prefix; a discard score arriving as +65535×len. Every one was found by a spec comparing against an external oracle (CPython, or arithmetic done by hand), and several had survived under green suites because nothing asked.

The defenses, in the order they pay: compare against an oracle, not your own expectations; refuse to trust a conversion or shortcut with input the program supplied (parse it yourself and raise the lang’s own error); and when you must diverge, make the divergence loud or make it a pending spec. A loud error is a gift; the silent wrong number is the one that ships.