x-lang

x-lang

x-lang, with Bitwise the owl

    ., .,
    {O,O}
    (   )
     " "

CI

x-lang is a language built from computational-expression layers over a minimal, type-agnostic engine. The engine provides atom/pair primitives, an adaptive type system, and fexpr-based evaluation; s-expressions are the deliberately simple initial syntax — the reader itself is extensible, and whole surface languages load on top. Everything above it — the language semantics, standard library, object system, numeric tower, JIT compiler, and the toolchain itself — is written in x-lang.

This repository is the language, not an engine. An engine is acquired as a pinned, verified artifact behind a published contract; make engine fetches one.

Status

x-lang is a research and teaching vehicle for layered language construction — an investigation of how much a minimal, type-agnostic fexpr core can bootstrap without ever being modified. The answer so far: the semantics, standard library, object system, numeric tower, JIT compiler, and the entire toolchain, all written in x-lang itself.

It is useful if you want to read or borrow a working design for extensible readers, fexpr evaluation, runtime type systems, or self-hosted toolchains. It is not a general-purpose application language, and it is not trying to displace one.

Maturity — latest release. The language and the xenon dialect are covered by a full spec suite with CI on macOS and Linux plus a hard AddressSanitizer gate. The surface API is not frozen and may change between versions. The radon dialect is explicitly experimental. x86_64 parity for the automatic native-code compiler is in progress.

A taste

; Values dispatch to their class, subject-last -- and a list is callable
("hello,world" split ",")      ; -> ("hello" "world")
((list 10 20 30) 1)            ; -> 20

; Fexprs at the core: op receives its arguments unevaluated,
; plus the caller's environment as e
(def my-quote (op (x) e x))
(my-quote (+ 1 2))             ; -> the list (+ 1 2), unevaluated

; Every closure receives itself as argument 0 -- recursion needs no global name
(def fact (fn (self n) (if (= n 0) 1 (* n (self (- n 1))))))
(fact 20)                      ; -> 2432902008176640000

; The xenon dialect adds a full numeric tower with automatic promotion
(+ 1/3 1/6)                    ; -> 1/2
(* 1+2i 3+4i)                  ; -> -5+10i

More in examples/ — start with the examples guide, or run one directly with sh x.sh -f examples/x/hello.x.

A larger demonstration: other languages, on top

A lang is a different surface language implemented in x-lang and loaded over a dialect — the worked proof that whole languages sit on top of this one rather than being bolted into its core. Each lives in its own repository and is acquired as a pinned, verified bundle:

x --install-lang https://github.com/jonruttan/x-logo/releases/latest/download/lang.pin.xon
x -l logo                   # REPL + viewer at http://localhost:8080
lang is release
x-logo Logo turtle graphics, with a live browser viewer latest
x-r5rs R5RS Scheme latest
x-r7rs R7RS Scheme, on top of x-r5rs latest
x-krn Kernel, with $vau latest
x-sweet SRFI-105/110 sweet-expressions latest
x-ash a POSIX-ish shell latest
x-python a Python 3 surface unreleased
x-awk an awk: the language and the x -l awk -- CLI unreleased
x-grep a grep: BRE by translation, ERE native, -F on bytes unreleased
x-sed a sed, riding x-grep’s regex doors — the first cross-bundle lang unreleased
x-make a make: the GNU subset x-lang’s own Makefiles use unreleased
x-coreutils forty-four applets in one busybox-shaped bundle, sort to sha256sum unreleased
x-cc a C front end and evaluator; eligible functions lower to native through the engine unreleased

No version numbers in that column, deliberately — and no dialect column any more, for the same reason. Each would be a copy of a fact that lives in the bundle’s own lang.xon, and a copy nobody checks goes stale and tells the next reader something false — the failure scaling to many langs measures as one fact written in eighteen places, and the dialect column proved it here: two of its cells had already rotted when the table was next read. releases/latest cannot go stale, the --install-lang line above needs no editing when a bundle publishes, and a row without a release says unreleased rather than linking a page that would 404.

The rows from x-awk down are a second kind of demonstration — the self-hosting arc: tools x-lang’s own build invokes, reimplemented as langs, with the bootstrap tool closure as the scorecard. x -l cc runs C, and eligible functions lower to native code through the engine’s compile lane, no external toolchain.

Bundles live in their own repositories, but this tree still answers for the ones present on disk: tools/contract/langs.x records each bundle’s suite counts and make check-langs holds them — advisory about presence, strict about regression.

Logo is the largest of them — its own tokenizer types, an infix expression parser, an HTTP server and an animated SVG turtle, in ~2,400 lines. It lived in this repository’s apps/logo/ until the bundle format could carry it.

The Lang Contract is the terms a lang is held to, what it may rely on, and how one is written or extracted.

Features

Architecture

The system is layered. Each layer expands capabilities without modifying those below it.

  1. Atom/pair bootstrap (the engine) — One storage shape, two blessed lengths: every object is a fixed-size vector of slots, and the two smallest — the atom (one) and the pair (two) — are sufficient for evaluation and data construction. The evaluator dispatches through type methods, so these two suffice to get the system running.
  2. Adaptive type system — Runtime type definitions with dispatch methods (call, eval, write, length, etc.). Types and the base object share the same nested-list contract structure, extensible by appending pairs.
  3. Modular library (lib/) — ~100 modules organized by domain: core operations, custom types (vectors, strings, promises), a numeric tower (bigint, float, rational, complex, decimal), system interfaces (POSIX, FFI, GC), self-hosted tools (linter, formatter, coverage, profiler, doc generator), and platform-specific code (x86_64, ARM64).
  4. FFI and native code — Dynamic library loading via dlopen/dlsym, typed foreign calls, raw pointer operations, and a JIT compiler that compiles x-lang functions to native machine code via a data-driven assembler.

See docs/ for complete reference documentation.

Engines

Layers 1 and 2 are the engine’s, and the engine is not part of this repository. It is acquired as a pinned, verified artifact and reached through one path — the engine symlink — so nothing downstream names a particular implementation.

What an engine must provide, promise and report is docs/engine-contract.md. The conformance suite that judges one lives here rather than in any engine: an implementation that owned it would become the arbiter every other implementation is measured against.

Dialects

The library is composed into dialects that control what capabilities are loaded:

Dialects are selected via the -l flag on the shell wrapper. Langs load on top of a dialect — the table in other languages, on top names each one and the dialect it requires.

Build

One command, if you just want a working x (clones, fetches the verified engine release this tree pins, and — with --install — puts x on your PATH under ~/.local, no sudo):

curl -fsSL https://raw.githubusercontent.com/jonruttan/x-lang/main/bootstrap.sh | sh

Add --install to install (… | sh -s -- --install); the script prints how to run and how to install either way. Knobs: X_REF (branch/tag), X_PREFIX, X_SRC — see the header of bootstrap.sh.

By hand. This repository is the language. The engine is a separate project — x-engine-c — and is acquired rather than carried:

git clone https://github.com/jonruttan/x-lang.git
cd x-lang
make engine     # fetch the release tools/engine/engine.pin.xon names, verified
make

make engine downloads the engine built for your platform, checks it against the digest the pin records, and links it as engine. Nothing is compiled: the engine arrives built.

Two other ways to get one, for when that is not what you want:

make engine-source          # clone the pinned release and build it here
make X_ENGINE_DIR=../mine   # use an engine you already have

make engine-source is what you want on a platform nobody publishes for (the Pi, 32-bit), and when you are working on the engine itself. make alone, in a tree that has never acquired one, prints these three options rather than failing obscurely.

make points engine at the engine this tree builds against, builds it there if it is a checkout, and copies the x-bin binary to this repo’s root, where the wrapper and every test runner expect to find it. A C compiler is needed only for that case: a fetched release arrives built.

engine is a symlink, and it is how x-lang stays implementation-agnostic: everything downstream — the boot’s contract includes, the JIT’s -I flags, the gates — names that one path, never a particular engine. Point it somewhere else with make X_ENGINE_DIR=/path/to/engine, and the choice sticks until you change it. The target may be a checkout (built here) or an unpacked engine release (used as it comes).

The engine’s expression core needs nothing beyond libc; the full binary adds -ldl for the FFI/JIT layer. There is no -lm — float math resolves libm at runtime through the FFI, the same way it resolves any other library. One optional tool needs more: x/tool/compile — the C-emitting compiler — invokes cc at runtime and dlopens the result, so it needs a host C toolchain present when it runs. Nothing else does.

Run

The interpreter reads from stdin. Libraries are loaded by concatenation:

# Shell wrapper (recommended)
sh x.sh                     # helium (the default) + REPL
sh x.sh -l xe               # xenon: full-stack with numeric tower
sh x.sh -l rn               # radon: experimental + file I/O

# Direct invocation
cat lib/x.x - | ./x-bin         # the default dialect (helium)
cat lib/xe.x - | ./x-bin        # xenon dialect
cat lib/rn.x - | ./x-bin        # radon dialect

# Evaluate a file
cat lib/x.x program.x | ./x-bin
sh x.sh -f program.x

# Evaluate an expression, or a program on stdin
sh x.sh -q -c '(write (+ 1 2))'
echo '(write (+ 1 2))' | sh x.sh -q

The - in cat ... - | ./x-bin connects stdin for interactive use after library loading.

-c/--eval is repeatable and its expressions run in order, so a definition and its use fit in one command; it implies “and then exit”, the same bargain -f makes. A non-terminal stdin is program text, appended after the library — the wrapper’s spelling of cat lib/x.x - | ./x-bin. Nothing prints unless the program prints: use (write x) for the machine-readable form and (display x) for the human one.

Inside a session, (help) shows the documentation index; (quit) or ctrl-d exits. The session has a built-in line editor: arrow keys, the readline chords, history that persists across sessions, Tab completion over every documented name, and colour applied as you type. Nothing to wrap it in and nothing to install — see docs/repl.md.

(apropos "split") searches every documented name, (help Str8/split) gives a signature with argument types and a runnable example, and (modules) lists what is available — from a shell too, via -c.

A project can pin the library modules it depends on — keep the exact files it was written against in its own tree and declare them in a pin.xon manifest, which the wrapper finds beside the program (announced on stderr; --no-pin skips). Walkthrough: docs/pinning-tutorial.md; reference: docs/modules.md, “Pinning”.

Install

A prebuilt binary — no toolchain, no compile. Each release ships a relocatable tarball per platform (x-<tag>-<os>-<arch>.tar.gz):

tar -xzf x-<tag>-<os>-<arch>.tar.gz
sha256sum -c x-<tag>-<os>-<arch>.tar.gz.sha256   # verify the download
x-<tag>/bin/x                                     # run it, or add bin/ to PATH

It is the full install tree (wrapper + engine + library) under one versioned directory; the wrapper finds its engine and library beside itself, so it runs wherever you unpack it. macOS release binaries are Developer ID signed and notarized, so they run as downloaded; if macOS still blocks one (an unnotarized build), clear the quarantine bit with xattr -dr com.apple.quarantine x-<tag>.

From source. With a C compiler:

make install                # /usr/local by default
make install PREFIX=~/.local

Installs the wrapper as bin/x (the user-facing command), the engine binary under libexec/x/, and the runtime tree under share/x/: the library and apps byte-identical to the repo’s (diff -r runs inside the install as proof), plus generated amalgamated boot entries under share/x/boot/ — so x, x -l xe, and x -f program.x work from any directory. DESTDIR is honoured for staged/packaged installs. Remove with make uninstall (same PREFIX).

Test

make test-x                          # x-lang spec suite (2,500+ cases)
make test-c                          # the engine's C unit tests (delegated)
make test                            # all tests

Test specs are markdown files in tests/x/specs/ organized by category: core language, closures and applicatives, extensions (types, numeric tower, compile), standard library, end-to-end, and tools. CI runs the full suite on macOS and Linux, plus a hard AddressSanitizer gate.

make test-c delegates to the engine, and a fetched release ships no C to test: it announces that it skipped and names what covers that ground instead. The same is true of check-isa, check-obj-layout and check-base-paths — docs/contributing.md says which is which, and why a gate that goes quiet says so out loud.

Documentation

Start here: docs/index.md — the documentation front door, with a suggested reading order. The whole of it, both generated references included, is published at https://jonruttan.github.io/x-lang/. New to the vocabulary? The Glossary defines the load-bearing terms (fexpr, operative, dialect, the base, contract).

Coding agents start at AGENTS.md (CLAUDE.md symlinks to it) — the short briefing: how to run it, the Lisp assumptions that fail here, and the three calls that answer “what exists” without reading any of the above. llms.txt is the machine-readable index of this list.

Guides

References

Tools

License

MIT No Attribution (MIT-0)

Bitwise