The contract between the writer (tools/dev/image-write.x), the loader
(tools/dev/image-read.x), and (image rebuild!) in the engine.
state-images.md argues why; this says what. When code
and this document disagree, this document is what gets fixed first, and
nothing enters the format that is not written here.
This is version 1 of the format. Every claim below about the engine was
read from ext/x-expr and x-engine-c source, and names the file it came
from.
All state is on the base. x-expr’s base is one atom (x_base_make,
x-base.c) whose data word points at a tree of pairs and atoms; the
engine’s (x_eval_make, x-eval.c) has a second unit, its slot vector.
Every leaf is a field cell, (value . saved-values), and everything the
evaluator, the type system, the reader, the printer, the collector and the
allocator know is a node of that tree. There is no table anywhere that is
not reachable from the base object.
The tree’s layout is the contract engine/tools/contract/base-layout.x, and
its labels already say which parts belong to whom:
| label | who builds it | what it holds |
|---|---|---|
(build …) subtrees |
x_eval_make (x-eval.c) from the descriptor |
language state: env, ctrl, type-alist, io-state, profile counters, the state fields |
(todo …) subtrees |
x_base_make (x-base.c) |
process state: file descriptors, the read buffer, the hooks, the heap fields, the allocation fields |
That split is the whole design. An image is a base’s language state and every object reachable from it. A loader has a base of its own with its own process state, and the image never touches that.
Language state, cell by cell (the x_eval_field_* accessors of the
generated x-eval-layout.h):
| fields | cells | notes |
|---|---|---|
| env | env, env-root (slots) |
the current environment and the root, each one pair of bindings and parent; the root’s bindings are the tree over globals |
| ctrl | save-stack, error-handler, tco-expr, tco-env |
evaluator transients; nil at image time and at install (§6) |
| io-fields | type-alist (cell) |
the type registry: ((name-stack . struct) …), keyed by the name-stack node (x_alist_assoc compares first(key), x-alist.c) |
| io-state | line, true, false |
true/false hold the engine statics x_true_obj/x_false_obj |
| profile | ten counters | plain integers; the loader may keep its own |
| state | eval-list, token-cache, sigint, err, prims, file, err-line, err-file, file-registry, parent |
err is the one base-resident ERR instance every raise fills (x_type_err_register); prims is the catalogue, a list of (ns (method . PRIMITIVE) …) (x_prim_register); sigint is a shared atom; parent is the base this one was made from, nil in the root |
Process state, which the loader keeps: files (descriptors, write-buf, and
the read buffer — a BUFFER over a C array on main’s stack, x-cli.c),
the slot vector (the engine’s routines, C function pointers, in the base’s
second unit and outside the tree), the four hooks and the heap fields’
mark/free hooks (static atoms holding C function pointers), mark-hooks,
free-hooks, mark-roots, the root-chain (off-chain stack objects pushed
by frames, marked never swept, x-base.h), obj-meta-extra, and the
alloc fields.
Two more facts about where things live:
x_type_struct_make (x-type.c) builds a
pair tree per type; the registry cell lists them. A type struct’s cells
hold handler lists; the built-in types’ handlers, name atoms, and the
default units/length values are static objects in the engine binary
(x_type_int_name, x_type_int_make_prim, x_type_units_pair_obj, …).
The library’s pushed handlers are heap closures. The SYMBOL type’s data
slot holds the intern list and BST (symbol.c), so symbol identity is
part of the tree too.x_obj_alloc: x_obj_heap(obj) = x_obj_heap(base); x_obj_heap(base)
= obj). Walking from the base object’s heap link enumerates everything the
base ever allocated and not yet freed. A child base (base make) has a
separate chain.An image is: the values of the language-state cells of one base, and the transitive closure of objects they reach, written so a different process running the same engine release can rebuild them as objects of its base and set its language-state cells to them.
Consequences:
save and
load (§4.3): a type turns its own objects into labelled words and fixes a
rebuilt one up. The writer never reads a type’s units from outside; it
asks the type, through (image save!).Words are %word-size bytes, little-endian. Sections are contiguous:
| # | section | length |
|---|---|---|
| 0 | header | fixed |
| 1 | externals table | XWORDS |
| 2 | roots table | RTWORDS |
| 3 | object table | OBJW |
| 4 | byte blob | BLOBN bytes |
| word | name | meaning |
|---|---|---|
| 0 | magic | "XIMG" little-endian |
| 1 | version | 1 |
| 2 | word size | refused on mismatch |
| 3 | byte order probe | 1; refused if not |
| 4 | N |
object count |
| 5 | OBJW |
object table words |
| 6 | BLOBN |
blob bytes |
| 7 | XCOUNT |
external entries |
| 8 | XWORDS |
|
| 9 | RTCOUNT |
root entries |
| 10 | RTWORDS |
|
| 11 | META |
extra metadata words per object the writer’s base used (obj-meta-extra; 2 in a CLI base) |
| 12 | RELEASE |
blob offset of the engine release string (x-release) |
The loader refuses a RELEASE that is not its own: an image is a heap laid
out by one build of the engine, and its externals are named against that
build’s catalogue and statics.
A name is [len][bytes…] in words-for(len) = 1 + ((len + 1) >> 3) words —
room for a NUL, which a zeroed buffer supplies. Steps, where used, are
[n][s…] with 0 = first, 1 = rest.
There is no unit-label table. Every record (§3.6) says how many units it has and
what label each is, because the type’s save handler wrote it that way.
Labels: ref 0, word 1, bytes 2, foreign 3.
Three types have no struct and are saved structurally, by role:
| role | label | payload |
|---|---|---|
spair |
x_type_pair_obj — a type-struct node |
2 ref |
satom |
x_type_atom_obj — a static atom, or a type handle: the name atom make-type allocates |
1 unit: bytes when the object is OWN (the handle’s name), else a word |
nil-typed |
NULL type — #t, #f, the base’s own atom |
1 word |
An ordinary pair is not a role: its type is the base’s PAIR struct, in
the registry like any other, and its save says two references.
[label][name]
A unit that refers to something outside the image stores an external
index (negative in a ref unit, plain in a foreign unit). Labels:
| label | names | loader resolves to |
|---|---|---|
1 catalogue |
ns/method |
the C function pointer behind its own (prim-ref ns method) |
2 bare |
a global’s name | the function pointer of the callable bound to that name in a fresh base |
3 dlsym |
a C symbol | dlsym |
4 typecall |
PROCEDURE or OPERATIVE |
the call pointer a fresh closure of that type carries |
5 dlopen |
— | the process handle |
6 static |
a role name | one of the engine’s static objects, by role — see below |
7 type-static |
a type name and a row of base-paths.x |
the static object a freshly registered type of that name holds at that row |
8 base-row |
a base-rooted row of base-paths.x, or base |
that node of the loader’s base tree — the spine cell the library cached (%reflect-base-cell), or the base object itself |
Label 8 exists because the library holds spine cells by reference:
boot/reflect.x and boot/registry.x cache type-alist, prims, false,
err-line, err-file, file-registry, obj-meta-extra. A spine node is
never imaged; a reference to one resolves to the loader’s node of the same
row, and a spine node with no row is unnameable.
Label 6 names the
statics that are not inside any type struct: true, false (x_true_obj,
x_false_obj), label-atom, label-pair (the two labels, when they appear as
type words), units-atom, units-pair, length-atom, length-pair,
token-eof (x_token_eof_prim). Label 7 names the statics that are: a
built-in type’s name atom, its C handler atoms, its default units value.
x_type_int_register builds INTEGER’s struct from the same static
descriptors in every base, so (type-make INTEGER) is the same object in
the writer’s process and the loader’s.
The writer discovers label-7 names by walking a pristine (base make) and
recording every off-chain node (heap link 0) under the row that reaches
it. That is why a C reader buried under four library pushes in STRING’s read
stack still has a name: it is what a fresh STRING holds at type-read.
Label 6 comes from a fixed list; label 7 is derived; nothing else is static.
An external the writer cannot name is stored as XCOUNT + 1; the loader
resolves it to nil, counts it, and reports the count. A nonzero count
is a broken image (§6.9).
[cell name][ref]
ref is a ref unit word (§3.6): an object index, or a negative external —
the true and false cells hold engine statics.
One entry per language-state cell of base-layout.x (§1). The name is the
contract’s, so the loader finds the cell by the same path the C accessors
use. cell rows get the object in their first slot; slot rows get it in
the half of the parent the last step names — the distinction is the
contract’s, not the loader’s.
N records, no length word:
[type][flags][n][label word]…n
type is the object index of its type struct, or -1 spair, -2 satom,
-3 nil-typed. n and the labels are what (image save!) answered for this
object. flags are the writer’s flags masked to WRAP 0x01, COV 0x02,
RO 0x40 (bits 0x04 and 0x08, once FRAME and FNFRAME, are free
since environments became values); SHARED is set on every rebuilt
object regardless (the image lives as long as the process); OWN, META,
MARK are never replayed.
Unit words: ref — > 0 object index, < 0 external, 0 nil; word —
verbatim; bytes — blob byte offset; foreign — external index.
A word unit carries any value a machine word can hold, so an object of
word units carries arbitrary bytes, zero bytes included. A type whose
units are declared -1 with the labels (ref word) is such an object: slot 0
is its length, an INTEGER, and the payload after it is words.
[len][bytes… NUL] at the offset a bytes unit stores. The loader’s buffer
outlives the image; a rebuilt bytes unit points into it.
The writer measures a bytes unit as a C string, so len is the count up to
the first NUL. Bytes that may contain a NUL go in word units (§3.6).
The walk is the imaged base’s allocation chain, from the base object’s heap
link, filtered to objects the mark reached. The mark is
(heap tree-mark! base-object TRACE) — the collector’s own traversal from
the base, with a flag the collector does not use — so “reachable” means what
the collector means by it, type hooks included. The mark is taken
back with image-clear!: the engine’s chain-clear! alone leaves the flag
on the base object, which no chain holds, and a later mark from the base
then flags nothing. A spine cell the library
holds – the false cell, the registry cell – is reached by the mark and
is not language state: the writer takes the flag off it before the walk,
which is (image write!)’s (§7): the loop sees flags and nothing else.
The mark follows every object to its type, and a type’s handlers are
closures that hold their environment, so a mark from any object, even a
fresh pair, an integer, a symbol or a string, flags the whole environment.
Since the flag is also the mark’s visited test, a writer that wants less
flags its stopping points before the mark and takes those flags off after:
the image writer flags what the image names rather than writes (the statics,
and the parent base) with %trace-stops! in tools/dev/image-write.x.
pin! is not affected: its walk stops at the SHARED type nodes.
While a frame holds a cursor into a child base’s chain, this base must not collect: the collector marks through whatever a frame holds, and a sweep clears flags on this chain only, so each collect leaves stale mark bits on the child. Walk the child with the periodic collect off; collect this base between walks.
Nothing walks bytes; strings reach the blob through one copy each.
x-cli.c binds include, syscall (x_callable_bind) and args,
x-machine, x-version, x-release (x_value_bind) into the root base
only. A child base the writer images gets each of them as its own
object: the two primitives through (obj make-callable fnptr) evaluated
in the child, the strings copied in the child, args as a list the child
builds. Nothing the child’s language state reaches then lives on another
base’s chain, and the walk is the child’s chain alone.
Every type struct has two more cells, type-save and type-load
(base-paths.x rows type-save, type-load), beside mark, make,
read and write. Both are handlers like any other and are pushed like
any other; (type make name handlers) takes them under the keys save
and load.
save, applied by (image save! obj buf) with evaluated arguments,
writes the unit count at buf[0] and [label][word] pairs after it, and
returns the object. The engine provides one per built-in type whose
payload is not all references — STRING and SYMBOL say bytes, INTEGER
and CHARACTER word, POINTER foreign, PRIMITIVE foreign foreign
(the function pointer, then a second address the engine owns), PROCEDURE
and OPERATIVE foreign ref, BUFFER bytes ref word for its outer (the bytes,
the inner, the consumed count) and two words for its inner (0 and the
unread count) — and x_type_save_default, which walks the unit labels,
for every type that declares none. A type the library registers gets the
default, two references (make-instance’s layout), unless it pushes its
own. A handler evaluates nothing and allocates nothing: the per-type
files link without the evaluator.load, applied by the rebuild’s third pass to every rebuilt object whose
type has one, once every object’s units are in place. BUFFER’s outer puts
its inner’s read and write pointers back from the consumed count, the
unread count and the bytes’ new address; a rebuilt outer has the third
unit, a fresh one does not. Most types have none.A word is a reference iff its label says so. An INTEGER’s unit is word; it
is never resolved, even when it holds an address — a global that caches a
type word (%reflect-satom-tw, %print-int-tw) is written verbatim and is
wrong after load until the library recomputes it. A cached process
address is a boot-time computation the image cannot carry; the library
recomputes it in the loader’s last step (§5, step 5).
None. A type is its struct’s object index. Two bases’ STRING types are two structs and two indices; the loader does not care which is which.
(image rebuild!): three passes over the object table. The first
allocates every object on this base’s chain (its own obj-meta-extra
applies) with the record’s unit count, typed by the static label for a
role and untyped otherwise. The second sets each untyped object’s type
word to the rebuilt struct its record names and patches every unit by
the label the record carries. The third applies each object’s type load
where there is one. The loader hands the primitive the externals as one
vector whose entry k is an object (labels 6–8) or an integer address
(labels 1–5), and the blob’s address.ctrl cells last and to nil. Any operative or
procedure here pushes a TCO compound onto the save-stack that a later
restore would use to put this base’s stale env back (x_tco_restore,
x-eval.c); a primitive call pushes nothing. After the last write the
loader’s own names are gone, and the next form on stdin evaluates
inside the image. Nothing the library defined has been called so far,
except a type’s load handler in step 3.(%image-recache!), resolved in the image. A library module that
caches an address at boot, or writes into an engine static – the two
type labels boot/reflect.x keeps, the four struct words boot/printer.x
keeps, the relabel of #t and #f in type/bool.x – adds a thunk to
%image-recache-hooks, and this call runs them all, in the order they
were added. Every image’s
library binds the name; lib/img.x binds it to nothing.
A value the image cannot carry at all is a transient: the module
lists the global in %image-transients, the writer sets it to nil
inside the child before the walk, and the module’s own hook re-derives
it here. num/float.x’s dlopen handle is one – no symbol names it –
and so is every pointer resolved through it: a symbol of a library the
loader has not opened does not resolve (glibc keeps a dlopen’d libm out
of the global scope; macOS folds it into libSystem), so float records
each such binding as it makes it, one thunk among the transients clears
the handle, the bindings and the cells they read (the names are the
module’s own, which a symbol in the list would not reach: the writer
clears a symbol in the child’s root), and one hook remakes them all.
A value the image cannot carry but the library can REMAKE – the tower’s
compiled analysers, native code in a page the writing process mapped –
is the other half of the same rule: the module lists a THUNK in
%image-transients instead of a symbol, the writer runs it inside the
child before its walk, and that thunk puts the carryable form back.
lib/x/tool/compiled.x keeps the list of functions that have a compiled
version, and its thunk switches every one to interpreted: the tower’s
interpreted analysers are installed again and the compiled objects let
go. Each entry added a recache hook of its own as it was made, so here
they compile again in the order they were made, after the
hook of boot/tower-compiled.x that asks the lane again, because the
loading engine is not the writing one. The code itself is in the image:
lib/x/tool/asm-cache.x holds each compile’s code and relocation
records in the heap, so a compile here copies the code into a fresh
page and patches it.
A module adds to either list through the catalogue, (image transient!)
and (image recache-hook!), which boot/reflect.x files.The loader is silent when a runner drives it; with %IMG-VERBOSE bound it
prints one line of counts and the unresolved externals by name.
On a small base, seconds per cycle; the x-core image is the integration test, not the unit test.
(pair 1 2) after
load has the same type word.(new C …) makes an instance; object? and class-of agree; a method
dispatches; (Type of instance) is the rebuilt struct’s name-stack node’s
atom.same? as the imaged symbol of that name
(the intern table came with the image).Err is caught as an object; (Err label e) answers; a C
primitive’s type error is the resident ERR.eval! from inside a call, printing, then lookups of each — ASan-clean.#t after load is same? as the loader’s x_true_obj; the SYMBOL
type’s C reader is the same static as in a fresh base.ctrl cells are nil in the file and after install; the save-stack is
empty when the next form is read.num/float.x, the libm handle is a pointer after the load,
and the call-time path through it answers.#t the reader produces after load is eq? to the evaluated #t:
the interned name shares the singleton’s bytes, as it does in a fresh
base, where binding the name interned it on those bytes.x-base.x, every
analyser boot/tower-compiled.x compiles is a PRIMITIVE after the load,
%tower-jit? answers for the loading engine, and 1.5d, 1+2i and
(/ 1 3) read and answer as they do from source.save (§4.3), and any whose payload needs fixing up once its words are
in place provides a load.(image save!) applies a type’s save with evaluated arguments and saves
the three roles structurally; (image rebuild!) allocates from the
record’s own count and labels, sets SHARED, and runs the load pass.(image write!) is the writer’s loop: it walks the chain from a cursor,
indexes every object carrying the caller’s flag, and writes each one’s
record through the same save. It names nothing. A word it cannot place
– a reference to an object outside the image, a foreign address – goes
to a callable the caller supplies, (name word label obj), once per
distinct word; an integer answer is the external index and anything else
the sentinel. It also answers the object index of each root value the
caller lists, so the roots table needs no second index.line sits in the layout’s build region, but
the C reader holds the counter atom directly and counts into it, so a
cell swapped under the reader reads 0 from then on. Writer and loader
leave it out of the roots, beside the profile counters, sigint and
parent, the base an imaged child was made from: the writer’s, which is
no part of the image.obj-meta-extra: line and file ids) is not carried;
a loaded image’s error reports have no source positions. The loader
rebuilds with the metadata width at 0 and puts it back after, so a
rebuilt object carries no META flag: with the flag over zeroed slots,
evaluating any imaged form would set the line counter to 0.tools/dev/image-build.sh’s key does that
outside the file.