docs: system-image.md — the boot capsule documented in full

system.img was only mentioned in passing (efi.md, system-requirements.md); its format, builder, fallback chain, and kernel-side life were spread across pack-system-image.py, build.zig, efi.zig, initial-ramdisk.zig, and vfs.zig. New dedicated page covers all of it, cross-linked from every prior mention and added to the docs index.
2026-07-22 10:37:55 +01:00 · 2026-07-22 10:37:55 +01:00 · 1882161cb4
parent 08e139ebba
commit 1882161cb4
4 changed files with 162 additions and 24 deletions
--- a/docs/README.md
+++ b/docs/README.md
@ -7,79 +7,86 @@ rather than restate it. Roughly in the order things happen at runtime:
   runs the bootloader, what the loader gathers before `ExitBootServices`, how it
   loads the kernel ELF, and the ABI contract for the jump into the kernel. Start
   here.
-2. **[gop.md](gop.md) — the Graphics Output Protocol.** How UEFI exposes graphics
+2. **[system-image.md](system-image.md) — system.img, the boot capsule.** The
+   bundled user binaries packed into one file in the initial-ramdisk wire
+   format, because one open + one sequential read is the only file I/O shape
+   firmware is fast at. The trivial container format, the three artifacts one
+   build list derives (tree, manifest, capsule), the loader's three-strategy
+   fallback chain, and the capsule's kernel-side life as both the spawn table
+   and the read-only `/system` mount.
+3. **[gop.md](gop.md) — the Graphics Output Protocol.** How UEFI exposes graphics
   modes (unlike fixed VGA modes), how we detect the monitor's native resolution
   from EDID and switch to it, and the pixel formats we accept or reject.
-3. **[framebuffer.md](framebuffer.md) — the framebuffer.** What the linear
+4. **[framebuffer.md](framebuffer.md) — the framebuffer.** What the linear
   framebuffer the loader hands over actually is, and what **pitch** (stride)
   means versus width — the detail you have to get right to avoid a skewed image.
-4. **[memory-map.md](memory-map.md) — the memory map.** How the loader learns what
+5. **[memory-map.md](memory-map.md) — the memory map.** How the loader learns what
   physical RAM exists and hands it to the kernel in danos's own neutral format,
   rather than leaking UEFI's memory descriptors across the boundary.
-5. **[frame-allocator.md](frame-allocator.md) — the physical frame allocator.** The
+6. **[frame-allocator.md](frame-allocator.md) — the physical frame allocator.** The
   bitmap allocator that hands out and reclaims 4 KiB physical frames from that
   map — the primitive page tables and the heap are built on.
-6. **[interrupts.md](interrupts.md) — interrupts and exceptions.** The GDT, IDT and
+7. **[interrupts.md](interrupts.md) — interrupts and exceptions.** The GDT, IDT and
   TSS, the exception stubs, and the handler that reports a CPU fault in red instead
   of letting it triple-fault into a silent reset.
-7. **[paging.md](paging.md) — the kernel's page tables.** Building our own 4-level
+8. **[paging.md](paging.md) — the kernel's page tables.** Building our own 4-level
   page tables, identity-mapping the low 4 GiB, and switching CR3 off the firmware's
   tables onto ours.
-8. **[device-interrupts.md](device-interrupts.md) — device interrupts.** The Local
+9. **[device-interrupts.md](device-interrupts.md) — device interrupts.** The Local
   APIC and its timer — the kernel's first interrupt that is *handled and returned
   from*, giving it a heartbeat.
-9. **[heap.md](heap.md) — the kernel heap.** A growable free-list allocator built on
+10. **[heap.md](heap.md) — the kernel heap.** A growable free-list allocator built on
   the VMM, exposed as a `std.mem.Allocator` so std containers work — dynamic
   allocation for the kernel.
-10. **[scheduling.md](scheduling.md) — the scheduler.** Fixed-priority preemptive
+11. **[scheduling.md](scheduling.md) — the scheduler.** Fixed-priority preemptive
    multitasking: kernel threads, the context switch, O(1) priority selection, and
    blocking (sleep, wait queues) — the leap to a running system.
-11. **[ipc.md](ipc.md) — inter-process communication.** Bounded blocking
+12. **[ipc.md](ipc.md) — inter-process communication.** Bounded blocking
    message-passing channels, then synchronous call/reply between *processes* over
    endpoints — the backbone the microkernel's isolated servers talk over.
-12. **[syscall.md](syscall.md) — system calls.** How ring 3 asks the kernel for
+13. **[syscall.md](syscall.md) — system calls.** How ring 3 asks the kernel for
    something: the `syscall`/`sysret` fast path, the trap frame, and why the table is
    deliberately tiny. The numbers are a **private** ABI — [vdso.md](vdso.md) designs
    the public boundary that will hide them.
-13. **[vfs-protocol.md](vfs-protocol.md) — the VFS wire protocol.** The language-neutral
+14. **[vfs-protocol.md](vfs-protocol.md) — the VFS wire protocol.** The language-neutral
    byte-level spec of the file protocol spoken over IPC: request/reply headers,
    the operation table, mount routing, and the append-only evolution rules — the
    first IPC protocol documented as public ABI.
-14. **[drivers.md](drivers.md) — writing a driver.** The payoff: a driver is an
+15. **[drivers.md](drivers.md) — writing a driver.** The payoff: a driver is an
    ordinary ring-3 process that claims a device, maps its registers, and **sleeps
    until its hardware interrupts it**. The claim is the capability; `irq_ack` is the
    unmask.
-15. **[driver-model.md](driver-model.md) — buses, classes and host controllers.** How
+16. **[driver-model.md](driver-model.md) — buses, classes and host controllers.** How
    real driver stacks factor into three shapes and how families share code. The
    three primitives it proposed are long since built (M13 capability passing,
    M14 DMA + barriers, M15 MSI), and the driver *contract* on top of them —
    hello, supervision, restart — is built too (device-manager.md, M18).
-16. **[usb-hub.md](usb-hub.md) — USB hubs.** Built (M22): why hub topology is handled
+17. **[usb-hub.md](usb-hub.md) — USB hubs.** Built (M22): why hub topology is handled
    *inside* the `usb-xhci-bus` driver rather than a separate hub class driver — a
    device behind a hub is reached by the **controller**, programmed with a route
    string in its slot context — plus the compound-hub reality (a USB 3.0 hub is
    physically two hubs) and detection via the hub's status-change interrupt endpoint.
-17. **[process-management.md](process-management.md) — process management.** The
+18. **[process-management.md](process-management.md) — process management.** The
    microkernel's `ps`/`kill`/SIGCHLD: enumerate as a table snapshot, the
    supervision link as the kill authority, and child-exit notifications over the
    same endpoints IRQs arrive on.
-18. **[process-lifecycle.md](process-lifecycle.md) — the process lifecycle.** Built
+19. **[process-lifecycle.md](process-lifecycle.md) — the process lifecycle.** Built
    (M17): signals over IPC as the one lifecycle vocabulary every process speaks — the
    POSIX.1-1990 words with message delivery instead of stack hijack, the stable
    `runtime.process` interface, exit reasons, published exit events any stateful
    service can subscribe to (the VFS releasing dead clients' handles), and the two
    iron rules (cleanup is the kernel's job; kill is not a signal).
-19. **[device-manager.md](device-manager.md) — the device manager.** Built (M18,
+20. **[device-manager.md](device-manager.md) — the device manager.** Built (M18,
    through the app surface): the
    tree, the matcher, and the supervisor. Tree structure lives in the manager,
    authority stays in the kernel; bus drivers report what they see; drivers are
    restarted through the lifecycle vocabulary — the plan that turns
    [resilience.md](resilience.md)'s restart goal into increments.
-20. **[input.md](input.md) — the input module.** Broadcasting input events (keyboard,
+21. **[input.md](input.md) — the input module.** Broadcasting input events (keyboard,
    mouse, joystick): why a synchronous rendezvous can't fan out to many listeners, the
    asynchronous `ipc_send` primitive built to fix it, and the per-device subscribe/publish
    service layered on top.
-21. **[display.md](display.md) — the display service.** The display half of the GUI
+22. **[display.md](display.md) — the display service.** The display half of the GUI
    track: a user-space compositor that owns the framebuffer, composes a layer stack into
    a double buffer, and presents it. Why GOP and the PCI display device are two views of
    one controller, the device-node + write-combining handoff, and what flicker-free buys
@ -90,7 +97,7 @@ rather than restate it. Roughly in the order things happen at runtime:
    further out, two research snapshots survey what a *native* driver for real GPU silicon
    would take as another `.scanout` backend: [nvidia-gpus.md](nvidia-gpus.md) (RTX 3060 /
    Ampere) and [intel-igpu.md](intel-igpu.md) (Intel iGPU).
-22. **[halting.md](halting.md) — halting.** Why a kernel can't just "exit", and
+23. **[halting.md](halting.md) — halting.** Why a kernel can't just "exit", and
    how `while (true) hlt` parks the CPU safely once there's nothing left to do.

 Start with the north star:
--- a/docs/efi.md
+++ b/docs/efi.md
@ -27,7 +27,8 @@ The boot volume is **FHS-shaped** (see the repository-layout note in
 [README.md](README.md)): `build.zig` installs `boot/efi.zig` (built for the `uefi`
 target) at `EFI/BOOT/BOOTX64.efi` — the one path UEFI firmware fixes — and lays
 the rest out by FHS path: the kernel at `system/kernel`, init at
-`system/services/init`, the pre-packed boot capsule at `boot/system.img`.
+`system/services/init`, the pre-packed boot capsule at `boot/system.img`
+([system-image.md](system-image.md)).
 `zig-out` mirrors that tree, but what a machine actually boots is the
 self-contained FAT32 image `tools/make-fat-image.py` builds from the same files
 (`danos-usb.img`). The `run-x86-64` step points QEMU at OVMF (UEFI firmware for
@ -67,7 +68,8 @@ services are still up), loads the system binaries into an in-RAM
 **initial ramdisk** (`loadSystemTree` — normally a single read of the pre-packed
 `boot\system.img` capsule, which already *is* the ramdisk wire format; it falls
 back to opening each manifest-listed path, and walks the `/system` tree only as
-a last resort for hand-assembled sticks. Best-effort either way — a kernel-only
+a last resort for hand-assembled sticks — the capsule's format, builder, and
+fallback chain are documented in [system-image.md](system-image.md). Best-effort either way — a kernel-only
 volume still boots), and builds the **bootstrap page tables** the kernel starts
 life on (`buildBootstrapTables`), all before the jump:

--- a/docs/system-image.md
+++ b/docs/system-image.md
@ -0,0 +1,128 @@
+# system.img — the boot capsule
+
+## What it is
+
+`boot/system.img` is the **boot capsule**: every bundled user binary — init, the
+services, the drivers, the test programs — packed into **one file** on the boot
+volume. It is not a filesystem image and it is not compressed; it is exactly the
+kernel's **initial-ramdisk wire format** (`system/initial-ramdisk.zig`, format
+v2), written to disk ahead of time. The EFI loader reads it in a single
+sequential pass and hands the bytes to the kernel unmodified.
+
+The capsule is a *performance artifact*, not a source of truth. The boot
+volume's `/system` file tree remains the canonical layout (see
+[danos-file-system-hierarchy-FSH.md](danos-file-system-hierarchy-FSH.md));
+the capsule is a pre-baked snapshot of the same binaries, derived from the same
+build graph, so the running system is identical whether the loader read the
+capsule or walked the tree.
+
+## Why it exists
+
+Firmware file I/O has exactly one fast shape: **one open + one sequential
+read**. Everything else is a lottery. Loading the system per-file — dozens of
+opens, seeks, and short reads through the firmware's FAT driver — measured
+**minutes** on real hardware, against milliseconds in QEMU/OVMF. Packing the
+binaries into a single file turns the whole of user space into the shape
+firmware is good at.
+
+Because the capsule already *is* the ramdisk wire format, the loader doesn't
+even repack it: `loadCapsule` (`boot/efi.zig`) validates the magic and passes
+the buffer straight through as `BootInformation.initial_ramdisk_base`/`len`.
+
+## The format
+
+The container is deliberately trivial — danos owns both producer and consumer,
+so it need be no fancier. Little-endian throughout:
+
+```
+Header        magic: u32 = "DNR2" (0x32524E44), count: u32
+Entry × count name: [64]u8 (NUL-padded FHS path), offset: u64, len: u64
+blobs...      each entry's file bytes, at its offset within the image
+```
+
+- **Names are full FHS paths** (`/system/services/init`), not basenames — that
+  is what "v2" means. The 64-byte capacity matches `abi.maximum_process_name`,
+  so a task named after its binary path is never truncated. Paths longer than
+  63 bytes are a build error (`pack-system-image.py` rejects them).
+- **The v1 magic (`"DNRD"`, basename entries) is rejected**, not tolerated: a
+  stale image should fail loudly at `Reader.init`, not misparse names.
+- `initial_ramdisk.Reader` is the one validated view over the bytes — magic
+  check, table bounds, per-blob bounds — used by the kernel and shared with the
+  loader. `Reader.find` resolves a binary by exact path first, then by unique
+  basename, ASCII case-insensitively (the entries come from a FAT volume, whose
+  name lookups are case-insensitive by definition).
+
+## How it is built
+
+`build.zig` maintains one `bundled` list — every user binary and its FHS home.
+Three artifacts are derived from that same list, in the same build graph, so
+they cannot drift apart:
+
+1. **The tree**: each binary installed at its FHS path (`zig-out/system/...`,
+   mirrored onto the FAT boot volume by `tools/make-fat-image.py`).
+2. **The manifest** (`system/manifest`): the FHS path of every bundled binary,
+   one per line — the loader's per-file fallback input.
+3. **The capsule**: `tools/pack-system-image.py` packs the same binaries into
+   the v2 container, installed at `zig-out/boot/system.img` and placed on the
+   boot volume at `boot/system.img`.
+
+Note what the capsule does *not* contain: the kernel (`system/kernel` is loaded
+separately by `loadKernel`, as an ELF) and the EFI loader itself. It is user
+space only.
+
+## How it is loaded
+
+`loadSystemTree` (`boot/efi.zig`) tries three strategies, most portable first —
+the running system cannot tell which one ran, because all three produce the
+same in-RAM ramdisk image:
+
+1. **The capsule** — open `boot\system.img`, read it whole, check the magic,
+   hand it over as-is. The normal path on any build-produced volume.
+2. **The manifest** — read `system\manifest` and open each listed path *by
+   name*. FAT name lookup is case-insensitive and firmware-portable, unlike
+   directory enumeration. The loader assembles the v2 image in RAM itself.
+3. **The tree walk** — enumerate `/system` recursively. Last resort for
+   hand-assembled sticks with neither file: some firmware FAT drivers return
+   bare 8.3 names uppercase from enumeration, which is why this is the
+   fallback and not the primary path.
+
+All three are best-effort: a **kernel-only volume still boots** — the kernel
+just has no user binaries to spawn and reports the absence.
+
+One operational consequence of the ordering: the capsule *shadows* the tree.
+If you hand-edit binaries on a stick that also carries a `boot/system.img`,
+your edits are invisible — the loader boots the capsule's snapshot. Delete
+`boot/system.img` from the volume to force the manifest/tree path.
+
+## What the kernel does with it
+
+The loader records the image's physical base and length in `BootInformation`;
+the kernel (`kernel.zig`) then publishes the same bytes twice, to two
+consumers:
+
+- **The process layer** (`process.zig`): `system_spawn` looks binaries up in
+  the ramdisk via `Reader.find` — exact FHS path, or unique basename for
+  pre-path callers — and loads them as fresh ring-3 processes. The stored path
+  becomes the task's name.
+- **The VFS root** (`vfs.zig`, `setInitialRamdisk`): the image is mounted as
+  the kernel-backed, read-only `/system` mount. Directory nodes are derived
+  from the entry paths (the unique parents), so `/system` is listable and its
+  files readable over the normal VFS protocol — the FHS boot tree every
+  process sees comes straight out of the capsule bytes.
+
+The image is never copied after the handoff and never mutated: the initrd is
+immutable, which is what makes the VFS's node serving lock-free.
+
+## What it is not
+
+- **Not `danos-usb.img`.** That is the 64 MiB FAT32 *boot volume* built by
+  `tools/make-fat-image.py` — the thing a machine actually boots, which
+  *contains* `boot/system.img` alongside the loader, kernel, manifest, and
+  tree. See [efi.md](efi.md) and [release-iso.md](release-iso.md).
+- **Not a mountable filesystem.** No FAT, no block device, no driver — just a
+  header, a table, and concatenated blobs, parsed by ~90 lines of
+  `initial-ramdisk.zig`.
+- **Not required.** It is the fast path, with two slower equivalents behind
+  it.
+- **Not a place where state lives.** It is regenerated on every build from the
+  bundled binaries; nothing writes to it, at build time or runtime.
--- a/docs/system-requirements.md
+++ b/docs/system-requirements.md
@ -117,7 +117,8 @@ hypervisor configured for UEFI firmware and an xHCI USB controller.
  FADT (power / PM timer). Optionally consumed: HPET, DMAR, SPCR.
  (`system/devices/acpi.zig:3`)
 - The loader reads `/system/kernel` off the FAT boot volume, then loads user
-  space: a prebuilt `boot\system.img` capsule when present, otherwise it walks
+  space: a prebuilt `boot\system.img` capsule
+  ([system-image.md](system-image.md)) when present, otherwise it walks
  the volume's `/system` tree (init included) into the initial ramdisk. The
  kernel can boot "kernel-only" without either. (`efi.zig:16`, `efi.zig:68`)