docs: catch the build docs up with the package split

Review findings: the plan doc claimed no implementation existed, had the
domain dependency order wrong (kernel depends on protocol; device on
kernel + protocol + csv), never placed the three shared contracts, and
named a nonexistent new-driver-checklist.md. Its status now records the
implemented phases (and the deliberate pci-bus-first pilot), the target
shape carries the contract placements and the path-dependency-only
constraint on the kernel package's out-of-root abi export, and the
execution notes describe the post-pilot build for whichever session
runs the remaining waves. README's repo layout gains build-support/
and the packages-note; driver-model, threading, system-requirements,
and the two display plan docs stop citing root build.zig for recipe
facts that now live in build-support.
This commit is contained in:
Daniel Samson
2026-07-26 23:12:14 +01:00
parent 15575960bd
commit 902e4a0a9e
8 changed files with 101 additions and 56 deletions
+10 -1
View File
@@ -263,9 +263,18 @@ test/ → /test the test tree: the QEMU harness (qemu_test.py, h
system/services/ beside the on-image test fixtures — vfs-test/ thread-test/ system/services/ beside the on-image test fixtures — vfs-test/ thread-test/
crash-test/ … — whose repo path IS their boot-volume path crash-test/ … — whose repo path IS their boot-volume path
(/test/system/services/<name>) (/test/system/services/<name>)
build-support/ the danos build API (build-time only, nothing on the image):
the shared user-binary recipe + default-import wiring every
build file consumes (docs/build-packages-plan.md)
tools/ host-side build scripts tools/ host-side build scripts
``` ```
**Builds are packages** (docs/build-packages-plan.md): each `library/` domain owns a
`build.zig`/`build.zig.zon` exporting its modules (with a standalone `zig build test`),
binaries are converting one directory at a time to ~15-line package builds (`pci-bus`
is the first), and the root `build.zig` orchestrates — image assembly, QEMU, the
aggregate test step.
**Wire protocols live in `library/protocol/`**, one module per directory **Wire protocols live in `library/protocol/`**, one module per directory
(`library/protocol/vfs/vfs-protocol.zig` is the `vfs-protocol` module), imported by module (`library/protocol/vfs/vfs-protocol.zig` is the `vfs-protocol` module), imported by module
name. A protocol is the seam between a low-level driver and the higher-level service it name. A protocol is the seam between a low-level driver and the higher-level service it
@@ -320,5 +329,5 @@ exception in [coding-standards.md](coding-standards.md) applies to that seam.
| System services (init, the `fat` filesystem, the device-manager) | `system/services/` | | System services (init, the `fat` filesystem, the device-manager) | `system/services/` |
| Device drivers, one sub-project each (`pci-bus`, `ps2-bus`, `usb-xhci-bus` bus drivers) | `system/drivers/` | | Device drivers, one sub-project each (`pci-bus`, `ps2-bus`, `usb-xhci-bus` bus drivers) | `system/drivers/` |
| On-image test fixtures for the QEMU cases (`vfs-test`, `crash-test`, `thread-test`, …) → `/test/system/services` | `test/system/services/` | | On-image test fixtures for the QEMU cases (`vfs-test`, `crash-test`, `thread-test`, …) → `/test/system/services` | `test/system/services/` |
| Build + `run-x86-64` (QEMU/OVMF) + `release-x86-64` (the flashable ISO) | `build.zig` | | Build orchestration + `run-x86-64` (QEMU/OVMF) + `release-x86-64` (the flashable ISO) | `build.zig` (root; the shared user-binary recipe is `build-support/`, and each `library/` domain + packaged binary carries its own `build.zig`) |
| QEMU integration test harness | `test/qemu_test.py` | | QEMU integration test harness | `test/qemu_test.py` |
+57 -33
View File
@@ -1,10 +1,17 @@
# Plan: packages — hierarchical builds for libraries and binaries # Plan: packages — hierarchical builds for libraries and binaries
**Status: proposed, awaiting sign-off. No implementation yet.** **Status: in progress.** Implemented on branch `claude/build-packages-plan-174144`:
phase 0 (`build-support`), phase 1 (all six library domains as packages, root as
the pilot consumer), and the first phase-2 binary package — `pci-bus`, a
deliberate deviation from wave A's two-small-services opener, because a driver
with per-binary extras (device-manager-protocol, pci-class) exercises the
template harder than a plain service. Every phase landed green (unit tests, the
QEMU suite at parity with main, boot-image file list unchanged). Remaining:
waves B–D of phase 2, then phase 3.
## Why ## Why
`build.zig` is ~1,250 lines and grows by three hand-written stanzas per binary; `build.zig` was ~1,250 lines, growing by three hand-written stanzas per binary;
at a driver per device family that does not scale. More fundamentally: in one at a driver per device family that does not scale. More fundamentally: in one
monolithic build every binary compiles against library *source*, so a library monolithic build every binary compiles against library *source*, so a library
interface break is silently absorbed by whoever edits everything in one commit — interface break is silently absorbed by whoever edits everything in one commit —
@@ -21,18 +28,27 @@ know where anything lives.
## Target shape ## Target shape
``` ```
build-support/ package: the danos build API (userBinary(), targets, default imports) build-support/ package: the danos build API (userBinary(), defaultImports(), targets)
library/kernel/ package "kernel": modules ipc, service, memory, process, logging, time, ... library/kernel/ package "kernel": modules abi, ipc, service, memory, process, logging, time, ... (depends on protocol)
library/device/ package "device": modules driver, pci, usb-abi, model, ... (depends on kernel) library/device/ package "device": modules driver, pci, usb-abi, model, ... (depends on kernel, protocol, csv)
library/protocol/ package "protocol": the wire protocols library/protocol/ package "protocol": the wire protocols
library/client/ package "client" (depends on kernel, protocol) library/client/ package "client" (depends on kernel, protocol)
library/csv/ package "csv" library/csv/ package "csv"
library/xkeyboard-config/ package "xkeyboard-config" library/xkeyboard-config/ package "xkeyboard-config"
system/services/<name>/ one package per binary: ~10-line build.zig + zon system/services/<name>/ one package per binary: ~15-line build.zig + zon
system/drivers/<name>/ one package per binary system/drivers/<name>/ one package per binary
build.zig (root) orchestrator: dependency() per binary, image assembly, QEMU, test steps build.zig (root) orchestrator: dependency() per binary, image assembly, QEMU, test steps
``` ```
The three shared contracts: `boot-handoff` stays a root module (only the
loader↔kernel pair speaks it); `abi` is exported by the kernel package from
`../../system/abi.zig` (the source stays with the kernel; userspace's one view
of it lives in the package, so every consumer names the same module instance);
`device-abi` is exported by device. Reaching outside the package root means the
kernel package is valid only as an in-repo path dependency — it could never be
fetched by hash — which is fine: path dependencies are the only way any of
these packages is consumed.
Rules: Rules:
- **Dependencies are declared at domain level** (a binary's zon names `kernel`, - **Dependencies are declared at domain level** (a binary's zon names `kernel`,
@@ -72,41 +88,47 @@ the freestanding target setup, and the default-import wiring into the
`build-support` package. Root build consumes it; nothing else moves. This is `build-support` package. Root build consumes it; nothing else moves. This is
the cross-cutting-change home, so it lands first. the cross-cutting-change home, so it lands first.
**Phase 1 — library domains become packages.** In dependency order: `kernel` **Phase 1 — library domains become packages.** In dependency order: `protocol`
(no deps) → `csv`, `protocol` → `device`, `client` → `xkeyboard-config`. Each and `csv` (the roots) → `kernel` (depends on protocol: file-system speaks
gets build.zig + zon + a standalone test step. The root build swaps its vfs-protocol) → `device`, `client`; `xkeyboard-config` stands alone. Each gets
`createModule` calls for `b.dependency("<domain>").module("<name>")`. **No build.zig + zon + a standalone test step (client's is empty until its modules
binary moves in this phase** — the root build is the pilot consumer, which grow host tests — kept for uniformity, since the root aggregate depends on
proves the packages without touching 30 binaries. every domain's test step). The root build swaps its `createModule` calls for
`b.dependency("<domain>").module("<name>")`. **No binary moves in this phase**
— the root build is the pilot consumer, which proves the packages without
touching 30 binaries.
**Phase 2 — binaries become packages, in waves.** Wave A: two small services **Phase 2 — binaries become packages, in waves.** The template was shaken out
(e.g. logger, display-demo) to shake out the template. Wave B: remaining by the pci-bus pilot (see Status). Wave A: services. Wave B: the remaining
services. Wave C: drivers. Wave D: test fixtures. Root build shrinks to drivers. Wave C: test fixtures. Root build shrinks to orchestration per wave.
orchestration per wave.
**Phase 3 — root cleanup.** Split what remains of the root build into **Phase 3 — root cleanup.** Split what remains of the root build into
`build/images.zig`, `build/qemu.zig`, imported by a short root `build.zig`. `build/images.zig`, `build/qemu.zig`, imported by a short root `build.zig`.
**Afterwards** (outside this plan): the intel-uhd-graphics-750 driver is **Afterwards** (outside this plan): the intel-uhd-graphics-750 driver is
(re)created as a greenfield package — the new-driver checklist's step 2 gets (re)created as a greenfield package — the "Adding a driver" checklist's build
rewritten against the package template at that point. step (docs/device-driver-development/devices-csv.md, step 1) gets rewritten
against the package template at that point.
## Execution notes (for whichever session runs this) ## Execution notes (for whichever session runs the remaining waves)
Anchors in today's root `build.zig` (~1,250 lines): Anchors in the root `build.zig` as it stands after the pilot:
- `addUserBinary` / `addThreadedUserBinary` / `addUserBinaryImpl` and the - The shared recipe lives in `build-support/build.zig`: `userBinary` (domains
`default_imports` plumbing start around line 63 — this is what phase 0 form, what binary packages call), `userBinaryFromImports` (the underlying
extracts into `build-support`. `addUserBinaryImpl` also wires the `start` recipe root's stanzas still use), and `defaultImports` — the ONE list of
root shim from `default_imports`; that trick must survive the move. default modules; root and the packages both draw from it. The `start` root
- `programModule(<exe>).addImport(...)` calls (search `programModule`) are the shim and `user.ld` are named through the kernel package (Dependency.path).
per-binary extra imports — the data for each binary's future build.zig. - `programModule(<exe>).addImport(...)` calls in root (search `programModule`)
are the per-binary extra imports — the data for each binary's future
build.zig. `system/drivers/pci-bus/build.zig` is the template to copy.
- The boot-tree array (search `"etc/init.csv"` or `.getEmittedBin()`) is the - The boot-tree array (search `"etc/init.csv"` or `.getEmittedBin()`) is the
image file list — the authoritative before/after comparison target. image file list — the authoritative before/after comparison target. A
- Protocol/module definitions (search `createModule`) map module names to converted binary's stanza becomes
`library/...` source paths — the data for each domain package's exports. `b.dependency("<name>", .{}).artifact("<name>")` plus a zon entry.
- The QEMU size-check test hardcodes source paths (search - The QEMU size-check tests hardcode source paths (search
`virtio-gpu-protocol.zig` near line 1149) — moves with phase 2 wave C. `virtio-gpu-protocol.zig` in the root test list) — they move with their
binaries' waves.
Verification per phase: Verification per phase:
@@ -128,5 +150,7 @@ docs/coding-standards.md (kebab-case names, no abbreviations), and the
upgrade lands separately, never mid-phase. upgrade lands separately, never mid-phase.
- The QEMU size-check tests hardcode source paths (e.g. virtio-gpu protocol - The QEMU size-check tests hardcode source paths (e.g. virtio-gpu protocol
struct sizes in the root build) — phase 2 wave C must carry those along. struct sizes in the root build) — phase 2 wave C must carry those along.
- Doc updates ride each phase: drivers.md, new-driver-checklist.md, and - Doc updates ride each phase: docs/README.md (repo layout + source map),
docs/README.md reference build steps that will change shape. docs/device-driver-development/devices-csv.md ("Adding a driver", step 1),
and the docs that cite the build recipe (driver-model.md, threading.md,
system-requirements.md) reference build shapes that keep changing.
@@ -20,9 +20,10 @@ Read [display.md](display.md) first for the *why*; this is the *what* and the *o
Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations in Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations in
full, kebab-case file names, no `Co-Authored-By` trailers on commits. New user binaries full, kebab-case file names, no `Co-Authored-By` trailers on commits. New user binaries
go through `addUserBinary` in [build.zig](../../build.zig) and get packed into the go through build-support's shared user-binary recipe and get packed into the
initial-ramdisk; protocols are `b.addModule("…-protocol", …)` and imported into the initial-ramdisk; protocols are modules exported by the `library/protocol` package.
`runtime` module. (This section predates the build-packages split; see
[build-packages-plan.md](../build-packages-plan.md) for the current build shape.)
## How to verify along the way ## How to verify along the way
@@ -18,9 +18,11 @@ lands on its own and ends in a **verifiable gate** — shaped for a `/loop` run,
Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations, Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations,
kebab-case file names, no `Co-Authored-By` trailers. New user binaries go through kebab-case file names, no `Co-Authored-By` trailers. New user binaries go through
`addUserBinary` and get packed into the initial-ramdisk; protocols are build-support's shared user-binary recipe and get packed into the initial-ramdisk;
`b.addModule("…-protocol", …)` imported into `runtime`; new syscalls extend protocols are modules exported by the `library/protocol` package; new syscalls extend
[abi.zig](../../system/abi.zig) `SystemCall` + a `library/runtime` wrapper. [abi.zig](../../system/abi.zig) `SystemCall` + a `library/kernel` wrapper.
(This section predates the build-packages split; see
[build-packages-plan.md](../build-packages-plan.md) for the current build shape.)
## How to verify along the way ## How to verify along the way
+11 -7
View File
@@ -138,13 +138,17 @@ a higher-level service (block ↔ filesystem, a scanout driver ↔ the composito
private wire to its *hardware* — virtio-gpu's command set — is not that; it stays a private wire to its *hardware* — virtio-gpu's command set — is not that; it stays a
driver-private file, like the virtio-pci transport beside it. driver-private file, like the virtio-pci transport beside it.
The build side of this has since landed: [`addUserBinary`](build.zig) injects the The build side of this has since landed: the shared recipe in
default modules — the library/kernel concern modules (`ipc`, `memory`, `process`, `time`, [`build-support/build.zig`](../../build-support/build.zig) (`defaultImports` +
`logging`, `file-system`, `thread`, `service`), the device/service clients (`driver`, `userBinary`) injects the default modules — the library/kernel concern modules (`ipc`,
`block`, `display`, `input`), plus `mmio`, `xkeyboard-config`, `acpi-ids` — into every user `memory`, `process`, `time`, `logging`, `file-system`, `thread`, `service`), the
binary, and per-binary extras — protocol modules, bus logic — are added with device/service clients (`driver`, `block`, `display`, `input`), plus `mmio`,
`programModule(exe).addImport(...)`. That's the *entire* mechanism — Zig modules `xkeyboard-config`, `acpi-ids` — into every user binary, and per-binary extras —
already give you everything else. protocol modules, bus logic — are added with `programModule(exe).addImport(...)`.
Most binaries are still built by the root `build.zig`'s stanzas through that recipe;
a binary can instead own a package with its own ~15-line `build.zig` (pci-bus is the
first — see [build-packages-plan.md](../build-packages-plan.md)). That's the *entire*
mechanism — Zig modules already give you everything else.
The discipline that makes this work: **a class driver must not import a bus's *hardware* The discipline that makes this work: **a class driver must not import a bus's *hardware*
logic module.** `usb-hid` imports `usb` (the transfer client) and `input-protocol`, never logic module.** `usb-hid` imports `usb` (the transfer client) and `input-protocol`, never
+8 -6
View File
@@ -61,9 +61,10 @@ runtime — rebuilt in lockstep — knows the mapping.
backend would either bake danos syscall numbers into std (breaking ABI privacy and backend would either bake danos syscall numbers into std (breaking ABI privacy and
renumbering) or fork std to route back through the runtime — a permanent rebase renumbering) or fork std to route back through the runtime — a permanent rebase
cost that buys nothing the native type doesn't. cost that buys nothing the native type doesn't.
2. **Our user binaries are built `single_threaded = true`** ([build.zig](../../build.zig) 2. **Our user binaries are built `single_threaded = true`** (the shared recipe in
`addUserBinary`), which compiles threading out entirely and makes atomics and TLS [build-support/build.zig](../../build-support/build.zig)), which compiles threading
single-threaded. Threads need this flipped per binary regardless. out entirely and makes atomics and TLS single-threaded. Threads need this flipped
per binary regardless.
So we take the *shape* of `std.Thread`, not the *type*. The cost of replicating the So we take the *shape* of `std.Thread`, not the *type*. The cost of replicating the
surface (spawn/join/Mutex/Condition) is small; the cost of the std type is the ABI surface (spawn/join/Mutex/Condition) is small; the cost of the std type is the ABI
@@ -236,9 +237,10 @@ see the intro). Two scoped pieces, as built:
### Build: multi-threaded codegen, opt-in ### Build: multi-threaded codegen, opt-in
A binary opts in by being added with `addThreadedUserBinary` — as `addUserBinary`, A binary opts in with `addThreadedUserBinary` in the root `build.zig` (or
but the shared implementation builds it `single_threaded = false` — so atomics and `.threaded = true` in a binary package's `build_support.userBinary` call) — the
(later) TLS are real. Threads and atomics are unsound in a `single_threaded` image, shared recipe in build-support then builds it `single_threaded = false` — so atomics
and (later) TLS are real. Threads and atomics are unsound in a `single_threaded` image,
so a binary must opt in **before** it may call `Thread.spawn`. Everyone else so a binary must opt in **before** it may call `Thread.spawn`. Everyone else
stays single-threaded and lean. stays single-threaded and lean.
+2 -2
View File
@@ -95,9 +95,9 @@ hypervisor configured for UEFI firmware and an xHCI USB controller.
| Requirement | Detail | Source | | Requirement | Detail | Source |
|---|---|---| |---|---|---|
| **x86-64, 64-bit only** | Kernel and loader are built exclusively for `x86_64`; the loader rejects any non-x86-64 kernel ELF (`error.WrongArchitecture`). | `build.zig:481`, `boot/efi.zig:622` | | **x86-64, 64-bit only** | Kernel and loader are built exclusively for `x86_64`; the loader rejects any non-x86-64 kernel ELF (`error.WrongArchitecture`). | `build-support/build.zig` (`freestandingTarget`), `boot/efi.zig:622` |
| **Long mode + PAE + NX** | AP trampoline sets `CR4.PAE`, `EFER.LME`, `EFER.NXE`; NX is used in kernel page-table entries. | `system/kernel/architecture/x86_64/trampoline.s:62` | | **Long mode + PAE + NX** | AP trampoline sets `CR4.PAE`, `EFER.LME`, `EFER.NXE`; NX is used in kernel page-table entries. | `system/kernel/architecture/x86_64/trampoline.s:62` |
| **SSE / SSE2** | Baseline: the compiler emits SSE for ordinary struct copies. Trampoline enables `CR4.OSFXSR` + `OSXMMEXCPT` and clears `CR0.EM`. | `build.zig:477`, `trampoline.s:62` | | **SSE / SSE2** | Baseline: the compiler emits SSE for ordinary struct copies. Trampoline enables `CR4.OSFXSR` + `OSXMMEXCPT` and clears `CR0.EM`. | `build-support/build.zig` (`freestandingTarget`), `trampoline.s:62` |
| **`syscall` / `sysret`** | Primary user↔kernel entry path. `EFER.SCE` enabled; `STAR`/`LSTAR`/`SFMASK` programmed per core. (`int 0x80` exists as a parallel gate.) | `architecture/x86_64/per-cpu.zig:71`, `isr.s:196` | | **`syscall` / `sysret`** | Primary user↔kernel entry path. `EFER.SCE` enabled; `STAR`/`LSTAR`/`SFMASK` programmed per core. (`int 0x80` exists as a parallel gate.) | `architecture/x86_64/per-cpu.zig:71`, `isr.s:196` |
| **Local APIC (xAPIC)** | LAPIC accessed via MMIO at `0xFEE00000`. LAPIC ID read as a `u8` — classic xAPIC. **x2APIC is not supported** (no MSR path). | `apic.zig:67`, `apic.zig:646` | | **Local APIC (xAPIC)** | LAPIC accessed via MMIO at `0xFEE00000`. LAPIC ID read as a `u8` — classic xAPIC. **x2APIC is not supported** (no MSR path). | `apic.zig:67`, `apic.zig:646` |
| **CPUID + RDTSC** | CPUID leaf `0x15` for TSC frequency; RDTSC is the monotonic clock. | `apic.zig:333`, `apic.zig:113` | | **CPUID + RDTSC** | CPUID leaf `0x15` for TSC frequency; RDTSC is the monotonic clock. | `apic.zig:333`, `apic.zig:113` |
+4 -1
View File
@@ -6,7 +6,10 @@
//! This package also exports `abi` — the kernel <-> user contract (SystemCall //! This package also exports `abi` — the kernel <-> user contract (SystemCall
//! numbers, mmap prot flags, page_size). Its source lives with the kernel in //! numbers, mmap prot flags, page_size). Its source lives with the kernel in
//! system/abi.zig, outside this directory, but userspace's one view of it is //! system/abi.zig, outside this directory, but userspace's one view of it is
//! exported here so every consumer names the same module instance. //! exported here so every consumer names the same module instance. Reaching
//! outside the package root means this package is valid only as an in-repo
//! path dependency (never fetchable by hash) — fine, since path dependencies
//! are the only way danos packages are consumed.
//! //!
//! The root shim (root.zig) and the user link script (user.ld) are plain //! The root shim (root.zig) and the user link script (user.ld) are plain
//! files, not modules; build-support reaches them through this package's //! files, not modules; build-support reaches them through this package's