danos/docs/build-packages-plan.md

Plan: packages — hierarchical builds for libraries and binaries

Status: in progress. Implemented on branch claude/build-packages-plan-174144: phase 0 (build-support), phase 1 (all six library domains as packages, root as the pilot consumer), and the first phase-2 binary package — pci-bus, a deliberate deviation from wave A's two-small-services opener, because a driver with per-binary extras (device-manager-protocol, pci-class) exercises the template harder than a plain service. Every phase landed green (unit tests, the QEMU suite at parity with main, boot-image file list unchanged). Remaining: waves B–D of phase 2, then phase 3.

Why

build.zig was ~1,250 lines, growing by three hand-written stanzas per binary; at a driver per device family that does not scale. More fundamentally: in one monolithic build every binary compiles against library source, so a library interface break is silently absorbed by whoever edits everything in one commit — the interface never has to be honest. danos is about isolation; the build should mirror it.

A package here is a build-time unit only — a directory owning a build.zig (recipe: what it exports, how to test it) and a build.zig.zon (manifest: name

• dependencies). Binaries remain fully static freestanding ELFs; packages change who declares what, not what links to what. Source code is untouched: @import uses module names ("pci", "service") exactly as today — only build files know where anything lives.

Target shape

build-support/                 package: the danos build API (userBinary(), defaultImports(), targets)
library/kernel/                package "kernel":   modules abi, ipc, service, memory, process, logging, time, ...   (depends on protocol)
library/device/                package "device":   modules driver, pci, usb-abi, model, ...   (depends on kernel, protocol, csv)
library/protocol/              package "protocol": the wire protocols
library/client/                package "client"    (depends on kernel, protocol)
library/csv/                   package "csv"
library/xkeyboard-config/      package "xkeyboard-config"
system/services/<name>/        one package per binary: ~15-line build.zig + zon
system/drivers/<name>/         one package per binary
build.zig (root)               orchestrator: dependency() per binary, image assembly, QEMU, test steps

The three shared contracts: boot-handoff stays a root module (only the loader↔kernel pair speaks it); abi is exported by the kernel package from ../../system/abi.zig (the source stays with the kernel; userspace's one view of it lives in the package, so every consumer names the same module instance); device-abi is exported by device. Reaching outside the package root means the kernel package is valid only as an in-repo path dependency — it could never be fetched by hash — which is fine: path dependencies are the only way any of these packages is consumed.

Rules:

• Dependencies are declared at domain level (a binary's zon names kernel, device), imports stay module-level (@import("pci")). build-support pre-wires the core set every binary uses (service, ipc, memory, process, logging); per-binary build files name only extras.
• Modules export source, not artifacts — each consumer compiles libraries with its own flags, so per-binary optimization choices keep working; Zig's cache deduplicates.
• Zon paths are relative and that is accepted. Binaries sit exactly three levels deep, so the ../../../ prefix is a constant idiom; a library-domain move is a rare, already-breaking event fixed by one sed across manifests, and a stale path fails loudly before anything compiles.
• Cross-cutting build changes live in build-support only — that is the contract that keeps per-binary build files declarative.

What this buys

1. Library interfaces become machine-checked: a consumer can only import what a domain exports, and each domain's zon declares what it needs (claim-before- touch, applied to source).
2. Each library domain gets a standalone zig build test — runtime-library stability testing in isolation.
3. Adding a binary = adding a directory (source + two small files), not editing three places in a 1,250-line file.
4. Later: lazyDependency lets an image target build only what it ships.

Phases

Each phase ends green: zig build test passes (88/88 QEMU) and the boot image's file list is unchanged. Byte-identical binaries are expected but not required (module reorganization can perturb symbol order); file list is the hard gate.

Phase 0 — build-support. Extract addUserBinary/addThreadedUserBinary, the freestanding target setup, and the default-import wiring into the build-support package. Root build consumes it; nothing else moves. This is the cross-cutting-change home, so it lands first.

Phase 1 — library domains become packages. In dependency order: protocol and csv (the roots) → kernel (depends on protocol: file-system speaks vfs-protocol) → device, client; xkeyboard-config stands alone. Each gets build.zig + zon + a standalone test step (client's is empty until its modules grow host tests — kept for uniformity, since the root aggregate depends on every domain's test step). The root build swaps its createModule calls for b.dependency("<domain>").module("<name>"). No binary moves in this phase — the root build is the pilot consumer, which proves the packages without touching 30 binaries.

Phase 2 — binaries become packages, in waves. The template was shaken out by the pci-bus pilot (see Status). Wave A: services. Wave B: the remaining drivers. Wave C: test fixtures. Root build shrinks to orchestration per wave.

Phase 3 — root cleanup. Split what remains of the root build into build/images.zig, build/qemu.zig, imported by a short root build.zig.

Afterwards (outside this plan): the intel-uhd-graphics-750 driver is (re)created as a greenfield package — the "Adding a driver" checklist's build step (docs/device-driver-development/devices-csv.md, step 1) gets rewritten against the package template at that point.

Execution notes (for whichever session runs the remaining waves)

Anchors in the root build.zig as it stands after the pilot:

• The shared recipe lives in build-support/build.zig: userBinary (domains form, what binary packages call), userBinaryFromImports (the underlying recipe root's stanzas still use), and defaultImports — the ONE list of default modules; root and the packages both draw from it. The start root shim and user.ld are named through the kernel package (Dependency.path).
• programModule(<exe>).addImport(...) calls in root (search programModule) are the per-binary extra imports — the data for each binary's future build.zig. system/drivers/pci-bus/build.zig is the template to copy.
• The boot-tree array (search "etc/init.csv" or .getEmittedBin()) is the image file list — the authoritative before/after comparison target. A converted binary's stanza becomes b.dependency("<name>", .{}).artifact("<name>") plus a zon entry.
• The QEMU size-check tests hardcode source paths (search virtio-gpu-protocol.zig in the root test list) — they move with their binaries' waves.

Verification per phase:

• Unit tests: zig build test.
• QEMU integration suite: python3 test/qemu_test.py (docs/testing.md; the full suite, all cases must pass).
• Image file list: the boot-tree array is the source of truth — snapshot it (paths only) before phase 0 and diff after each phase; zig build check-fat-image must also stay green.

Context a fresh session should read first: this doc, docs/testing.md, docs/coding-standards.md (kebab-case names, no abbreviations), and the addUserBinaryImpl body. Commit style: no Co-Authored-By trailers.

Risks / notes

• Zig version churn: the package API (b.dependency, zon schema) has moved between releases; the work pins against the repo's current Zig and any upgrade lands separately, never mid-phase.
• The QEMU size-check tests hardcode source paths (e.g. virtio-gpu protocol struct sizes in the root build) — phase 2 wave C must carry those along.
• Doc updates ride each phase: docs/README.md (repo layout + source map), docs/device-driver-development/devices-csv.md ("Adding a driver", step 1), and the docs that cite the build recipe (driver-model.md, threading.md, system-requirements.md) reference build shapes that keep changing.