danos/docs/build-packages-plan.md

# Plan: packages — hierarchical builds for libraries and binaries

**Status: complete** (branch `claude/build-packages-plan-174144`). Phase 0
(`build-support`), phase 1 (all six library domains), phase 2 (every binary —
the pci-bus pilot first, then services, drivers, and test fixtures in waves;
multi-binary directories like ps2-bus and usb-hid are one package exporting
several artifacts, and the acpi/fdt discovery pair each export an artifact
named "discovery" that the root's -Ddiscovery picks between), and phase 3 (the
root split into `build/images.zig` + `build/qemu.zig`; the root `build.zig` is
~460 lines of orchestration, down from ~1,250). Every phase landed green: unit
tests, the QEMU suite at parity with main, boot-image file list unchanged.
The `lazyDependency` payoff (What-this-buys #4) is in too: the /test fixtures
and the unselected discovery package are lazy — a build loads and compiles
only what it ships. And imports are exact: the pre-wired default set is gone;
every binary names precisely the modules its source imports and carries only
those domains in its manifest (rule 1 below).

## Why

`build.zig` was ~1,250 lines, growing by three hand-written stanzas per binary;
at a driver per device family that does not scale. More fundamentally: in one
monolithic build every binary compiles against library *source*, so a library
interface break is silently absorbed by whoever edits everything in one commit —
the interface never has to be honest. danos is about isolation; the build should
mirror it.

A **package** here is a build-time unit only — a directory owning a `build.zig`
(recipe: what it exports, how to test it) and a `build.zig.zon` (manifest: name
+ dependencies). Binaries remain fully static freestanding ELFs; packages change
who declares what, not what links to what. Source code is untouched: `@import`
uses module names (`"pci"`, `"service"`) exactly as today — only build files
know where anything lives.

## Target shape

```
build-support/                 package: the danos build API (userBinary(), defaultImports(), targets)
library/kernel/                package "kernel":   modules abi, ipc, service, memory, process, logging, time, ...   (depends on protocol)
library/device/                package "device":   modules driver, pci, usb-abi, model, ...   (depends on kernel, protocol, csv)
library/protocol/              package "protocol": the wire protocols
library/client/                package "client"    (depends on kernel, protocol)
library/csv/                   package "csv"
library/xkeyboard-config/      package "xkeyboard-config"
system/services/<name>/        one package per binary: ~15-line build.zig + zon
system/drivers/<name>/         one package per binary
build.zig (root)               orchestrator: dependency() per binary, image assembly, QEMU, test steps
```

The three shared contracts: `boot-handoff` stays a root module (only the
loader↔kernel pair speaks it); `abi` is exported by the kernel package from
`../../system/abi.zig` (the source stays with the kernel; userspace's one view
of it lives in the package, so every consumer names the same module instance);
`device-abi` is exported by device. Reaching outside the package root means the
kernel package is valid only as an in-repo path dependency — it could never be
fetched by hash — which is fine: path dependencies are the only way any of
these packages is consumed.

Rules:

- **Imports are exact and per binary.** A binary's build.zig names precisely
  the modules its source `@import`s — the moral equivalent of a C file's
  include list — and its zon names only the domains those modules come from
  (plus `build-support` and `kernel`, which is implicit in every binary: the
  root shim and user link script live there). Nothing is pre-wired: an
  undeclared `@import` is a compile error, and build-support resolves each
  name by searching the packages the zon declares — the domains' own
  addModule exports are the single statement of who owns what, with no name
  table anywhere to drift. Availability
  never meant bloat — Zig only compiles what a program actually imports — but
  exactness makes the declared interface honest and machine-checked.
- **Modules export source, not artifacts** — each consumer compiles libraries
  with its own flags, so per-binary optimization choices keep working; Zig's
  cache deduplicates.
- **Zon paths are relative and that is accepted.** Binaries sit exactly three
  levels deep, so the `../../../` prefix is a constant idiom; a library-domain
  move is a rare, already-breaking event fixed by one sed across manifests, and
  a stale path fails loudly before anything compiles.
- **Cross-cutting build changes live in `build-support` only** — that is the
  contract that keeps per-binary build files declarative.

## What this buys

1. Library interfaces become machine-checked: a consumer can only import what
   it declared — per binary, down to the single module — and each domain's zon
   declares what it needs (claim-before-touch, applied to source). A keyboard
   driver carries `xkeyboard-config` in its manifest; nothing else does.
2. Each library domain gets a standalone `zig build test` — runtime-library
   stability testing in isolation.
3. Adding a binary = adding a directory (source + two small files), not editing
   three places in a 1,250-line file.
4. `lazyDependency` lets an image target build only what it ships: the /test
   fixtures resolve only under -Dtest-case, and only the -Ddiscovery-selected
   discovery package ever loads.

## Phases

Each phase ends green: `zig build test` passes (88/88 QEMU) and the boot
image's file list is unchanged. Byte-identical binaries are expected but not
required (module reorganization can perturb symbol order); file list is the
hard gate.

**Phase 0 — `build-support`.** Extract `addUserBinary`/`addThreadedUserBinary`,
the freestanding target setup, and the default-import wiring into the
`build-support` package. Root build consumes it; nothing else moves. This is
the cross-cutting-change home, so it lands first.

**Phase 1 — library domains become packages.** In dependency order: `protocol`
and `csv` (the roots) → `kernel` (depends on protocol: file-system speaks
vfs-protocol) → `device`, `client`; `xkeyboard-config` stands alone. Each gets
build.zig + zon + a standalone test step (client's is empty until its modules
grow host tests — kept for uniformity, since the root aggregate depends on
every domain's test step). The root build swaps its `createModule` calls for
`b.dependency("<domain>").module("<name>")`. **No binary moves in this phase**
— the root build is the pilot consumer, which proves the packages without
touching 30 binaries.

**Phase 2 — binaries become packages, in waves.** The template was shaken out
by the pci-bus pilot (see Status). Wave A: services (done). Wave B: the
remaining drivers (done). Wave C: test fixtures (done). Root build shrank to
orchestration per wave. init's `-Dserial` heartbeat flag rides a dependency
option; a directory with several binaries (ps2-bus, usb-hid) is one package
exporting several artifacts.

**Phase 3 — root cleanup (done).** What remained of the root build split into
`build/images.zig` (the FHS install tree, boot manifest + capsule, FAT32
images, release ISO, check steps) and `build/qemu.zig` (the run steps + OVMF
probing), imported by a short root `build.zig`.

**Afterwards** (outside this plan): the intel-uhd-graphics-750 driver is
(re)created as a greenfield package. The new-driver checklist's build step
(docs/device-driver-development/new-driver-checklist.md, step 2) is already
rewritten against the package template.

## Execution notes (the finished shape)

- The shared recipe lives in `build-support/build.zig`: `userBinary` (what
  every binary package calls; each named import resolves by searching the
  packages the binary's zon declares) and `programModule` (for per-binary
  addOptions modules). The `start` root shim and `user.ld` are named through the kernel
  package (Dependency.path).
- Adding a binary = adding a directory with source + a ~15-line build.zig +
  zon (copy any existing binary package, e.g.
  `system/drivers/pci-bus/build.zig`) listing exactly the modules the source
  imports and the domains they come from, then one dependency + one bundled
  entry in the root build.zig and one zon line.
- The boot-tree array in the root (search `"etc/init.csv"` or
  `.getEmittedBin()`) is the image file list — the authoritative comparison
  target for any future build change.
- Package unit tests live in each package's own `test` step; the root
  aggregate depends on every test-bearing package's step, so `zig build test`
  at the root still runs everything.

Verification per phase:

- Unit tests: `zig build test`.
- QEMU integration suite: `python3 test/qemu_test.py` (docs/testing.md; the
  full suite, all cases must pass).
- Image file list: the boot-tree array is the source of truth — snapshot it
  (paths only) before phase 0 and diff after each phase; `zig build
  check-fat-image` must also stay green.

Context a fresh session should read first: this doc, docs/testing.md,
docs/coding-standards.md (kebab-case names, no abbreviations), and the
`userBinary`/`userBinaryFromImports` bodies in build-support/build.zig. Commit
style: no Co-Authored-By trailers.

## Risks / notes

- Zig version churn: the package API (`b.dependency`, zon schema) has moved
  between releases; the work pins against the repo's current Zig and any
  upgrade lands separately, never mid-phase.
- The QEMU size-check tests hardcode source paths (e.g. virtio-gpu protocol
  struct sizes) — they moved into their binaries' packages with their waves,
  discharging the carry-along obligation.
- Doc updates ride each phase: docs/README.md (repo layout + source map),
  docs/device-driver-development/new-driver-checklist.md (step 2) and
  devices-csv.md ("Adding a driver"), and the docs that cite the build recipe
  (driver-model.md, threading.md, system-requirements.md) reference build
  shapes that keep changing.