Daniel Samson
902e4a0a9e
Review findings: the plan doc claimed no implementation existed, had the domain dependency order wrong (kernel depends on protocol; device on kernel + protocol + csv), never placed the three shared contracts, and named a nonexistent new-driver-checklist.md. Its status now records the implemented phases (and the deliberate pci-bus-first pilot), the target shape carries the contract placements and the path-dependency-only constraint on the kernel package's out-of-root abi export, and the execution notes describe the post-pilot build for whichever session runs the remaining waves. README's repo layout gains build-support/ and the packages-note; driver-model, threading, system-requirements, and the two display plan docs stop citing root build.zig for recipe facts that now live in build-support.
157 lines
8.4 KiB
Markdown
157 lines
8.4 KiB
Markdown
# Plan: packages — hierarchical builds for libraries and binaries
|
||
|
||
**Status: in progress.** Implemented on branch `claude/build-packages-plan-174144`:
|
||
phase 0 (`build-support`), phase 1 (all six library domains as packages, root as
|
||
the pilot consumer), and the first phase-2 binary package — `pci-bus`, a
|
||
deliberate deviation from wave A's two-small-services opener, because a driver
|
||
with per-binary extras (device-manager-protocol, pci-class) exercises the
|
||
template harder than a plain service. Every phase landed green (unit tests, the
|
||
QEMU suite at parity with main, boot-image file list unchanged). Remaining:
|
||
waves B–D of phase 2, then phase 3.
|
||
|
||
## Why
|
||
|
||
`build.zig` was ~1,250 lines, growing by three hand-written stanzas per binary;
|
||
at a driver per device family that does not scale. More fundamentally: in one
|
||
monolithic build every binary compiles against library *source*, so a library
|
||
interface break is silently absorbed by whoever edits everything in one commit —
|
||
the interface never has to be honest. danos is about isolation; the build should
|
||
mirror it.
|
||
|
||
A **package** here is a build-time unit only — a directory owning a `build.zig`
|
||
(recipe: what it exports, how to test it) and a `build.zig.zon` (manifest: name
|
||
+ dependencies). Binaries remain fully static freestanding ELFs; packages change
|
||
who declares what, not what links to what. Source code is untouched: `@import`
|
||
uses module names (`"pci"`, `"service"`) exactly as today — only build files
|
||
know where anything lives.
|
||
|
||
## Target shape
|
||
|
||
```
|
||
build-support/ package: the danos build API (userBinary(), defaultImports(), targets)
|
||
library/kernel/ package "kernel": modules abi, ipc, service, memory, process, logging, time, ... (depends on protocol)
|
||
library/device/ package "device": modules driver, pci, usb-abi, model, ... (depends on kernel, protocol, csv)
|
||
library/protocol/ package "protocol": the wire protocols
|
||
library/client/ package "client" (depends on kernel, protocol)
|
||
library/csv/ package "csv"
|
||
library/xkeyboard-config/ package "xkeyboard-config"
|
||
system/services/<name>/ one package per binary: ~15-line build.zig + zon
|
||
system/drivers/<name>/ one package per binary
|
||
build.zig (root) orchestrator: dependency() per binary, image assembly, QEMU, test steps
|
||
```
|
||
|
||
The three shared contracts: `boot-handoff` stays a root module (only the
|
||
loader↔kernel pair speaks it); `abi` is exported by the kernel package from
|
||
`../../system/abi.zig` (the source stays with the kernel; userspace's one view
|
||
of it lives in the package, so every consumer names the same module instance);
|
||
`device-abi` is exported by device. Reaching outside the package root means the
|
||
kernel package is valid only as an in-repo path dependency — it could never be
|
||
fetched by hash — which is fine: path dependencies are the only way any of
|
||
these packages is consumed.
|
||
|
||
Rules:
|
||
|
||
- **Dependencies are declared at domain level** (a binary's zon names `kernel`,
|
||
`device`), **imports stay module-level** (`@import("pci")`). `build-support`
|
||
pre-wires the core set every binary uses (service, ipc, memory, process,
|
||
logging); per-binary build files name only extras.
|
||
- **Modules export source, not artifacts** — each consumer compiles libraries
|
||
with its own flags, so per-binary optimization choices keep working; Zig's
|
||
cache deduplicates.
|
||
- **Zon paths are relative and that is accepted.** Binaries sit exactly three
|
||
levels deep, so the `../../../` prefix is a constant idiom; a library-domain
|
||
move is a rare, already-breaking event fixed by one sed across manifests, and
|
||
a stale path fails loudly before anything compiles.
|
||
- **Cross-cutting build changes live in `build-support` only** — that is the
|
||
contract that keeps per-binary build files declarative.
|
||
|
||
## What this buys
|
||
|
||
1. Library interfaces become machine-checked: a consumer can only import what a
|
||
domain exports, and each domain's zon declares what it needs (claim-before-
|
||
touch, applied to source).
|
||
2. Each library domain gets a standalone `zig build test` — runtime-library
|
||
stability testing in isolation.
|
||
3. Adding a binary = adding a directory (source + two small files), not editing
|
||
three places in a 1,250-line file.
|
||
4. Later: `lazyDependency` lets an image target build only what it ships.
|
||
|
||
## Phases
|
||
|
||
Each phase ends green: `zig build test` passes (88/88 QEMU) and the boot
|
||
image's file list is unchanged. Byte-identical binaries are expected but not
|
||
required (module reorganization can perturb symbol order); file list is the
|
||
hard gate.
|
||
|
||
**Phase 0 — `build-support`.** Extract `addUserBinary`/`addThreadedUserBinary`,
|
||
the freestanding target setup, and the default-import wiring into the
|
||
`build-support` package. Root build consumes it; nothing else moves. This is
|
||
the cross-cutting-change home, so it lands first.
|
||
|
||
**Phase 1 — library domains become packages.** In dependency order: `protocol`
|
||
and `csv` (the roots) → `kernel` (depends on protocol: file-system speaks
|
||
vfs-protocol) → `device`, `client`; `xkeyboard-config` stands alone. Each gets
|
||
build.zig + zon + a standalone test step (client's is empty until its modules
|
||
grow host tests — kept for uniformity, since the root aggregate depends on
|
||
every domain's test step). The root build swaps its `createModule` calls for
|
||
`b.dependency("<domain>").module("<name>")`. **No binary moves in this phase**
|
||
— the root build is the pilot consumer, which proves the packages without
|
||
touching 30 binaries.
|
||
|
||
**Phase 2 — binaries become packages, in waves.** The template was shaken out
|
||
by the pci-bus pilot (see Status). Wave A: services. Wave B: the remaining
|
||
drivers. Wave C: test fixtures. Root build shrinks to orchestration per wave.
|
||
|
||
**Phase 3 — root cleanup.** Split what remains of the root build into
|
||
`build/images.zig`, `build/qemu.zig`, imported by a short root `build.zig`.
|
||
|
||
**Afterwards** (outside this plan): the intel-uhd-graphics-750 driver is
|
||
(re)created as a greenfield package — the "Adding a driver" checklist's build
|
||
step (docs/device-driver-development/devices-csv.md, step 1) gets rewritten
|
||
against the package template at that point.
|
||
|
||
## Execution notes (for whichever session runs the remaining waves)
|
||
|
||
Anchors in the root `build.zig` as it stands after the pilot:
|
||
|
||
- The shared recipe lives in `build-support/build.zig`: `userBinary` (domains
|
||
form, what binary packages call), `userBinaryFromImports` (the underlying
|
||
recipe root's stanzas still use), and `defaultImports` — the ONE list of
|
||
default modules; root and the packages both draw from it. The `start` root
|
||
shim and `user.ld` are named through the kernel package (Dependency.path).
|
||
- `programModule(<exe>).addImport(...)` calls in root (search `programModule`)
|
||
are the per-binary extra imports — the data for each binary's future
|
||
build.zig. `system/drivers/pci-bus/build.zig` is the template to copy.
|
||
- The boot-tree array (search `"etc/init.csv"` or `.getEmittedBin()`) is the
|
||
image file list — the authoritative before/after comparison target. A
|
||
converted binary's stanza becomes
|
||
`b.dependency("<name>", .{}).artifact("<name>")` plus a zon entry.
|
||
- The QEMU size-check tests hardcode source paths (search
|
||
`virtio-gpu-protocol.zig` in the root test list) — they move with their
|
||
binaries' waves.
|
||
|
||
Verification per phase:
|
||
|
||
- Unit tests: `zig build test`.
|
||
- QEMU integration suite: `python3 test/qemu_test.py` (docs/testing.md; the
|
||
full suite, all cases must pass).
|
||
- Image file list: the boot-tree array is the source of truth — snapshot it
|
||
(paths only) before phase 0 and diff after each phase; `zig build
|
||
check-fat-image` must also stay green.
|
||
|
||
Context a fresh session should read first: this doc, docs/testing.md,
|
||
docs/coding-standards.md (kebab-case names, no abbreviations), and the
|
||
`addUserBinaryImpl` body. Commit style: no Co-Authored-By trailers.
|
||
|
||
## Risks / notes
|
||
|
||
- Zig version churn: the package API (`b.dependency`, zon schema) has moved
|
||
between releases; the work pins against the repo's current Zig and any
|
||
upgrade lands separately, never mid-phase.
|
||
- The QEMU size-check tests hardcode source paths (e.g. virtio-gpu protocol
|
||
struct sizes in the root build) — phase 2 wave C must carry those along.
|
||
- Doc updates ride each phase: docs/README.md (repo layout + source map),
|
||
docs/device-driver-development/devices-csv.md ("Adding a driver", step 1),
|
||
and the docs that cite the build recipe (driver-model.md, threading.md,
|
||
system-requirements.md) reference build shapes that keep changing.
|