From 902e4a0a9e927e5d02b9da52d0119116ee9518cb Mon Sep 17 00:00:00 2001 From: Daniel Samson <12231216+daniel-samson@users.noreply.github.com> Date: Sun, 26 Jul 2026 23:12:14 +0100 Subject: [PATCH] docs: catch the build docs up with the package split Review findings: the plan doc claimed no implementation existed, had the domain dependency order wrong (kernel depends on protocol; device on kernel + protocol + csv), never placed the three shared contracts, and named a nonexistent new-driver-checklist.md. Its status now records the implemented phases (and the deliberate pci-bus-first pilot), the target shape carries the contract placements and the path-dependency-only constraint on the kernel package's out-of-root abi export, and the execution notes describe the post-pilot build for whichever session runs the remaining waves. README's repo layout gains build-support/ and the packages-note; driver-model, threading, system-requirements, and the two display plan docs stop citing root build.zig for recipe facts that now live in build-support. --- docs/README.md | 11 ++- docs/build-packages-plan.md | 90 ++++++++++++------- .../device-driver-development/display-plan.md | 7 +- .../display-v2-plan.md | 8 +- .../device-driver-development/driver-model.md | 18 ++-- docs/os-development/threading.md | 14 +-- docs/system-requirements.md | 4 +- library/kernel/build.zig | 5 +- 8 files changed, 101 insertions(+), 56 deletions(-) diff --git a/docs/README.md b/docs/README.md index 7eaeec0..e862288 100644 --- a/docs/README.md +++ b/docs/README.md @@ -263,9 +263,18 @@ test/ → /test the test tree: the QEMU harness (qemu_test.py, h system/services/ beside the on-image test fixtures — vfs-test/ thread-test/ crash-test/ … — whose repo path IS their boot-volume path (/test/system/services/) +build-support/ the danos build API (build-time only, nothing on the image): + the shared user-binary recipe + default-import wiring every + build file consumes (docs/build-packages-plan.md) tools/ host-side build scripts ``` +**Builds are packages** (docs/build-packages-plan.md): each `library/` domain owns a +`build.zig`/`build.zig.zon` exporting its modules (with a standalone `zig build test`), +binaries are converting one directory at a time to ~15-line package builds (`pci-bus` +is the first), and the root `build.zig` orchestrates — image assembly, QEMU, the +aggregate test step. + **Wire protocols live in `library/protocol/`**, one module per directory (`library/protocol/vfs/vfs-protocol.zig` is the `vfs-protocol` module), imported by module name. A protocol is the seam between a low-level driver and the higher-level service it @@ -320,5 +329,5 @@ exception in [coding-standards.md](coding-standards.md) applies to that seam. | System services (init, the `fat` filesystem, the device-manager) | `system/services/` | | Device drivers, one sub-project each (`pci-bus`, `ps2-bus`, `usb-xhci-bus` bus drivers) | `system/drivers/` | | On-image test fixtures for the QEMU cases (`vfs-test`, `crash-test`, `thread-test`, …) → `/test/system/services` | `test/system/services/` | -| Build + `run-x86-64` (QEMU/OVMF) + `release-x86-64` (the flashable ISO) | `build.zig` | +| Build orchestration + `run-x86-64` (QEMU/OVMF) + `release-x86-64` (the flashable ISO) | `build.zig` (root; the shared user-binary recipe is `build-support/`, and each `library/` domain + packaged binary carries its own `build.zig`) | | QEMU integration test harness | `test/qemu_test.py` | diff --git a/docs/build-packages-plan.md b/docs/build-packages-plan.md index 8479771..7b8a646 100644 --- a/docs/build-packages-plan.md +++ b/docs/build-packages-plan.md @@ -1,10 +1,17 @@ # Plan: packages — hierarchical builds for libraries and binaries -**Status: proposed, awaiting sign-off. No implementation yet.** +**Status: in progress.** Implemented on branch `claude/build-packages-plan-174144`: +phase 0 (`build-support`), phase 1 (all six library domains as packages, root as +the pilot consumer), and the first phase-2 binary package — `pci-bus`, a +deliberate deviation from wave A's two-small-services opener, because a driver +with per-binary extras (device-manager-protocol, pci-class) exercises the +template harder than a plain service. Every phase landed green (unit tests, the +QEMU suite at parity with main, boot-image file list unchanged). Remaining: +waves B–D of phase 2, then phase 3. ## Why -`build.zig` is ~1,250 lines and grows by three hand-written stanzas per binary; +`build.zig` was ~1,250 lines, growing by three hand-written stanzas per binary; at a driver per device family that does not scale. More fundamentally: in one monolithic build every binary compiles against library *source*, so a library interface break is silently absorbed by whoever edits everything in one commit — @@ -21,18 +28,27 @@ know where anything lives. ## Target shape ``` -build-support/ package: the danos build API (userBinary(), targets, default imports) -library/kernel/ package "kernel": modules ipc, service, memory, process, logging, time, ... -library/device/ package "device": modules driver, pci, usb-abi, model, ... (depends on kernel) +build-support/ package: the danos build API (userBinary(), defaultImports(), targets) +library/kernel/ package "kernel": modules abi, ipc, service, memory, process, logging, time, ... (depends on protocol) +library/device/ package "device": modules driver, pci, usb-abi, model, ... (depends on kernel, protocol, csv) library/protocol/ package "protocol": the wire protocols library/client/ package "client" (depends on kernel, protocol) library/csv/ package "csv" library/xkeyboard-config/ package "xkeyboard-config" -system/services// one package per binary: ~10-line build.zig + zon +system/services// one package per binary: ~15-line build.zig + zon system/drivers// one package per binary build.zig (root) orchestrator: dependency() per binary, image assembly, QEMU, test steps ``` +The three shared contracts: `boot-handoff` stays a root module (only the +loader↔kernel pair speaks it); `abi` is exported by the kernel package from +`../../system/abi.zig` (the source stays with the kernel; userspace's one view +of it lives in the package, so every consumer names the same module instance); +`device-abi` is exported by device. Reaching outside the package root means the +kernel package is valid only as an in-repo path dependency — it could never be +fetched by hash — which is fine: path dependencies are the only way any of +these packages is consumed. + Rules: - **Dependencies are declared at domain level** (a binary's zon names `kernel`, @@ -72,41 +88,47 @@ the freestanding target setup, and the default-import wiring into the `build-support` package. Root build consumes it; nothing else moves. This is the cross-cutting-change home, so it lands first. -**Phase 1 — library domains become packages.** In dependency order: `kernel` -(no deps) → `csv`, `protocol` → `device`, `client` → `xkeyboard-config`. Each -gets build.zig + zon + a standalone test step. The root build swaps its -`createModule` calls for `b.dependency("").module("")`. **No -binary moves in this phase** — the root build is the pilot consumer, which -proves the packages without touching 30 binaries. +**Phase 1 — library domains become packages.** In dependency order: `protocol` +and `csv` (the roots) → `kernel` (depends on protocol: file-system speaks +vfs-protocol) → `device`, `client`; `xkeyboard-config` stands alone. Each gets +build.zig + zon + a standalone test step (client's is empty until its modules +grow host tests — kept for uniformity, since the root aggregate depends on +every domain's test step). The root build swaps its `createModule` calls for +`b.dependency("").module("")`. **No binary moves in this phase** +— the root build is the pilot consumer, which proves the packages without +touching 30 binaries. -**Phase 2 — binaries become packages, in waves.** Wave A: two small services -(e.g. logger, display-demo) to shake out the template. Wave B: remaining -services. Wave C: drivers. Wave D: test fixtures. Root build shrinks to -orchestration per wave. +**Phase 2 — binaries become packages, in waves.** The template was shaken out +by the pci-bus pilot (see Status). Wave A: services. Wave B: the remaining +drivers. Wave C: test fixtures. Root build shrinks to orchestration per wave. **Phase 3 — root cleanup.** Split what remains of the root build into `build/images.zig`, `build/qemu.zig`, imported by a short root `build.zig`. **Afterwards** (outside this plan): the intel-uhd-graphics-750 driver is -(re)created as a greenfield package — the new-driver checklist's step 2 gets -rewritten against the package template at that point. +(re)created as a greenfield package — the "Adding a driver" checklist's build +step (docs/device-driver-development/devices-csv.md, step 1) gets rewritten +against the package template at that point. -## Execution notes (for whichever session runs this) +## Execution notes (for whichever session runs the remaining waves) -Anchors in today's root `build.zig` (~1,250 lines): +Anchors in the root `build.zig` as it stands after the pilot: -- `addUserBinary` / `addThreadedUserBinary` / `addUserBinaryImpl` and the - `default_imports` plumbing start around line 63 — this is what phase 0 - extracts into `build-support`. `addUserBinaryImpl` also wires the `start` - root shim from `default_imports`; that trick must survive the move. -- `programModule().addImport(...)` calls (search `programModule`) are the - per-binary extra imports — the data for each binary's future build.zig. +- The shared recipe lives in `build-support/build.zig`: `userBinary` (domains + form, what binary packages call), `userBinaryFromImports` (the underlying + recipe root's stanzas still use), and `defaultImports` — the ONE list of + default modules; root and the packages both draw from it. The `start` root + shim and `user.ld` are named through the kernel package (Dependency.path). +- `programModule().addImport(...)` calls in root (search `programModule`) + are the per-binary extra imports — the data for each binary's future + build.zig. `system/drivers/pci-bus/build.zig` is the template to copy. - The boot-tree array (search `"etc/init.csv"` or `.getEmittedBin()`) is the - image file list — the authoritative before/after comparison target. -- Protocol/module definitions (search `createModule`) map module names to - `library/...` source paths — the data for each domain package's exports. -- The QEMU size-check test hardcodes source paths (search - `virtio-gpu-protocol.zig` near line 1149) — moves with phase 2 wave C. + image file list — the authoritative before/after comparison target. A + converted binary's stanza becomes + `b.dependency("", .{}).artifact("")` plus a zon entry. +- The QEMU size-check tests hardcode source paths (search + `virtio-gpu-protocol.zig` in the root test list) — they move with their + binaries' waves. Verification per phase: @@ -128,5 +150,7 @@ docs/coding-standards.md (kebab-case names, no abbreviations), and the upgrade lands separately, never mid-phase. - The QEMU size-check tests hardcode source paths (e.g. virtio-gpu protocol struct sizes in the root build) — phase 2 wave C must carry those along. -- Doc updates ride each phase: drivers.md, new-driver-checklist.md, and - docs/README.md reference build steps that will change shape. +- Doc updates ride each phase: docs/README.md (repo layout + source map), + docs/device-driver-development/devices-csv.md ("Adding a driver", step 1), + and the docs that cite the build recipe (driver-model.md, threading.md, + system-requirements.md) reference build shapes that keep changing. diff --git a/docs/device-driver-development/display-plan.md b/docs/device-driver-development/display-plan.md index d77fbe7..ab3ae6f 100644 --- a/docs/device-driver-development/display-plan.md +++ b/docs/device-driver-development/display-plan.md @@ -20,9 +20,10 @@ Read [display.md](display.md) first for the *why*; this is the *what* and the *o Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations in full, kebab-case file names, no `Co-Authored-By` trailers on commits. New user binaries -go through `addUserBinary` in [build.zig](../../build.zig) and get packed into the -initial-ramdisk; protocols are `b.addModule("…-protocol", …)` and imported into the -`runtime` module. +go through build-support's shared user-binary recipe and get packed into the +initial-ramdisk; protocols are modules exported by the `library/protocol` package. +(This section predates the build-packages split; see +[build-packages-plan.md](../build-packages-plan.md) for the current build shape.) ## How to verify along the way diff --git a/docs/device-driver-development/display-v2-plan.md b/docs/device-driver-development/display-v2-plan.md index 179a8cf..70e8004 100644 --- a/docs/device-driver-development/display-v2-plan.md +++ b/docs/device-driver-development/display-v2-plan.md @@ -18,9 +18,11 @@ lands on its own and ends in a **verifiable gate** — shaped for a `/loop` run, Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations, kebab-case file names, no `Co-Authored-By` trailers. New user binaries go through -`addUserBinary` and get packed into the initial-ramdisk; protocols are -`b.addModule("…-protocol", …)` imported into `runtime`; new syscalls extend -[abi.zig](../../system/abi.zig) `SystemCall` + a `library/runtime` wrapper. +build-support's shared user-binary recipe and get packed into the initial-ramdisk; +protocols are modules exported by the `library/protocol` package; new syscalls extend +[abi.zig](../../system/abi.zig) `SystemCall` + a `library/kernel` wrapper. +(This section predates the build-packages split; see +[build-packages-plan.md](../build-packages-plan.md) for the current build shape.) ## How to verify along the way diff --git a/docs/device-driver-development/driver-model.md b/docs/device-driver-development/driver-model.md index 589a20c..1e41a86 100644 --- a/docs/device-driver-development/driver-model.md +++ b/docs/device-driver-development/driver-model.md @@ -138,13 +138,17 @@ a higher-level service (block ↔ filesystem, a scanout driver ↔ the composito private wire to its *hardware* — virtio-gpu's command set — is not that; it stays a driver-private file, like the virtio-pci transport beside it. -The build side of this has since landed: [`addUserBinary`](build.zig) injects the -default modules — the library/kernel concern modules (`ipc`, `memory`, `process`, `time`, -`logging`, `file-system`, `thread`, `service`), the device/service clients (`driver`, -`block`, `display`, `input`), plus `mmio`, `xkeyboard-config`, `acpi-ids` — into every user -binary, and per-binary extras — protocol modules, bus logic — are added with -`programModule(exe).addImport(...)`. That's the *entire* mechanism — Zig modules -already give you everything else. +The build side of this has since landed: the shared recipe in +[`build-support/build.zig`](../../build-support/build.zig) (`defaultImports` + +`userBinary`) injects the default modules — the library/kernel concern modules (`ipc`, +`memory`, `process`, `time`, `logging`, `file-system`, `thread`, `service`), the +device/service clients (`driver`, `block`, `display`, `input`), plus `mmio`, +`xkeyboard-config`, `acpi-ids` — into every user binary, and per-binary extras — +protocol modules, bus logic — are added with `programModule(exe).addImport(...)`. +Most binaries are still built by the root `build.zig`'s stanzas through that recipe; +a binary can instead own a package with its own ~15-line `build.zig` (pci-bus is the +first — see [build-packages-plan.md](../build-packages-plan.md)). That's the *entire* +mechanism — Zig modules already give you everything else. The discipline that makes this work: **a class driver must not import a bus's *hardware* logic module.** `usb-hid` imports `usb` (the transfer client) and `input-protocol`, never diff --git a/docs/os-development/threading.md b/docs/os-development/threading.md index 4675a44..eaa94f4 100644 --- a/docs/os-development/threading.md +++ b/docs/os-development/threading.md @@ -61,9 +61,10 @@ runtime — rebuilt in lockstep — knows the mapping. backend would either bake danos syscall numbers into std (breaking ABI privacy and renumbering) or fork std to route back through the runtime — a permanent rebase cost that buys nothing the native type doesn't. -2. **Our user binaries are built `single_threaded = true`** ([build.zig](../../build.zig) - `addUserBinary`), which compiles threading out entirely and makes atomics and TLS - single-threaded. Threads need this flipped per binary regardless. +2. **Our user binaries are built `single_threaded = true`** (the shared recipe in + [build-support/build.zig](../../build-support/build.zig)), which compiles threading + out entirely and makes atomics and TLS single-threaded. Threads need this flipped + per binary regardless. So we take the *shape* of `std.Thread`, not the *type*. The cost of replicating the surface (spawn/join/Mutex/Condition) is small; the cost of the std type is the ABI @@ -236,9 +237,10 @@ see the intro). Two scoped pieces, as built: ### Build: multi-threaded codegen, opt-in -A binary opts in by being added with `addThreadedUserBinary` — as `addUserBinary`, -but the shared implementation builds it `single_threaded = false` — so atomics and -(later) TLS are real. Threads and atomics are unsound in a `single_threaded` image, +A binary opts in with `addThreadedUserBinary` in the root `build.zig` (or +`.threaded = true` in a binary package's `build_support.userBinary` call) — the +shared recipe in build-support then builds it `single_threaded = false` — so atomics +and (later) TLS are real. Threads and atomics are unsound in a `single_threaded` image, so a binary must opt in **before** it may call `Thread.spawn`. Everyone else stays single-threaded and lean. diff --git a/docs/system-requirements.md b/docs/system-requirements.md index 4083d0a..2df286f 100644 --- a/docs/system-requirements.md +++ b/docs/system-requirements.md @@ -95,9 +95,9 @@ hypervisor configured for UEFI firmware and an xHCI USB controller. | Requirement | Detail | Source | |---|---|---| -| **x86-64, 64-bit only** | Kernel and loader are built exclusively for `x86_64`; the loader rejects any non-x86-64 kernel ELF (`error.WrongArchitecture`). | `build.zig:481`, `boot/efi.zig:622` | +| **x86-64, 64-bit only** | Kernel and loader are built exclusively for `x86_64`; the loader rejects any non-x86-64 kernel ELF (`error.WrongArchitecture`). | `build-support/build.zig` (`freestandingTarget`), `boot/efi.zig:622` | | **Long mode + PAE + NX** | AP trampoline sets `CR4.PAE`, `EFER.LME`, `EFER.NXE`; NX is used in kernel page-table entries. | `system/kernel/architecture/x86_64/trampoline.s:62` | -| **SSE / SSE2** | Baseline: the compiler emits SSE for ordinary struct copies. Trampoline enables `CR4.OSFXSR` + `OSXMMEXCPT` and clears `CR0.EM`. | `build.zig:477`, `trampoline.s:62` | +| **SSE / SSE2** | Baseline: the compiler emits SSE for ordinary struct copies. Trampoline enables `CR4.OSFXSR` + `OSXMMEXCPT` and clears `CR0.EM`. | `build-support/build.zig` (`freestandingTarget`), `trampoline.s:62` | | **`syscall` / `sysret`** | Primary user↔kernel entry path. `EFER.SCE` enabled; `STAR`/`LSTAR`/`SFMASK` programmed per core. (`int 0x80` exists as a parallel gate.) | `architecture/x86_64/per-cpu.zig:71`, `isr.s:196` | | **Local APIC (xAPIC)** | LAPIC accessed via MMIO at `0xFEE00000`. LAPIC ID read as a `u8` — classic xAPIC. **x2APIC is not supported** (no MSR path). | `apic.zig:67`, `apic.zig:646` | | **CPUID + RDTSC** | CPUID leaf `0x15` for TSC frequency; RDTSC is the monotonic clock. | `apic.zig:333`, `apic.zig:113` | diff --git a/library/kernel/build.zig b/library/kernel/build.zig index 03595b1..d5d4c6c 100644 --- a/library/kernel/build.zig +++ b/library/kernel/build.zig @@ -6,7 +6,10 @@ //! This package also exports `abi` — the kernel <-> user contract (SystemCall //! numbers, mmap prot flags, page_size). Its source lives with the kernel in //! system/abi.zig, outside this directory, but userspace's one view of it is -//! exported here so every consumer names the same module instance. +//! exported here so every consumer names the same module instance. Reaching +//! outside the package root means this package is valid only as an in-repo +//! path dependency (never fetchable by hash) — fine, since path dependencies +//! are the only way danos packages are consumed. //! //! The root shim (root.zig) and the user link script (user.ld) are plain //! files, not modules; build-support reaches them through this package's