docs: catch the build docs up with the package split
Review findings: the plan doc claimed no implementation existed, had the domain dependency order wrong (kernel depends on protocol; device on kernel + protocol + csv), never placed the three shared contracts, and named a nonexistent new-driver-checklist.md. Its status now records the implemented phases (and the deliberate pci-bus-first pilot), the target shape carries the contract placements and the path-dependency-only constraint on the kernel package's out-of-root abi export, and the execution notes describe the post-pilot build for whichever session runs the remaining waves. README's repo layout gains build-support/ and the packages-note; driver-model, threading, system-requirements, and the two display plan docs stop citing root build.zig for recipe facts that now live in build-support.
This commit is contained in:
Daniel Samson
+10
-1
@@ -263,9 +263,18 @@ test/ → /test the test tree: the QEMU harness (qemu_test.py, h
|
||||
system/services/ beside the on-image test fixtures — vfs-test/ thread-test/
|
||||
crash-test/ … — whose repo path IS their boot-volume path
|
||||
(/test/system/services/<name>)
|
||||
build-support/ the danos build API (build-time only, nothing on the image):
|
||||
the shared user-binary recipe + default-import wiring every
|
||||
build file consumes (docs/build-packages-plan.md)
|
||||
tools/ host-side build scripts
|
||||
```
|
||||
|
||||
**Builds are packages** (docs/build-packages-plan.md): each `library/` domain owns a
|
||||
`build.zig`/`build.zig.zon` exporting its modules (with a standalone `zig build test`),
|
||||
binaries are converting one directory at a time to ~15-line package builds (`pci-bus`
|
||||
is the first), and the root `build.zig` orchestrates — image assembly, QEMU, the
|
||||
aggregate test step.
|
||||
|
||||
**Wire protocols live in `library/protocol/`**, one module per directory
|
||||
(`library/protocol/vfs/vfs-protocol.zig` is the `vfs-protocol` module), imported by module
|
||||
name. A protocol is the seam between a low-level driver and the higher-level service it
|
||||
@@ -320,5 +329,5 @@ exception in [coding-standards.md](coding-standards.md) applies to that seam.
|
||||
| System services (init, the `fat` filesystem, the device-manager) | `system/services/` |
|
||||
| Device drivers, one sub-project each (`pci-bus`, `ps2-bus`, `usb-xhci-bus` bus drivers) | `system/drivers/` |
|
||||
| On-image test fixtures for the QEMU cases (`vfs-test`, `crash-test`, `thread-test`, …) → `/test/system/services` | `test/system/services/` |
|
||||
| Build + `run-x86-64` (QEMU/OVMF) + `release-x86-64` (the flashable ISO) | `build.zig` |
|
||||
| Build orchestration + `run-x86-64` (QEMU/OVMF) + `release-x86-64` (the flashable ISO) | `build.zig` (root; the shared user-binary recipe is `build-support/`, and each `library/` domain + packaged binary carries its own `build.zig`) |
|
||||
| QEMU integration test harness | `test/qemu_test.py` |
|
||||
|
||||
+57
-33
@@ -1,10 +1,17 @@
|
||||
# Plan: packages — hierarchical builds for libraries and binaries
|
||||
|
||||
**Status: proposed, awaiting sign-off. No implementation yet.**
|
||||
**Status: in progress.** Implemented on branch `claude/build-packages-plan-174144`:
|
||||
phase 0 (`build-support`), phase 1 (all six library domains as packages, root as
|
||||
the pilot consumer), and the first phase-2 binary package — `pci-bus`, a
|
||||
deliberate deviation from wave A's two-small-services opener, because a driver
|
||||
with per-binary extras (device-manager-protocol, pci-class) exercises the
|
||||
template harder than a plain service. Every phase landed green (unit tests, the
|
||||
QEMU suite at parity with main, boot-image file list unchanged). Remaining:
|
||||
waves B–D of phase 2, then phase 3.
|
||||
|
||||
## Why
|
||||
|
||||
`build.zig` is ~1,250 lines and grows by three hand-written stanzas per binary;
|
||||
`build.zig` was ~1,250 lines, growing by three hand-written stanzas per binary;
|
||||
at a driver per device family that does not scale. More fundamentally: in one
|
||||
monolithic build every binary compiles against library *source*, so a library
|
||||
interface break is silently absorbed by whoever edits everything in one commit —
|
||||
@@ -21,18 +28,27 @@ know where anything lives.
|
||||
## Target shape
|
||||
|
||||
```
|
||||
build-support/ package: the danos build API (userBinary(), targets, default imports)
|
||||
library/kernel/ package "kernel": modules ipc, service, memory, process, logging, time, ...
|
||||
library/device/ package "device": modules driver, pci, usb-abi, model, ... (depends on kernel)
|
||||
build-support/ package: the danos build API (userBinary(), defaultImports(), targets)
|
||||
library/kernel/ package "kernel": modules abi, ipc, service, memory, process, logging, time, ... (depends on protocol)
|
||||
library/device/ package "device": modules driver, pci, usb-abi, model, ... (depends on kernel, protocol, csv)
|
||||
library/protocol/ package "protocol": the wire protocols
|
||||
library/client/ package "client" (depends on kernel, protocol)
|
||||
library/csv/ package "csv"
|
||||
library/xkeyboard-config/ package "xkeyboard-config"
|
||||
system/services/<name>/ one package per binary: ~10-line build.zig + zon
|
||||
system/services/<name>/ one package per binary: ~15-line build.zig + zon
|
||||
system/drivers/<name>/ one package per binary
|
||||
build.zig (root) orchestrator: dependency() per binary, image assembly, QEMU, test steps
|
||||
```
|
||||
|
||||
The three shared contracts: `boot-handoff` stays a root module (only the
|
||||
loader↔kernel pair speaks it); `abi` is exported by the kernel package from
|
||||
`../../system/abi.zig` (the source stays with the kernel; userspace's one view
|
||||
of it lives in the package, so every consumer names the same module instance);
|
||||
`device-abi` is exported by device. Reaching outside the package root means the
|
||||
kernel package is valid only as an in-repo path dependency — it could never be
|
||||
fetched by hash — which is fine: path dependencies are the only way any of
|
||||
these packages is consumed.
|
||||
|
||||
Rules:
|
||||
|
||||
- **Dependencies are declared at domain level** (a binary's zon names `kernel`,
|
||||
@@ -72,41 +88,47 @@ the freestanding target setup, and the default-import wiring into the
|
||||
`build-support` package. Root build consumes it; nothing else moves. This is
|
||||
the cross-cutting-change home, so it lands first.
|
||||
|
||||
**Phase 1 — library domains become packages.** In dependency order: `kernel`
|
||||
(no deps) → `csv`, `protocol` → `device`, `client` → `xkeyboard-config`. Each
|
||||
gets build.zig + zon + a standalone test step. The root build swaps its
|
||||
`createModule` calls for `b.dependency("<domain>").module("<name>")`. **No
|
||||
binary moves in this phase** — the root build is the pilot consumer, which
|
||||
proves the packages without touching 30 binaries.
|
||||
**Phase 1 — library domains become packages.** In dependency order: `protocol`
|
||||
and `csv` (the roots) → `kernel` (depends on protocol: file-system speaks
|
||||
vfs-protocol) → `device`, `client`; `xkeyboard-config` stands alone. Each gets
|
||||
build.zig + zon + a standalone test step (client's is empty until its modules
|
||||
grow host tests — kept for uniformity, since the root aggregate depends on
|
||||
every domain's test step). The root build swaps its `createModule` calls for
|
||||
`b.dependency("<domain>").module("<name>")`. **No binary moves in this phase**
|
||||
— the root build is the pilot consumer, which proves the packages without
|
||||
touching 30 binaries.
|
||||
|
||||
**Phase 2 — binaries become packages, in waves.** Wave A: two small services
|
||||
(e.g. logger, display-demo) to shake out the template. Wave B: remaining
|
||||
services. Wave C: drivers. Wave D: test fixtures. Root build shrinks to
|
||||
orchestration per wave.
|
||||
**Phase 2 — binaries become packages, in waves.** The template was shaken out
|
||||
by the pci-bus pilot (see Status). Wave A: services. Wave B: the remaining
|
||||
drivers. Wave C: test fixtures. Root build shrinks to orchestration per wave.
|
||||
|
||||
**Phase 3 — root cleanup.** Split what remains of the root build into
|
||||
`build/images.zig`, `build/qemu.zig`, imported by a short root `build.zig`.
|
||||
|
||||
**Afterwards** (outside this plan): the intel-uhd-graphics-750 driver is
|
||||
(re)created as a greenfield package — the new-driver checklist's step 2 gets
|
||||
rewritten against the package template at that point.
|
||||
(re)created as a greenfield package — the "Adding a driver" checklist's build
|
||||
step (docs/device-driver-development/devices-csv.md, step 1) gets rewritten
|
||||
against the package template at that point.
|
||||
|
||||
## Execution notes (for whichever session runs this)
|
||||
## Execution notes (for whichever session runs the remaining waves)
|
||||
|
||||
Anchors in today's root `build.zig` (~1,250 lines):
|
||||
Anchors in the root `build.zig` as it stands after the pilot:
|
||||
|
||||
- `addUserBinary` / `addThreadedUserBinary` / `addUserBinaryImpl` and the
|
||||
`default_imports` plumbing start around line 63 — this is what phase 0
|
||||
extracts into `build-support`. `addUserBinaryImpl` also wires the `start`
|
||||
root shim from `default_imports`; that trick must survive the move.
|
||||
- `programModule(<exe>).addImport(...)` calls (search `programModule`) are the
|
||||
per-binary extra imports — the data for each binary's future build.zig.
|
||||
- The shared recipe lives in `build-support/build.zig`: `userBinary` (domains
|
||||
form, what binary packages call), `userBinaryFromImports` (the underlying
|
||||
recipe root's stanzas still use), and `defaultImports` — the ONE list of
|
||||
default modules; root and the packages both draw from it. The `start` root
|
||||
shim and `user.ld` are named through the kernel package (Dependency.path).
|
||||
- `programModule(<exe>).addImport(...)` calls in root (search `programModule`)
|
||||
are the per-binary extra imports — the data for each binary's future
|
||||
build.zig. `system/drivers/pci-bus/build.zig` is the template to copy.
|
||||
- The boot-tree array (search `"etc/init.csv"` or `.getEmittedBin()`) is the
|
||||
image file list — the authoritative before/after comparison target.
|
||||
- Protocol/module definitions (search `createModule`) map module names to
|
||||
`library/...` source paths — the data for each domain package's exports.
|
||||
- The QEMU size-check test hardcodes source paths (search
|
||||
`virtio-gpu-protocol.zig` near line 1149) — moves with phase 2 wave C.
|
||||
image file list — the authoritative before/after comparison target. A
|
||||
converted binary's stanza becomes
|
||||
`b.dependency("<name>", .{}).artifact("<name>")` plus a zon entry.
|
||||
- The QEMU size-check tests hardcode source paths (search
|
||||
`virtio-gpu-protocol.zig` in the root test list) — they move with their
|
||||
binaries' waves.
|
||||
|
||||
Verification per phase:
|
||||
|
||||
@@ -128,5 +150,7 @@ docs/coding-standards.md (kebab-case names, no abbreviations), and the
|
||||
upgrade lands separately, never mid-phase.
|
||||
- The QEMU size-check tests hardcode source paths (e.g. virtio-gpu protocol
|
||||
struct sizes in the root build) — phase 2 wave C must carry those along.
|
||||
- Doc updates ride each phase: drivers.md, new-driver-checklist.md, and
|
||||
docs/README.md reference build steps that will change shape.
|
||||
- Doc updates ride each phase: docs/README.md (repo layout + source map),
|
||||
docs/device-driver-development/devices-csv.md ("Adding a driver", step 1),
|
||||
and the docs that cite the build recipe (driver-model.md, threading.md,
|
||||
system-requirements.md) reference build shapes that keep changing.
|
||||
|
||||
@@ -20,9 +20,10 @@ Read [display.md](display.md) first for the *why*; this is the *what* and the *o
|
||||
|
||||
Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations in
|
||||
full, kebab-case file names, no `Co-Authored-By` trailers on commits. New user binaries
|
||||
go through `addUserBinary` in [build.zig](../../build.zig) and get packed into the
|
||||
initial-ramdisk; protocols are `b.addModule("…-protocol", …)` and imported into the
|
||||
`runtime` module.
|
||||
go through build-support's shared user-binary recipe and get packed into the
|
||||
initial-ramdisk; protocols are modules exported by the `library/protocol` package.
|
||||
(This section predates the build-packages split; see
|
||||
[build-packages-plan.md](../build-packages-plan.md) for the current build shape.)
|
||||
|
||||
## How to verify along the way
|
||||
|
||||
|
||||
@@ -18,9 +18,11 @@ lands on its own and ends in a **verifiable gate** — shaped for a `/loop` run,
|
||||
|
||||
Follow [coding-standards.md](../coding-standards.md): spell out non-acronym abbreviations,
|
||||
kebab-case file names, no `Co-Authored-By` trailers. New user binaries go through
|
||||
`addUserBinary` and get packed into the initial-ramdisk; protocols are
|
||||
`b.addModule("…-protocol", …)` imported into `runtime`; new syscalls extend
|
||||
[abi.zig](../../system/abi.zig) `SystemCall` + a `library/runtime` wrapper.
|
||||
build-support's shared user-binary recipe and get packed into the initial-ramdisk;
|
||||
protocols are modules exported by the `library/protocol` package; new syscalls extend
|
||||
[abi.zig](../../system/abi.zig) `SystemCall` + a `library/kernel` wrapper.
|
||||
(This section predates the build-packages split; see
|
||||
[build-packages-plan.md](../build-packages-plan.md) for the current build shape.)
|
||||
|
||||
## How to verify along the way
|
||||
|
||||
|
||||
@@ -138,13 +138,17 @@ a higher-level service (block ↔ filesystem, a scanout driver ↔ the composito
|
||||
private wire to its *hardware* — virtio-gpu's command set — is not that; it stays a
|
||||
driver-private file, like the virtio-pci transport beside it.
|
||||
|
||||
The build side of this has since landed: [`addUserBinary`](build.zig) injects the
|
||||
default modules — the library/kernel concern modules (`ipc`, `memory`, `process`, `time`,
|
||||
`logging`, `file-system`, `thread`, `service`), the device/service clients (`driver`,
|
||||
`block`, `display`, `input`), plus `mmio`, `xkeyboard-config`, `acpi-ids` — into every user
|
||||
binary, and per-binary extras — protocol modules, bus logic — are added with
|
||||
`programModule(exe).addImport(...)`. That's the *entire* mechanism — Zig modules
|
||||
already give you everything else.
|
||||
The build side of this has since landed: the shared recipe in
|
||||
[`build-support/build.zig`](../../build-support/build.zig) (`defaultImports` +
|
||||
`userBinary`) injects the default modules — the library/kernel concern modules (`ipc`,
|
||||
`memory`, `process`, `time`, `logging`, `file-system`, `thread`, `service`), the
|
||||
device/service clients (`driver`, `block`, `display`, `input`), plus `mmio`,
|
||||
`xkeyboard-config`, `acpi-ids` — into every user binary, and per-binary extras —
|
||||
protocol modules, bus logic — are added with `programModule(exe).addImport(...)`.
|
||||
Most binaries are still built by the root `build.zig`'s stanzas through that recipe;
|
||||
a binary can instead own a package with its own ~15-line `build.zig` (pci-bus is the
|
||||
first — see [build-packages-plan.md](../build-packages-plan.md)). That's the *entire*
|
||||
mechanism — Zig modules already give you everything else.
|
||||
|
||||
The discipline that makes this work: **a class driver must not import a bus's *hardware*
|
||||
logic module.** `usb-hid` imports `usb` (the transfer client) and `input-protocol`, never
|
||||
|
||||
@@ -61,9 +61,10 @@ runtime — rebuilt in lockstep — knows the mapping.
|
||||
backend would either bake danos syscall numbers into std (breaking ABI privacy and
|
||||
renumbering) or fork std to route back through the runtime — a permanent rebase
|
||||
cost that buys nothing the native type doesn't.
|
||||
2. **Our user binaries are built `single_threaded = true`** ([build.zig](../../build.zig)
|
||||
`addUserBinary`), which compiles threading out entirely and makes atomics and TLS
|
||||
single-threaded. Threads need this flipped per binary regardless.
|
||||
2. **Our user binaries are built `single_threaded = true`** (the shared recipe in
|
||||
[build-support/build.zig](../../build-support/build.zig)), which compiles threading
|
||||
out entirely and makes atomics and TLS single-threaded. Threads need this flipped
|
||||
per binary regardless.
|
||||
|
||||
So we take the *shape* of `std.Thread`, not the *type*. The cost of replicating the
|
||||
surface (spawn/join/Mutex/Condition) is small; the cost of the std type is the ABI
|
||||
@@ -236,9 +237,10 @@ see the intro). Two scoped pieces, as built:
|
||||
|
||||
### Build: multi-threaded codegen, opt-in
|
||||
|
||||
A binary opts in by being added with `addThreadedUserBinary` — as `addUserBinary`,
|
||||
but the shared implementation builds it `single_threaded = false` — so atomics and
|
||||
(later) TLS are real. Threads and atomics are unsound in a `single_threaded` image,
|
||||
A binary opts in with `addThreadedUserBinary` in the root `build.zig` (or
|
||||
`.threaded = true` in a binary package's `build_support.userBinary` call) — the
|
||||
shared recipe in build-support then builds it `single_threaded = false` — so atomics
|
||||
and (later) TLS are real. Threads and atomics are unsound in a `single_threaded` image,
|
||||
so a binary must opt in **before** it may call `Thread.spawn`. Everyone else
|
||||
stays single-threaded and lean.
|
||||
|
||||
|
||||
@@ -95,9 +95,9 @@ hypervisor configured for UEFI firmware and an xHCI USB controller.
|
||||
|
||||
| Requirement | Detail | Source |
|
||||
|---|---|---|
|
||||
| **x86-64, 64-bit only** | Kernel and loader are built exclusively for `x86_64`; the loader rejects any non-x86-64 kernel ELF (`error.WrongArchitecture`). | `build.zig:481`, `boot/efi.zig:622` |
|
||||
| **x86-64, 64-bit only** | Kernel and loader are built exclusively for `x86_64`; the loader rejects any non-x86-64 kernel ELF (`error.WrongArchitecture`). | `build-support/build.zig` (`freestandingTarget`), `boot/efi.zig:622` |
|
||||
| **Long mode + PAE + NX** | AP trampoline sets `CR4.PAE`, `EFER.LME`, `EFER.NXE`; NX is used in kernel page-table entries. | `system/kernel/architecture/x86_64/trampoline.s:62` |
|
||||
| **SSE / SSE2** | Baseline: the compiler emits SSE for ordinary struct copies. Trampoline enables `CR4.OSFXSR` + `OSXMMEXCPT` and clears `CR0.EM`. | `build.zig:477`, `trampoline.s:62` |
|
||||
| **SSE / SSE2** | Baseline: the compiler emits SSE for ordinary struct copies. Trampoline enables `CR4.OSFXSR` + `OSXMMEXCPT` and clears `CR0.EM`. | `build-support/build.zig` (`freestandingTarget`), `trampoline.s:62` |
|
||||
| **`syscall` / `sysret`** | Primary user↔kernel entry path. `EFER.SCE` enabled; `STAR`/`LSTAR`/`SFMASK` programmed per core. (`int 0x80` exists as a parallel gate.) | `architecture/x86_64/per-cpu.zig:71`, `isr.s:196` |
|
||||
| **Local APIC (xAPIC)** | LAPIC accessed via MMIO at `0xFEE00000`. LAPIC ID read as a `u8` — classic xAPIC. **x2APIC is not supported** (no MSR path). | `apic.zig:67`, `apic.zig:646` |
|
||||
| **CPUID + RDTSC** | CPUID leaf `0x15` for TSC frequency; RDTSC is the monotonic clock. | `apic.zig:333`, `apic.zig:113` |
|
||||
|
||||
@@ -6,7 +6,10 @@
|
||||
//! This package also exports `abi` — the kernel <-> user contract (SystemCall
|
||||
//! numbers, mmap prot flags, page_size). Its source lives with the kernel in
|
||||
//! system/abi.zig, outside this directory, but userspace's one view of it is
|
||||
//! exported here so every consumer names the same module instance.
|
||||
//! exported here so every consumer names the same module instance. Reaching
|
||||
//! outside the package root means this package is valid only as an in-repo
|
||||
//! path dependency (never fetchable by hash) — fine, since path dependencies
|
||||
//! are the only way danos packages are consumed.
|
||||
//!
|
||||
//! The root shim (root.zig) and the user link script (user.ld) are plain
|
||||
//! files, not modules; build-support reaches them through this package's
|
||||
|
||||
Reference in New Issue
Block a user