# The driver model: buses, classes, and host controllers [drivers.md](drivers.md) shows how to write *a* driver — claim a device, map its registers, sleep on its interrupt. That's enough for a leaf device like the HPET. It is not enough for a disk, a keyboard, or a network card, because those hang off a *controller*, on a *bus*, speaking a *protocol*, and no single process should have to know all three. Real driver stacks factor into three shapes. This document is about what each one is, what the kernel must give it, how they share code — and precisely which primitive each is still blocked on. ## Three shapes | Shape | Owns | Reaches hardware by | Talks to | |---|---|---|---| | **Host controller driver** (HCD) | a controller — an xHCI PCI function, an AHCI port block | `mmio_map` + `irq_bind` + DMA | the devices behind it, in its bus's language | | **Bus driver** | a bus — a PCI bridge, a USB hub | `device_register`, to publish what it finds | class drivers, over IPC | | **Class / protocol driver** | *nothing* | *nothing* | its bus driver, over IPC | The last row is the surprising one and the whole point. A USB keyboard driver touches no registers, takes no interrupts, and maps no memory. It sends HID protocol messages to whatever published the device, and it works identically whether the controller below is xHCI, EHCI, or a Raspberry Pi's DWC2. That is what buys you drivers that outlive the hardware they were written for. In practice **HCD and bus driver are usually the same process**. An xHCI driver is a host controller driver (it owns the PCI function, its BARs, its interrupt, its DMA rings) *and* a bus driver (it enumerates USB devices and publishes them). Splitting them is a fiction; what matters is that both *roles* have kernel support, because a plain bus driver with no controller — a USB hub — is also a real thing. ## The device table is the spine danos already has the right central structure. `system/kernel/devices-broker.zig` holds a table of `DeviceDesc`, each with a parent, a class, and a set of resources. Firmware discovery seeds it ([discovery.md](discovery.md)); `device_register` grows it. Three invariants make it a capability system rather than a directory: 1. **A claim is exclusive.** `device_claim(id)` succeeds once. Everything downstream — `mmio_map`, `irq_bind`, `device_register` — checks `devices_broker.ownerOf(id) == me`. 2. **A descriptor is a licence to map physical memory.** Whoever claims a device may map its `.memory` resources and bind its `.irq` resources. This is why `device_register` cannot be a free-for-all. 3. **Therefore: containment.** Every resource of a registered child must lie inside a resource of the same kind on its parent (`devices_broker.contains`). A bus driver can only ever *subdivide* what it already holds. Without this, `device_register` would be a syscall named "map any physical page you like." Containment is transitive by construction: a grandchild is contained in its child, which is contained in the bus. Nothing can be laundered through a chain. Note that firmware topology does **not** obey containment, and isn't asked to — a PCI function's BAR is not inside its host bridge's `bus_range`, because a bus-number range is not an address window. Discovery is trusted; user space is not. ### What a bus driver looks like danos ships no demo bus driver — the real ones are `pci-bus`, `ps2-bus`, and `usb-xhci-bus`. The smallest *honest* shape, illustrated here with an HPET register block as the "bus" and its comparators as the "devices", is: ```zig _ = dev.claim(bus.id); // 1. own the bus const base = dev.mmioMap(bus.id, 0).?; // 2. enumerate it — from the hardware const n = ((cap.* >> 8) & 0x1F) + 1; // GENERAL_CAP says how many children for (0..n) |i| { // 3. publish each child var child = std.mem.zeroes(dev.DeviceDesc); child.class = @intFromEnum(dev.DeviceClass.timer); child.resource_count = 1; child.resources[0] = .{ .kind = memory, .start = bus_mmio.start + 0x100 + 0x20 * i, .len = 0x20 }; _ = dev.register(bus.id, &child).?; // kernel checks containment } ``` Each child is left **unclaimed**, which is the handoff: a comparator driver can now `device_claim` one and `mmio_map` it, and will see only its own 0x20-byte window. A child whose window escapes the bus is refused; the in-kernel `containment` test asserts the kernel's table upholds that ([drivers.md](drivers.md)). A USB device has *no* resources at all: `resource_count = 0`, because it's addressed through its controller, not by MMIO. That case is allowed and is the common one. ## Families: sharing code between drivers A "family" is two modules, not one: - **A logic module** — the parts of the bus that every driver on it re-derives. Config space walking and BAR decode for PCI. Descriptor parsing, control transfers, and hub protocol for USB. - **A protocol module** — the IPC message types that let a class driver talk to *whatever* published its device. This is the part that makes class drivers portable. danos already has one of each: `library/runtime/device.zig` is a logic module, [`system/services/vfs/protocol.zig`](system/services/vfs/protocol.zig) is a protocol module shared by `system/services/vfs/vfs.zig` and its clients. The pattern generalises directly: ``` library/ runtime/ module "runtime" — syscalls, heap, ipc, device, stdio mmio/ module "mmio" — volatile register access + barriers [M14] bus/ pci/ module "pci" — ECAM, BAR decode, capability walk usb/ module "usb" — descriptors, control transfers, hubs proto/ vfs/ module "vfs-protocol" (today: system/services/vfs/protocol.zig) block/ module "block-protocol" hid/ module "hid-protocol" system/drivers/ one sub-project each → /system/drivers (no `d` suffix) xhci/ HCD + bus driver imports runtime, pci, usb, mmio usb-hid/ class driver imports runtime, usb, hid-protocol block/ class driver imports runtime, block-protocol ``` The only build change needed: [`addUserBinary`](build.zig) currently takes exactly one module (`rt_mod`) and injects it. It should take a slice of modules. That's a five-line change, and it's the *entire* mechanism — Zig modules already give you everything else. The discipline that makes this work: **a class driver must not import a bus's logic module.** `usbhid` imports `proto.hid` and `usb` (for descriptor types), never `pci`. If a class driver needs `mmio`, it has become an HCD and should be one. ## What exists today - **M10** — `device_enumerate`, `device_claim`, `mmio_map`. Strong-uncacheable device grants, `device_grant` teardown. - **M11** — `irq_bind` / `irq_ack`. IRQ delivered as an IPC notification; mask before EOI; `irq_ack` is the unmask. - **M12** — `parent` in `DeviceDesc`, `device_register` with resource containment. - **M13** — capability passing. `ipc_call` / `ipc_reply_wait` grew a `send_cap` argument and a `received_cap` return (r8): an endpoint travels with a message, installed into the receiver's handle table (shared, refcount-bumped — a copy, not a move). A full table fails `-ENOSPC` and does not half-deliver. This is the "open" primitive — a bus driver mints a per-device endpoint and hands it to a class driver. The runtime exposes `callCap` and `replyWait(..., send_cap)`; no class driver consumes it yet. - **M14** — DMA memory + the memory-ordering layer. `/lib/mmio` gives drivers typed volatile access and `mb`/`rmb`/`wmb` (per-arch); `dma_alloc`/`dma_free` grant physically-contiguous, pinned, uncacheable, reclaim-on-teardown buffers with the physical address exposed (`pmm.allocContiguous`, a DMA arena, `mapUserDmaInto`). `dma_below_4g` caps the address for legacy engines; `dma_write_combining` is accepted but falls back to coherent until PAT is programmed. The bus drivers use `/lib/mmio`; no DMA driver consumes `dma_alloc` yet. - **M15** — interrupts for PCI devices, the MSI half. Discovery now gives every PCI function its 4 KiB ECAM config space as resource 0 (unblocking the capability walk with no new syscall), and `msi_bind(device_id, endpoint) -> address, data` allocates a per-device edge-triggered vector, delivered as an IPC notification with no mask and no ack cycle. Legacy INTx (`_PRT` parsing + shared lines) is deliberately skipped — MSI is the real answer. QEMU's HPET has no MSI, so delivery is proven with a self-IPI; the first PCI driver is the first real consumer. - **Port I/O** — `io_read`/`io_write(device_id, resource_index, offset, width[, value])`: a claimed device's `io_port` resource lets a driver read/write its ports, gated exactly like `mmio_map` gates memory (direct ring-3 `in`/`out` stays a #GP). This is what makes a PS/2 or 16550 driver possible; the low-rate legacy hardware that needs it is fine with a syscall per access. `io_port` resources were recorded by discovery and ignored — now they're used. - **M16 (detection)** — the IOMMU is now *found*: discovery parses the ACPI DMAR table, maps the first VT-d unit, and reads its version + capabilities (`iommu_present` in the platform info). This is detection only — **no translation domains are programmed, so DMA is still unprotected** (the caveat below). Enforcement lands with the first DMA driver, which is what there is to protect and test against. Proven in the `iommu` test, booted with an emulated `intel-iommu`. - **`system_spawn`** — a user-space supervisor starts a driver: `system_spawn(name, arguments)` loads a binary bundled in the initial-ramdisk as a fresh ring-3 process; `name` becomes the child's argv[0] and the optional NUL-separated `arguments` blob its argv[1..], delivered on a SysV entry stack ([sysv.md](sysv.md)). This is what turned the device manager from "log the match" into "run the driver": the kernel now spawns only `init`, `init` spawns the services, and the **device-manager** discovers the hardware and spawns each driver ([drivers.md](drivers.md)). Ungated for now — a spawn capability is future work. So: **bus drivers work now, and they're started by the device manager, not the kernel.** HCDs and class drivers do not work yet. Here is exactly why, and exactly what would fix it. --- # Proposed ABI ## M13 — capability passing, for class drivers ✅ done *Implemented as described below (see "What exists today"). The signatures landed verbatim: `send_cap` in r9, `received_cap` returned in r8, `-ENOSPC` on a full receiver table with no delivery. The rest of this section is the original design note.* **The blocker.** A class driver has to reach *its* device. Today the only way to find an endpoint is the name registry: `ipc_register(service_id, h)` / `ipc_lookup(id)`, where `ServiceId` is a global integer namespace with `max_services = 8`. You cannot mint one endpoint per USB device that way, and there is no way for a bus driver to *hand* a class driver an endpoint. M7 deferred this deliberately. **The fix.** Let a message carry one handle. Sender names a handle in its own table; the kernel installs the endpoint into the receiver's table (bumping `refcount`) and tells the receiver the index it landed at. ``` ipc_call(h, msg, message_len, reply, reply_cap, send_cap) -> reply_len ipc_reply_wait(h, reply, reply_len, recv, recv_cap, send_cap) -> recv_len (rax), badge (rdx), received_cap (r8) ``` `send_cap` is a handle or `no_cap` (`~0`). `received_cap` is the index the transferred endpoint was installed at in the receiver's table, or `no_cap`. - Both calls grow from 5 args to 6, which fits: `syscall5` uses `rdi/rsi/rdx/r10/r8`, leaving `r9`. `ipc_reply_wait` already returns two values via `setSyscallResult2`; this needs a third (`setSyscallResult3`). - If the receiver's handle table is full, the call fails `-ENOSPC` and **the message is not delivered** — a half-delivered capability is worse than a failed send. - `closeHandles` already drops references on exit, so the lifetime story is unchanged. That single primitive gives you the standard `open` pattern: ```zig // class driver // bus driver const h = ipc.lookup(.usb).?; const r = ipc.replyWait(ep, ...); const dev_ep = ipc.callCap(h, // ... mint a per-device endpoint, .{ .op = .open, .id = dev_id }); // reply with it as send_cap // now dev_ep is a private channel to that one device ``` ## M14 — DMA memory and the memory-ordering contract, for HCDs ✅ done *Implemented: `/lib/mmio` (typed volatile access + `mb`/`rmb`/`wmb`, per-arch) and `dma_alloc`/`dma_free` (contiguous, pinned, uncacheable, reclaim-on-teardown, physical address exposed). `dma_write_combining` still falls back to coherent — real WC needs PAT, a small follow-up. The rest of this section is the original design note.* **The blocker.** An HCD is a DMA-engine programmer. It needs a descriptor ring the device can read, which means memory that is (a) physically contiguous, (b) at a physical address the driver knows, (c) of the right cacheability, and (d) pinned. [`sysMmap`](system/kernel/process.zig) gives you *none* of the four: it calls `pmm.alloc()` once per page, maps writeback-cached, and never reveals a physical address. **The fix.** ``` dma_alloc(len, flags) -> vaddr (rax), paddr (rdx) dma_free(vaddr, len) -> 0 flags: dma_coherent (1) uncacheable; the default and the only one that's portable dma_wc (2) write-combining — needs PAT programmed; for framebuffers dma_below_4g (4) for devices with 32-bit DMA addressing ``` Guarantees: page-aligned, physically contiguous, zeroed, pinned for the life of the mapping, and the physical address is stable. It needs one thing the kernel lacks — `pmm.allocContiguous(n, max_phys)`; today `pmm.alloc()` hands out one frame at a time with no adjacency guarantee. **The memory-ordering contract.** danos has, at the time of writing, **zero memory barriers anywhere in the tree.** That is currently correct-by-accident and won't survive the first DMA driver, or the first ARM boot. `volatile` is not a barrier. In Zig it means: don't elide this access, and don't reorder it against *other volatile* accesses. It says nothing about your *ordinary* stores — the descriptor you just filled in normal WB memory — which LLVM may freely sink past a volatile MMIO write. The canonical bug: ```zig ring[i] = descriptor; // ordinary store to WB RAM doorbell.* = i; // volatile store to UC MMIO // nothing stops the compiler reordering these; the device reads a stale descriptor ``` So the rules, which belong in `library/mmio.zig` and behind `arch`: | Situation | Required | |---|---| | MMIO register read/write | `mmio.read` / `mmio.write` (volatile) | | Fill DMA descriptor, then ring doorbell | `wmb()` between them | | Woken by IRQ, then read what the device wrote | `rmb()` before the read | | MMIO write that must complete before the next read | `mb()` | And the per-arch lowering — the reason this must be an `arch` primitive and not a sprinkling of `asm volatile`: | | x86_64 | aarch64 | |---|---|---| | `mb()` | `mfence` | `dsb sy` | | `rmb()` | `lfence` | `dsb ld` | | `wmb()` | `sfence` | `dsb st` | | DMA cache coherency | coherent; nothing to do | **not guaranteed**; needs non-cacheable buffers or cache maintenance | x86 is forgiving here — TSO plus strong-uncacheable MMIO means you usually get away with a compiler barrier alone. ARM is not, and [vision.md](vision.md) makes ARM the win condition. Build the abstraction while there is one caller to fix. (Zig note: `@fence` was **removed in 0.16**. Use `@atomicRmw(..., .seq_cst)` for a full barrier, or per-arch inline asm — which is what `library/mmio.zig` should hide.) ## M15 — interrupts for PCI devices ✅ done (MSI) *Implemented the MSI half: ECAM config space per PCI function (resource 0) and `msi_bind` (per-device edge-triggered vector, delivered as a notification). Legacy INTx `_PRT` parsing is skipped on purpose. `msi_bind` returns (address, data) as two values rather than an out-struct. The rest of this section is the original design note.* **The blocker, and it's a hard one.** No PCI device can take an interrupt today. [`addBars`](system/devices/acpi.zig) records `.memory` and `.io_port` BARs and never an `.irq`; there is no `_PRT` parsing anywhere in the tree. The HPET is the one exception — it advertises its own interrupt routing in its own registers, a privilege no ordinary device has. **The fix, in two halves.** *Legacy INTx*: parse `_PRT` from the DSDT to map (device, INTA–D) → GSI, and record it as an `.irq` resource. Then `irq_bind` works unchanged. But INTx lines are **shared**, and `irq.bound[gsi]` holds one endpoint. Sharing needs a list, and every driver on the line must be polled on each interrupt — the reason everyone left INTx behind. *MSI/MSI-X*, which is the real answer: per-device vectors, edge-triggered, unshared, no mask/ack cycle, no 24-GSI ceiling. The kernel allocates a vector and hands the driver the (address, data) pair to program into its own MSI capability: ``` msi_bind(dev_id, endpoint, out) -> 0 // out: extern struct { addr: u64, data: u32 } ``` The driver writes those into config space itself — which means it needs config space, which means **discovery should give each `pci_device` a `.memory` resource for its 4 KiB ECAM slot**. That's a small change to `parseMcfg` and it unblocks the whole capability walk (MSI, MSI-X, PCIe extended caps) without any new syscall. Note QEMU's HPET reports `Tn_FSB_INT_DEL_CAP = 0` — no MSI — so an HPET timer could never exercise this path. The first MSI driver will be the first PCI driver. ## M16 — the IOMMU, and the honest caveat ◑ detection done, enforcement pending *The IOMMU is now detected (DMAR parsed, VT-d unit mapped and read — see the `iommu` test), but **enforcement is not built**: no translation domains are programmed, so the caveat below still holds in full. Detection can't be taken further usefully until there is a DMA driver to protect and QEMU's `intel-iommu` to test the protection against — building the per-device domains alongside that first driver is both the natural order and the only way to verify them. The rest of this section is the original caveat.* Everything above is capability-gated at the *CPU*. None of it is gated at the *device*. A driver that can program a bus-mastering engine can make that device write to any physical address, because page tables sit between the CPU and RAM, not between a device and RAM. Until VT-d/DMAR (or SMMU on ARM) is programmed from the DMAR table, **`device_claim` on any DMA-capable device is equivalent to granting ring 0.** This does not make the model useless — it's the same position Linux is in with the IOMMU off, and every other guarantee (crash isolation, restart, no shared address space) still holds. But "user-space drivers are memory-safe" is not true yet, and the gap should be named rather than implied. ## Ordering `M13` (capability passing) is independent of `M14`/`M15` and is the cheapest. It unlocks class drivers, which are the shape with no hardware requirements at all — you could write a real one against any device a bus driver publishes tomorrow. `M14` and `M15` together unlock the first HCD. `M14`'s barrier layer is worth landing on its own regardless: it's small, obviously correct, and stops every future driver from hand-rolling `*volatile` and getting ARM wrong. ## See also - [drivers.md](drivers.md) — how to write one, concretely. - [discovery.md](discovery.md) / [acpi.md](acpi.md) — where the device table comes from. - [ipc.md](ipc.md) — endpoints, badges, and the notification path an IRQ arrives on. - [resilience.md](resilience.md) — restart, the reason any of this is worth the trouble.