23 KiB

Raw Blame History

The driver model: buses, classes, and host controllers

drivers.md shows how to write a driver — claim a device, map its registers, sleep on its interrupt. That's enough for a leaf device like the HPET. It is not enough for a disk, a keyboard, or a network card, because those hang off a controller, on a bus, speaking a protocol, and no single process should have to know all three.

Real driver stacks factor into three shapes. This document is about what each one is, what the kernel must give it, how they share code — and precisely which primitive each is still blocked on.

Three shapes

Shape	Owns	Reaches hardware by	Talks to
Host controller driver (HCD)	a controller — an xHCI PCI function, an AHCI port block	`mmio_map` + `irq_bind` + DMA	the devices behind it, in its bus's language
Bus driver	a bus — a PCI bridge, a USB hub	`device_register`, to publish what it finds	class drivers, over IPC
Class / protocol driver	nothing	nothing	its bus driver, over IPC

The last row is the surprising one and the whole point. A USB keyboard driver touches no registers, takes no interrupts, and maps no memory. It sends HID protocol messages to whatever published the device, and it works identically whether the controller below is xHCI, EHCI, or a Raspberry Pi's DWC2. That is what buys you drivers that outlive the hardware they were written for.

In practice HCD and bus driver are usually the same process. An xHCI driver is a host controller driver (it owns the PCI function, its BARs, its interrupt, its DMA rings) and a bus driver (it enumerates USB devices and publishes them). Splitting them is a fiction; what matters is that both roles have kernel support, because a plain bus driver with no controller — a USB hub — is also a real thing.

The device table is the spine

danos already has the right central structure. system/kernel/devices-broker.zig holds a table of DeviceDescriptor, each with a parent, a class, and a set of resources. Firmware discovery seeds it (discovery.md); device_register grows it.

Three invariants make it a capability system rather than a directory:

A claim is exclusive. device_claim(id) succeeds once. Everything downstream — mmio_map, irq_bind, device_register — checks devices_broker.ownerOf(id) == me.
A descriptor is a licence to map physical memory. Whoever claims a device may map its .memory resources and bind its .irq resources. This is why device_register cannot be a free-for-all.
Therefore: containment. Every resource of a registered child must lie inside a resource of the same kind on its parent (devices_broker.contains). A bus driver can only ever subdivide what it already holds. Without this, device_register would be a syscall named "map any physical page you like."

Containment is transitive by construction: a grandchild is contained in its child, which is contained in the bus. Nothing can be laundered through a chain.

Note that firmware topology does not obey containment, and isn't asked to — a PCI function's BAR is not inside its host bridge's bus_range, because a bus-number range is not an address window. Discovery is trusted; user space is not.

What a bus driver looks like

danos ships no demo bus driver — the real ones are pci-bus, ps2-bus, and usb-xhci-bus. The smallest honest shape, illustrated here with an HPET register block as the "bus" and its comparators as the "devices", is:

_ = dev.claim(bus.id);                       // 1. own the bus
const base = dev.mmioMap(bus.id, 0).?;       // 2. enumerate it — from the hardware
const n = ((cap.* >> 8) & 0x1F) + 1;         //    GENERAL_CAP says how many children

for (0..n) |i| {                             // 3. publish each child
    var child = std.mem.zeroes(dev.DeviceDescriptor);
    child.class = @intFromEnum(dev.DeviceClass.timer);
    child.resource_count = 1;
    child.resources[0] = .{ .kind = memory,
                            .start = bus_mmio.start + 0x100 + 0x20 * i,
                            .len   = 0x20 };
    _ = dev.register(bus.id, &child).?;      //    kernel checks containment
}

Each child is left unclaimed, which is the handoff: a comparator driver can now device_claim one and mmio_map it, and will see only its own 0x20-byte window. A child whose window escapes the bus is refused; the in-kernel containment test asserts the kernel's table upholds that (drivers.md).

A USB device has no resources at all: resource_count = 0, because it's addressed through its controller, not by MMIO. That case is allowed and is the common one.

A "family" is two modules, not one:

A logic module — the parts of the bus that every driver on it re-derives. Config space walking and BAR decode for PCI. Descriptor parsing, control transfers, and hub protocol for USB.
A protocol module — the IPC message types that let a class driver talk to whatever published its device. This is the part that makes class drivers portable.

danos already has one of each: library/device/pci/pci.zig is a logic module (the Function view of a claimed PCI function), library/protocol/vfs/vfs-protocol.zig is a protocol module shared by the mount backends (today the fat server) and their clients. (The user-space VFS server it was originally written against has since retired — path routing moved into the kernel, system/kernel/vfs.zig's fs_resolve — but the protocol module outlived it, which is rather the point.) The pattern generalises directly:

library/
  kernel/           the system library (kernel32-style): the syscall surface split by concern
                    — ipc, memory (heap/dma/shared-memory), process, time, logging,
                    file-system, thread, service, plus system-call stubs + start/root
  device/           device code grouped by domain; each domain splits into a shareable
                    data module (enums/wire types, std-only) and a logic module (mmio/IPC)
    mmio/           module "mmio"    — typed volatile register access + barriers   [M14]
    model/          module "device-abi"                    — DeviceDescriptor, DeviceClass, ResourceKind
    pci/            "pci-class" (data) + "pci"             — config/BAR/capability walk (Function)
    usb/            "usb-abi" + "usb-ids" (data) + "usb"   — descriptors, control/interrupt/bulk client
    acpi/           "acpi-ids" (data) + "aml"              — _HID names, the AML interpreter
    driver/         module "driver"                        — device-access syscalls + device-manager hello
    block/          module "block"                         — the block-device client (a device type)
  client/           userspace service clients — display, input (a program's view of a service)
  protocol/         driver <-> service wire contracts, one module per directory
    vfs/ block/ display/ scanout/ input/ power/ device-manager/ usb-transfer/

system/drivers/     one sub-project each → /system/drivers (no `d` suffix)
  usb-xhci-bus/  HCD + bus driver    imports usb, mmio, usb-transfer-protocol   (+ kernel modules)
  usb-hid/       class driver        imports usb, input-protocol                (+ kernel modules)
  virtio-gpu/    scanout driver      imports pci, mmio, display-/scanout-protocol (+ kernel modules)

The split by dependency weight is what lets the microkernel stay out of device business: it imports only the device-abi data module (the descriptor types its broker marshals across the syscall boundary) — never a logic module, never a taxonomy. That one pure-data import is the only edge from system/kernel/ into library/; decoding a class code or _HID to a name is user space's job (the device manager owns those taxonomies).

A protocol lives in library/protocol/ when it is the seam between a low-level driver and a higher-level service (block ↔ filesystem, a scanout driver ↔ the compositor). A driver's private wire to its hardware — virtio-gpu's command set — is not that; it stays a driver-private file, like the virtio-pci transport beside it.

The build side of this has since landed: every binary owns a package whose ~15-line build.zig names EXACTLY the modules its source imports — the moral equivalent of a C file's include list — and the shared recipe in build-support/build.zig (userBinary) resolves each name from the library domain that exports it (kernel's concern modules, the device driver libraries, the service clients, the protocols). An undeclared @import is a compile error, and a domain none of the imports come from never appears in the binary's manifest — a keyboard driver declares xkeyboard-config; nothing else does (see build-packages-plan.md). That's the entire mechanism — Zig modules already give you everything else.

The discipline that makes this work: a class driver must not import a bus's hardware logic module. usb-hid imports usb (the transfer client) and input-protocol, never pci and never mmio. If a class driver needs mmio, it has become an HCD and should be one. The domain data modules (usb-abi, usb-ids, pci-class) carry no such weight — a class driver, the device manager, or the kernel may share them freely.

What exists today

M10 — device_enumerate, device_claim, mmio_map. Strong-uncacheable device grants, device_grant teardown.
M11 — irq_bind / irq_ack. IRQ delivered as an IPC notification; mask before EOI; irq_ack is the unmask.
M12 — parent in DeviceDescriptor, device_register with resource containment.
M13 — capability passing. ipc_call / ipc_reply_wait grew a send_cap argument and a received_cap return (r8): an endpoint travels with a message, installed into the receiver's handle table (shared, refcount-bumped — a copy, not a move). A full table fails -ENOSPC and does not half-deliver. This is the "open" primitive — a bus driver mints a per-device endpoint and hands it to a class driver. The ipc module exposes callCap and replyWait(..., send_cap), and class drivers consume them now: the PS/2 keyboard and mouse drivers attach to ps2-bus this way, and the usb / input client modules open their per-device and subscription channels with callCap.
M14 — DMA memory + the memory-ordering layer. /lib/device/mmio gives drivers typed volatile access and memoryBarrier/readMemoryBarrier/writeMemoryBarrier (per-arch); dma_alloc/dma_free grant physically-contiguous, pinned, uncacheable, reclaim-on-teardown buffers with the physical address exposed (pmm.allocContiguous, a DMA arena, mapUserDmaInto). dma_below_4g caps the address for legacy engines; dma_write_combining is accepted but falls back to coherent until PAT is programmed. The bus drivers use /lib/device/mmio, and dma_alloc has real consumers now: the xHCI driver's rings and contexts, usb-storage's command/status wrappers, virtio-gpu's virtqueue, and the fat service's bounce buffer.
M15 — interrupts for PCI devices, the MSI half. Discovery now gives every PCI function its 4 KiB ECAM config space as resource 0 (unblocking the capability walk with no new syscall), and msi_bind(device_id, endpoint) -> address, data allocates a per-device edge-triggered vector, delivered as an IPC notification with no mask and no ack cycle. Legacy INTx (_PRT parsing + shared lines) is deliberately skipped — MSI is the real answer. QEMU's HPET has no MSI, so delivery is proven with a self-IPI; the first PCI driver is the first real consumer.
Port I/O — io_read/io_write(device_id, resource_index, offset, width[, value]): a claimed device's io_port resource lets a driver read/write its ports, gated exactly like mmio_map gates memory (direct ring-3 in/out stays a #GP). This is what makes a PS/2 or 16550 driver possible; the low-rate legacy hardware that needs it is fine with a syscall per access. io_port resources were recorded by discovery and ignored — now they're used.
M16 (detection) — the IOMMU is now found: discovery parses the ACPI DMAR table, maps the first VT-d unit, and reads its version + capabilities (iommu_present in the platform info). This is detection only — no translation domains are programmed, so DMA is still unprotected (the caveat below). Enforcement lands with the first DMA driver, which is what there is to protect and test against. Proven in the iommu test, booted with an emulated intel-iommu.
system_spawn — a user-space supervisor starts a driver: system_spawn(name, arguments) loads a binary bundled in the initial-ramdisk as a fresh ring-3 process; name becomes the child's argv[0] and the optional NUL-separated arguments blob its argv[1..], delivered on a SysV entry stack (sysv.md). This is what turned the device manager from "log the match" into "run the driver": the kernel now spawns only init, init spawns the services, and the device-manager discovers the hardware and spawns each driver (drivers.md). Ungated for now — a spawn capability is future work.

So: all three shapes work now, and they're started by the device manager, not the kernel. The xHCI driver is the HCD-and-bus proof; the USB HID/storage and PS/2 class drivers reach their devices purely over IPC. What follows are the original design notes for the primitives that unblocked each shape — exactly why each was the blocker, and exactly what fixed it.

Proposed ABI

M13 — capability passing, for class drivers ✅ done

Implemented as described below (see "What exists today"). The signatures landed verbatim: send_cap in r9, received_cap returned in r8, -ENOSPC on a full receiver table with no delivery. The rest of this section is the original design note.

The blocker. A class driver has to reach its device. Today the only way to find an endpoint is the name registry: ipc_register(service_id, h) / ipc_lookup(id), where ServiceId is a global integer namespace with max_services = 8. You cannot mint one endpoint per USB device that way, and there is no way for a bus driver to hand a class driver an endpoint. M7 deferred this deliberately.

The fix. Let a message carry one handle. Sender names a handle in its own table; the kernel installs the endpoint into the receiver's table (bumping refcount) and tells the receiver the index it landed at.

ipc_call(h, msg, message_len, reply, reply_cap, send_cap)  -> reply_len
ipc_reply_wait(h, reply, reply_len, recv, recv_cap, send_cap)
      -> recv_len (rax), badge (rdx), received_cap (r8)

send_cap is a handle or no_cap (~0). received_cap is the index the transferred endpoint was installed at in the receiver's table, or no_cap.

Both calls grow from 5 args to 6, which fits: syscall5 uses rdi/rsi/rdx/r10/r8, leaving r9. ipc_reply_wait already returns two values via setSyscallResult2; this needs a third (setSyscallResult3).
If the receiver's handle table is full, the call fails -ENOSPC and the message is not delivered — a half-delivered capability is worse than a failed send.
closeHandles already drops references on exit, so the lifetime story is unchanged.

That single primitive gives you the standard open pattern:

// class driver                          // bus driver
const h = ipc.lookup(.usb).?;            const r = ipc.replyWait(ep, ...);
const dev_ep = ipc.callCap(h,            //   ... mint a per-device endpoint,
    .{ .op = .open, .id = dev_id });     //   reply with it as send_cap
// now dev_ep is a private channel to that one device

M14 — DMA memory and the memory-ordering contract, for HCDs ✅ done

Implemented: /lib/device/mmio (typed volatile access + memoryBarrier/readMemoryBarrier/writeMemoryBarrier, per-arch) and dma_alloc/dma_free (contiguous, pinned, uncacheable, reclaim-on-teardown, physical address exposed). dma_write_combining still falls back to coherent — real WC needs PAT, a small follow-up. The rest of this section is the original design note.

The blocker. An HCD is a DMA-engine programmer. It needs a descriptor ring the device can read, which means memory that is (a) physically contiguous, (b) at a physical address the driver knows, (c) of the right cacheability, and (d) pinned. sysMmap gives you none of the four: it calls pmm.alloc() once per page, maps writeback-cached, and never reveals a physical address.

The fix.

dma_alloc(len, flags) -> virtual_address (rax), physical_address (rdx)
dma_free(virtual_address, len)  -> 0

flags:  dma_coherent  (1)  uncacheable; the default and the only one that's portable
        dma_wc        (2)  write-combining — needs PAT programmed; for framebuffers
        dma_below_4g  (4)  for devices with 32-bit DMA addressing

Guarantees: page-aligned, physically contiguous, zeroed, pinned for the life of the mapping, and the physical address is stable. It needs one thing the kernel lacks — pmm.allocContiguous(n, max_phys); today pmm.alloc() hands out one frame at a time with no adjacency guarantee.

The memory-ordering contract. danos has, at the time of writing, zero memory barriers anywhere in the tree. That is currently correct-by-accident and won't survive the first DMA driver, or the first ARM boot.

volatile is not a barrier. In Zig it means: don't elide this access, and don't reorder it against other volatile accesses. It says nothing about your ordinary stores — the descriptor you just filled in normal WB memory — which LLVM may freely sink past a volatile MMIO write. The canonical bug:

ring[i] = descriptor;       // ordinary store to WB RAM
doorbell.* = i;             // volatile store to UC MMIO
// nothing stops the compiler reordering these; the device reads a stale descriptor

So the rules, which belong in library/device/mmio/mmio.zig and behind arch:

Situation	Required
MMIO register read/write	`mmio.read` / `mmio.write` (volatile)
Fill DMA descriptor, then ring doorbell	`writeMemoryBarrier()` between them
Woken by IRQ, then read what the device wrote	`readMemoryBarrier()` before the read
MMIO write that must complete before the next read	`memoryBarrier()`

And the per-arch lowering — the reason this must be an arch primitive and not a sprinkling of asm volatile:

	x86_64	aarch64
`memoryBarrier()`	`mfence`	`dsb sy`
`readMemoryBarrier()`	`lfence`	`dsb ld`
`writeMemoryBarrier()`	`sfence`	`dsb st`
DMA cache coherency	coherent; nothing to do	not guaranteed; needs non-cacheable buffers or cache maintenance

x86 is forgiving here — TSO plus strong-uncacheable MMIO means you usually get away with a compiler barrier alone. ARM is not, and vision.md makes ARM the win condition. Build the abstraction while there is one caller to fix.

(Zig note: @fence was removed in 0.16. Use @atomicRmw(..., .seq_cst) for a full barrier, or per-arch inline asm — which is what library/device/mmio/mmio.zig should hide.)

M15 — interrupts for PCI devices ✅ done (MSI)

Implemented the MSI half: ECAM config space per PCI function (resource 0) and msi_bind (per-device edge-triggered vector, delivered as a notification). Legacy INTx _PRT parsing is skipped on purpose. msi_bind returns (address, data) as two values rather than an out-struct. The rest of this section is the original design note.

The blocker, and it's a hard one. No PCI device can take an interrupt today. addBars (then in the kernel's ACPI discovery; BAR decode has since moved to the ring-3 pci-bus driver, system/drivers/pci-bus/pci-bus.zig) records .memory and .io_port BARs and never an .irq; there is no _PRT parsing anywhere in the tree. The HPET is the one exception — it advertises its own interrupt routing in its own registers, a privilege no ordinary device has.

The fix, in two halves.

Legacy INTx: parse _PRT from the DSDT to map (device, INTA–D) → GSI, and record it as an .irq resource. Then irq_bind works unchanged. But INTx lines are shared, and irq.bound[gsi] holds one endpoint. Sharing needs a list, and every driver on the line must be polled on each interrupt — the reason everyone left INTx behind.

MSI/MSI-X, which is the real answer: per-device vectors, edge-triggered, unshared, no mask/ack cycle, no 24-GSI ceiling. The kernel allocates a vector and hands the driver the (address, data) pair to program into its own MSI capability:

msi_bind(dev_id, endpoint, out) -> 0     // out: extern struct { addr: u64, data: u32 }

The driver writes those into config space itself — which means it needs config space, which means discovery should give each pci_device a .memory resource for its 4 KiB ECAM slot. That's a small change to parseMcfg and it unblocks the whole capability walk (MSI, MSI-X, PCIe extended caps) without any new syscall.

Note QEMU's HPET reports Tn_FSB_INT_DEL_CAP = 0 — no MSI — so an HPET timer could never exercise this path. The first MSI driver will be the first PCI driver.

M16 — the IOMMU, and the honest caveat ◑ detection done, enforcement pending

The IOMMU is now detected (DMAR parsed, VT-d unit mapped and read — see the iommu test), but enforcement is not built: no translation domains are programmed, so the caveat below still holds in full. Detection can't be taken further usefully until there is a DMA driver to protect and QEMU's intel-iommu to test the protection against — building the per-device domains alongside that first driver is both the natural order and the only way to verify them. The rest of this section is the original caveat.

Everything above is capability-gated at the CPU. None of it is gated at the device. A driver that can program a bus-mastering engine can make that device write to any physical address, because page tables sit between the CPU and RAM, not between a device and RAM. Until VT-d/DMAR (or SMMU on ARM) is programmed from the DMAR table, device_claim on any DMA-capable device is equivalent to granting ring 0.

This does not make the model useless — it's the same position Linux is in with the IOMMU off, and every other guarantee (crash isolation, restart, no shared address space) still holds. But "user-space drivers are memory-safe" is not true yet, and the gap should be named rather than implied.

Ordering

M13 (capability passing) is independent of M14/M15 and is the cheapest. It unlocks class drivers, which are the shape with no hardware requirements at all — you could write a real one against any device a bus driver publishes tomorrow.

M14 and M15 together unlock the first HCD. M14's barrier layer is worth landing on its own regardless: it's small, obviously correct, and stops every future driver from hand-rolling *volatile and getting ARM wrong.

23 KiB Raw Blame History Unescape Escape