danos/docs/drivers.md

Writing a driver

In a monolithic kernel a driver is a function call away from everything: it runs in ring 0, dereferences any physical address, and its interrupt handler is the ISR. In danos a driver is an ordinary ring-3 process. It has its own address space, it can crash without taking the kernel with it, and — the point of this document — it can be restarted (resilience).

That leaves three questions the kernel has to answer, because a process can't answer them for itself:

1. What hardware exists? → device_enumerate, over the device table discovery built (discovery, acpi).
2. How do I touch its registers? → device_claim + mmio_map: the kernel maps the device's physical MMIO window into your address space, and from then on it's plain memory. No syscall per register access.
3. How do I find out it wants something? → irq_bind: the interrupt is delivered to you as an IPC notification. You block; the hardware wakes you.

A driver is, in one sentence, a process that sleeps until its device has something to say.

The capability: claim before touch

The driver syscall numbers (system/abi.zig) with the device types they carry (system/devices/device-abi.zig), dispatched in system/kernel/process.zig:

#	Call	Meaning
11	device_enumerate(buf, max) -> total	Snapshot the device table
12	device_claim(id) -> ok	Take exclusive ownership
13	mmio_map(id, res_idx) -> vaddr	Map a claimed device's register window
14	irq_bind(id, res_idx, endpoint)	Deliver that device's IRQ as a notification
15	irq_ack(id, res_idx)	Re-arm the IRQ after servicing the device
16	device_register(parent_id, desc) -> id	Publish a child of a device you claimed

Notice that nothing takes a physical address or an interrupt number. Every call names a device by id and a resource by index. That indirection is the entire security model. If mmio_map took a physical address, any process could map the kernel's memory; if irq_bind took a GSI, any process could bind the keyboard's line and silently intercept it. Instead the kernel checks two things (process.ownedGsi, and the same check at the top of sysMmioMap):

• devices_broker.ownerOf(dev_id) == me — you claimed it, and claims are exclusive
• the resource at res_idx is of the right kind — memory for mmio_map, irq for irq_bind

The claim is the capability. Everything else follows from it.

Registers: mmio_map

mmio_map walks the caller's page tables and installs the device's physical frames with present | user | writable | nx | pcd | pwt (arch/x86_64/paging.zig:mapUserDeviceInto). Two of those bits are load-bearing:

• pcd | pwt — strong-uncacheable. A device register is not memory; a cached read would return a stale value and a write might never leave the CPU.
• device_grant (bit 9, one of the PTE's available bits) — marks the leaf as MMIO rather than RAM, so freeSubtree skips pmm.free on it when the address space is destroyed. Without this, killing a driver would hand the HPET's registers back to the frame allocator as if they were free RAM. The iopass test guards it.

Grants land in their own arena, 0x0000_7100_0000_0000 (PML4[226]), so device pages never widen an existing mapping.

Then you just… use it:

const base = dev.mmioMap(dev_id, mmio_res) orelse return;
const counter: *volatile u64 = @ptrFromInt(base + 0xF0);
const now = counter.*;   // a load, straight to the hardware. no kernel involved.

Interrupts: the cycle, and why it has that shape

An interrupt handler in a microkernel has a problem. The code that knows how to quiet the device is in ring 3, in another address space, and it will not run for microseconds or milliseconds — after a context switch, when the scheduler gets to it. But the CPU wants an EOI now, and a level-triggered line stays asserted until the device is quieted. EOI a still-asserted line and the I/O APIC redelivers immediately. Forever. The driver never gets to run at all.

The way out is to mask the line before acknowledging it:

kernel ISR   irqMask(gsi)      // line still asserted; stop it reaching a CPU
             irqEoi()          // now safe to tell the LAPIC we're done
             notifyFromIsr()   // wake the driver — it runs much later

driver       replyWait() -> badge with the notify bit set
             <clear the device's status register>   // NOW the line deasserts
             irq_ack(dev, res) // kernel unmasks: quiet, so it can't refire

irq_ack is not bookkeeping you could skip. It is the unmask. Forget it and the interrupt fires exactly once, ever; call it before the device is quiet and you get an interrupt storm. That single fact explains why irq_bind and irq_ack are two syscalls and not one.

This is also why interruptDispatch (arch/x86_64/idt.zig) no longer issues the EOI itself. It used to, before running the handler — correct for the LAPIC timer, and impossible for a routed device line. Each handler now owns its EOI, because only the handler knows which discipline its source needs.

The driver side is an event loop, not a callback

IPC_ReplyWait returns either a client request or a notification, told apart by the top bit of the badge (ipc_sync.notify_badge_bit). So a driver is one single-threaded loop over both of its event sources:

while (true) {
    const r = ipc.replyWait(endpoint, reply, &recv);
    if (r.isNotification()) {            // r.source() is the GSI
        service_device();                // clear the status register
        _ = dev.irqAck(id, irq_res);     // re-arm
    } else {
        handle_client_request(recv[0..r.len]);
    }
}

No reentrancy, no "what am I allowed to call from an interrupt handler", no shared state between ISR and task context. The interrupt is just a message.

Two properties worth knowing:

• An interrupt taken while you're elsewhere is not lost. If the driver is off in an ipc_call to another server when the IRQ fires, wakeLocked finds nobody waiting, but the badge is already on the endpoint's notify ring. The next replyWait pops it (ipc_sync.replyWait checks popNotify before the sender FIFO).
• Notifications coalesce, they don't count. The ring is 8 deep and drops on overflow. That's correct: an IRQ notification is a level ("the device wants attention"), not a tally. Re-read the device's status register; never assume one notification means exactly one event. Because the ISR masks the line until you irq_ack, at most one badge per GSI can be outstanding — so the ring can only overflow if you bind more than eight GSIs to a single endpoint. Don't.

A whole driver

system/drivers/hpet/hpet.zig is ~150 lines and does all of it. The shape:

const hpet = findHpet(buf) orelse return;      // device_enumerate, look for
                                               //   class=timer with memory + irq
_ = dev.claim(hpet.dev_id);                    // the capability
const base = dev.mmioMap(hpet.dev_id, hpet.mmio).?;
const endpoint = ipc.createEndpoint().?;

// program the hardware over the mapping we were just handed
reg(base, 0x100).* = level | int_enb | (hpet.gsi << 9);   // timer 0 config
reg(base, 0x108).* = reg(base, 0xF0).* + period;          // comparator
reg(base, 0x010).* |= 1;                                  // ENABLE

_ = dev.irqBind(hpet.dev_id, hpet.irq, endpoint);

while (...) {
    const r = ipc.replyWait(endpoint, &.{}, &recv);       // blocked. not polling.
    if (r.badge & notify_bit == 0) continue;
    reg(base, 0x020).* = 1;                               // clear status -> deassert
    reg(base, 0x108).* = reg(base, 0xF0).* + period;      // re-arm
    _ = dev.irqAck(hpet.dev_id, hpet.irq);                // unmask
}

The HPET is a good first driver for a reason that isn't obvious. Its counter is a clocksource — the only way to use it is to read it, so it proved mmio_map without needing interrupts at all. Its comparators are a clockevent, and can be configured level-triggered (Tn_INT_TYPE_CNF), which asserts a bit in GENERAL_INT_STATUS that the driver must write-1-to-clear. That's a genuine deassert step, so the full mask/ack cycle above is exercised for real rather than being decoration on an edge-triggered line that would have been fine without it.

One wrinkle it also demonstrates: the ACPI HPET table carries no interrupt number. Which I/O APIC inputs a comparator may drive is a bitmask in Tn_INT_ROUTE_CAP, in the device's own registers. So discovery (acpi.parseHpet) maps the block, reads the mask, and records one concrete GSI as an irq resource. The driver then programs Tn_INT_ROUTE_CNF to raise exactly that line — and the kernel will only bind the one it recorded. Hardware that describes itself at runtime still has to fit through a static capability.

Publishing children: device_register

A device that contains other devices — a PCI bridge, a USB hub, or the HPET's block of comparators — needs a driver that enumerates it and tells the kernel what it found. That's device_register, and it makes the device table a tree rather than a list (DeviceDesc.parent).

var child = std.mem.zeroes(dev.DeviceDesc);
child.class = @intFromEnum(dev.DeviceClass.timer);
child.resource_count = 1;
child.resources[0] = .{ .kind = memory, .start = bus_base + 0x100, .len = 0x20 };
const child_id = dev.register(bus_id, &child).?;

The child is left unclaimed, which is the whole point: another process claims it and mmio_maps it, and sees only that 0x20-byte window.

The rule the kernel enforces is containment: every resource of a child must lie inside a resource of the same kind on its parent. Ranges must nest; an IRQ must match exactly. This isn't bureaucracy — a DeviceDesc is a licence to map physical memory, so without containment device_register would be a syscall for mapping any page you like. A bus driver may only ever subdivide what it already owns.

A device with no resources is legal and common. A USB device is reached through its controller, not by MMIO, so it gets resource_count = 0.

See system/drivers/bus/bus.zig for a complete one, and driver-model.md for how bus drivers, class drivers and host controller drivers fit together.

What the kernel does not do for you

• It does not quiet your device. That's the whole reason irq_ack exists.
• It does not know your registers. mmio_map hands you a base address; every offset in this document came from the HPET spec, not from danos.
• It does not serialise your driver. Two clients calling one driver endpoint are serialised by replyWait, but nothing stops your driver from being preempted.

Limits, today

Worth knowing before you write the second driver:

• Ring 3 has no port I/O. The TSS I/O permission bitmap is absent (tss.zig: iomap_base = @sizeOf(Tss)), and IOPL is never raised, so in/out from a driver is a #GP. That rules out a user-space 16550 UART (0x3F8), PS/2 (0x60/0x64), and legacy PCI config (0xCF8/0xCFC). Everything must be MMIO. io_port resources are recorded by discovery and then ignored.
• Page granularity. mmio_map rounds to 4 KiB. Two devices sharing a page means granting one grants the other. A device_registered child's resource can be narrower than a page, but its mapping can't.
• No DMA memory. mmap gives you writeback-cached, non-contiguous pages and never tells you their physical address, so you cannot build a descriptor ring. Any driver for a bus-mastering device is blocked on this.
• No memory barriers. There are none in the tree, and volatile is not one — it won't stop the compiler sinking an ordinary store (your DMA descriptor) past a volatile MMIO store (your doorbell). On x86 you mostly get away with it; on ARM you will not. See driver-model.md.
• DMA is not contained. A driver that can program a bus-mastering device can make that device write to any physical address — page tables don't sit between a device and RAM; an IOMMU does. Until VT-d/DMAR is programmed, device_claim on a DMA-capable device is effectively equivalent to granting ring 0. This is the largest gap between the design's promise and what it delivers.
• One endpoint per GSI, so shared legacy PCI INTx lines can't be split between two drivers. MSI/MSI-X — one vector per device, edge-triggered, unshared — is the real answer, and QEMU's HPET doesn't offer it (Tn_FSB_INT_DEL_CAP = 0).
• Polarity is hardcoded active-high in irq.bind. A device whose MADT override says active-low needs that threaded through from discovery.
• 14 device vectors (33–46) and 24 GSIs, bounded by the stubs isr.s emits and by a single I/O APIC.
• Don't bind more than 8 GSIs to one endpoint. The notify ring is 8 deep and drops on overflow. With one GSI per endpoint that's unreachable — the line is masked from the ISR until irq_ack, so at most one badge is ever outstanding. Bind nine devices to one endpoint, though, and a dropped badge leaves that line masked with nobody left to ack it.
• A faulting driver still kills the machine. There is no per-process kill path: a ring-3 page fault halts the kernel, so releaseIrqs runs only on a voluntary exit. Fault isolation is the whole premise (vision) and it is not built yet.
• A dead driver's device is not reclaimed. releaseIrqs unbinds and masks the line on exit, but the claim is never released — restart is not built.
• On real hardware, the mask/EOI cycle may need a remote-IRR flush. Masking a level-triggered redirection entry with remote-IRR set doesn't clear it on some chipsets, and the line never fires again. QEMU clears it on EOI regardless, so the tests can't see this. Linux flushes remote-IRR by toggling the entry to edge and back. See the note at the top of system/kernel/irq.zig.

Verifying it

The hpet test spawns hpet from the initial ramdisk and watches the serial log. The driver prints hpet: ok only after being woken five times, and its loop's only exit is through replyWait returning a notification — it cannot reach that line by polling.

The last check doesn't trust the driver's self-report at all: the kernel reads the I/O APIC redirection entry back and asserts the line really is routed to a device vector, really is level-triggered, and really was left unmasked by the driver's final irq_ack.

$ python3 test/qemu_test.py hpet irqfree iopass
  hpet         ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
  irqfree      ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
  iopass       ... PASS  (matched 'DANOS-TEST-RESULT: PASS')

Two companions cover what hpet can't, because it never exits:

• irqfree — the teardown path. Binds two owners to one shared endpoint, releases one, and reads the I/O APIC back: the departing owner's line is masked, the sibling's is not. That second half is why bindings are keyed on the owning task and not on the endpoint pointer — endpoints are shared, so releasing "everything pointing at this endpoint" would silently mask a live driver's device.
• iopass — the device_grant teardown rule, so destroying a driver's address space never returns MMIO frames to the RAM pool.

What's next (not done here)

The big ones — capability passing (class drivers), DMA + barriers and MSI (host controller drivers), and the IOMMU — have proposed signatures in driver-model.md. Smaller items:

• Port I/O grants, so a PS/2 or 16550 driver is possible: either a per-device TSS I/O permission bitmap swapped on context switch, or io_in/io_out syscalls gated by the same claim. The legacy devices that need it are all low-rate, so the syscall is likely fast enough.
• Releasing a claim. There is no dev_release, and devices_broker never drops a claim on exit — only IRQ bindings are released. A dead driver's device stays owned forever, which blocks restart.
• Unregistering children. device_register only appends. A USB device that is unplugged cannot be removed, and a bus driver in a loop can exhaust the 64-entry table.
• Restart. A driver that dies should release its claim, have its device quiesced, and be respawned by a supervisor. Some pieces (releaseIrqs, device_grant teardown, the claim table) exist; the policy doesn't.
• Interrupt priority / threaded IRQ latency. notifyFromIsr enqueues the woken driver but doesn't preempt (wakeLocked deliberately leaves that to the caller), so a woken driver waits for the next scheduling point.