daniel/danos
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

M11–M12: IRQ-as-IPC and bus drivers; expand names tree-wide

Browse Source 
Two driver-model milestones plus a tree-wide naming pass. Suite 35/35
(QEMU) + host tests green.

M11 — IRQ-as-IPC. A ring-3 driver now sleeps until its device interrupts
it. New src/kernel/irq.zig: per-GSI endpoint bindings, comptime per-vector
trampolines, dispatch = mask GSI -> LAPIC EOI -> notifyLocked, all under one
lock region. irq_bind/irq_ack syscalls, gated by the device claim like
mmio_map. interruptDispatch no longer EOIs — each handler owns its EOI,
because a level line must be masked before it is acknowledged (irq_ack is
the unmask). Bindings are keyed on the owning task and released on exit
(a shared endpoint's siblings survive). hpetd rewritten interrupt-driven.
Tests: hpet (rewritten, reads back the I/O APIC routing) and irqfree.

M12 — bus drivers. DeviceDesc gains a parent, making the device table a
tree. dev_register (device_register) lets a process publish children below
a device it claimed; the kernel enforces resource containment (a child's
resources must nest in its parent's), so a descriptor can't fabricate a
window over kernel RAM. Descriptor copied in via copyFromUser (physmap
walk — an unmapped user pointer fails the call instead of faulting the
kernel). Per-parent child cap bounds table exhaustion. sbin/busd.zig is a
worked bus driver. Test: bus.

Naming — per docs/coding-standards.md: non-acronym abbreviations spelled
out (message, descriptor, device_service, scheduler, runtime, physical,
interpreter, ...); acronyms kept (IPC, MMIO, DMA, HCD, ...); files are
kebab-case (ipc-synchronous.zig, device-service.zig, vfs-protocol.zig, ...).
Exceptions: POSIX/C ABI names and Zig idioms (init/len/ptr) kept. Module
collisions resolved by specific naming (config -> parameters, device.zig
alias -> device_model). AML op/Op disambiguated: op = opcode, Op =
operation; per-opcode parse handlers renamed opX -> parseX.

New driver docs: drivers.md, driver-model.md (bus/class/HCD shapes + the
proposed M13–M16 ABI), coding-standards.md.
This commit is contained in:
Daniel Samson
2026-07-10 11:39:56 +01:00
parent 83881641ca
commit 15b70856c9
63 changed files with 4722 additions and 2690 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/drivers.md
+319
View File
@@ -0,0 +1,319 @@
# Writing a driver
In a monolithic kernel a driver is a function call away from everything: it runs in
ring 0, dereferences any physical address, and its interrupt handler *is* the ISR. In
danos a driver is **an ordinary ring-3 process**. It has its own address space, it
can crash without taking the kernel with it, and — the point of this document — it
can be restarted ([resilience](resilience.md)).
That leaves three questions the kernel has to answer, because a process can't answer
them for itself:
1. **What hardware exists?** → `device_enumerate`, over the device table discovery built
([discovery](discovery.md), [acpi](acpi.md)).
2. **How do I touch its registers?** → `device_claim` + `mmio_map`: the kernel maps the
device's physical MMIO window into your address space, and from then on it's plain
memory. No syscall per register access.
3. **How do I find out it wants something?** → `irq_bind`: the interrupt is delivered
to you as an IPC notification. You block; the hardware wakes you.
A driver is, in one sentence, *a process that sleeps until its device has something to
say.*
## The capability: claim before touch
The five driver syscalls (`src/root.zig`, dispatched in `src/kernel/process.zig`):
| # | Call | Meaning |
|---|------|---------|
| 11 | `device_enumerate(buf, max) -> total` | Snapshot the device table |
| 12 | `device_claim(id) -> ok` | Take **exclusive** ownership |
| 13 | `mmio_map(id, res_idx) -> vaddr` | Map a claimed device's register window |
| 14 | `irq_bind(id, res_idx, endpoint)` | Deliver that device's IRQ as a notification |
| 15 | `irq_ack(id, res_idx)` | Re-arm the IRQ after servicing the device |
| 16 | `device_register(parent_id, desc) -> id` | Publish a child of a device you claimed |
Notice that **nothing takes a physical address or an interrupt number.** Every call
names a device by id and a resource by index. That indirection is the entire security
model. If `mmio_map` took a physical address, any process could map the kernel's
memory; if `irq_bind` took a GSI, any process could bind the keyboard's line and
silently intercept it. Instead the kernel checks two things (`process.ownedGsi`, and
the same check at the top of `sysMmioMap`):
- `device_service.ownerOf(dev_id) == me` — you claimed it, and claims are exclusive
- the resource at `res_idx` is of the right *kind* — `memory` for `mmio_map`, `irq`
for `irq_bind`
The claim is the capability. Everything else follows from it.
## Registers: `mmio_map`
`mmio_map` walks the caller's page tables and installs the device's physical frames
with `present | user | writable | nx | pcd | pwt`
(`arch/x86_64/paging.zig:mapUserDeviceInto`). Two of those bits are load-bearing:
- **`pcd | pwt`** — strong-uncacheable. A device register is not memory; a cached read
would return a stale value and a write might never leave the CPU.
- **`device_grant`** (bit 9, one of the PTE's available bits) — marks the leaf as MMIO
rather than RAM, so `freeSubtree` skips `pmm.free` on it when the address space is
destroyed. Without this, killing a driver would hand the HPET's registers back to
the frame allocator as if they were free RAM. The `iopass` test guards it.
Grants land in their own arena, `0x0000_7100_0000_0000` (PML4[226]), so device pages
never widen an existing mapping.
Then you just… use it:
```zig
const base = dev.mmioMap(dev_id, mmio_res) orelse return;
const counter: *volatile u64 = @ptrFromInt(base + 0xF0);
const now = counter.*; // a load, straight to the hardware. no kernel involved.
```
## Interrupts: the cycle, and why it has that shape
An interrupt handler in a microkernel has a problem. The code that knows how to quiet
the device is in ring 3, in another address space, and it will not run for
microseconds or milliseconds — after a context switch, when the scheduler gets to it.
But the CPU wants an EOI *now*, and a **level-triggered** line stays asserted until
the device is quieted. EOI a still-asserted line and the I/O APIC redelivers
immediately. Forever. The driver never gets to run at all.
The way out is to mask the line before acknowledging it:
```
kernel ISR irqMask(gsi) // line still asserted; stop it reaching a CPU
irqEoi() // now safe to tell the LAPIC we're done
notifyFromIsr() // wake the driver — it runs much later
driver replyWait() -> badge with the notify bit set
<clear the device's status register> // NOW the line deasserts
irq_ack(dev, res) // kernel unmasks: quiet, so it can't refire
```
`irq_ack` is not bookkeeping you could skip. **It is the unmask.** Forget it and the
interrupt fires exactly once, ever; call it before the device is quiet and you get an
interrupt storm. That single fact explains why `irq_bind` and `irq_ack` are two
syscalls and not one.
This is also why `interruptDispatch` (`arch/x86_64/idt.zig`) no longer issues the EOI
itself. It used to, before running the handler — correct for the LAPIC timer, and
impossible for a routed device line. Each handler now owns its EOI, because only the
handler knows which discipline its source needs.
### The driver side is an event loop, not a callback
`IPC_ReplyWait` returns *either* a client request *or* a notification, told apart by
the top bit of the badge (`ipc_sync.notify_badge_bit`). So a driver is one
single-threaded loop over both of its event sources:
```zig
while (true) {
const r = ipc.replyWait(endpoint, reply, &recv);
if (r.isNotification()) { // r.source() is the GSI
service_device(); // clear the status register
_ = dev.irqAck(id, irq_res); // re-arm
} else {
handle_client_request(recv[0..r.len]);
}
}
```
No reentrancy, no "what am I allowed to call from an interrupt handler", no shared
state between ISR and task context. The interrupt is just a message.
Two properties worth knowing:
- **An interrupt taken while you're elsewhere is not lost.** If the driver is off in
an `ipc_call` to another server when the IRQ fires, `wakeLocked` finds nobody
waiting, but the badge is already on the endpoint's notify ring. The next
`replyWait` pops it (`ipc_sync.replyWait` checks `popNotify` before the sender FIFO).
- **Notifications coalesce, they don't count.** The ring is 8 deep and drops on
overflow. That's correct: an IRQ notification is a *level* ("the device wants
attention"), not a tally. Re-read the device's status register; never assume one
notification means exactly one event. Because the ISR masks the line until you
`irq_ack`, at most one badge per GSI can be outstanding — so the ring can only
overflow if you bind more than eight GSIs to a single endpoint. Don't.
## A whole driver
`sbin/hpetd.zig` is ~150 lines and does all of it. The shape:
```zig
const hpet = findHpet(buf) orelse return; // device_enumerate, look for
// class=timer with memory + irq
_ = dev.claim(hpet.dev_id); // the capability
const base = dev.mmioMap(hpet.dev_id, hpet.mmio).?;
const endpoint = ipc.createEndpoint().?;
// program the hardware over the mapping we were just handed
reg(base, 0x100).* = level | int_enb | (hpet.gsi << 9); // timer 0 config
reg(base, 0x108).* = reg(base, 0xF0).* + period; // comparator
reg(base, 0x010).* |= 1; // ENABLE
_ = dev.irqBind(hpet.dev_id, hpet.irq, endpoint);
while (...) {
const r = ipc.replyWait(endpoint, &.{}, &recv); // blocked. not polling.
if (r.badge & notify_bit == 0) continue;
reg(base, 0x020).* = 1; // clear status -> deassert
reg(base, 0x108).* = reg(base, 0xF0).* + period; // re-arm
_ = dev.irqAck(hpet.dev_id, hpet.irq); // unmask
}
```
The HPET is a good first driver for a reason that isn't obvious. Its *counter* is a
clocksource — the only way to use it is to read it, so it proved `mmio_map` without
needing interrupts at all. Its *comparators* are a clockevent, and can be configured
**level-triggered** (`Tn_INT_TYPE_CNF`), which asserts a bit in `GENERAL_INT_STATUS`
that the driver must write-1-to-clear. That's a genuine deassert step, so the full
mask/ack cycle above is exercised for real rather than being decoration on an
edge-triggered line that would have been fine without it.
One wrinkle it also demonstrates: the ACPI HPET table carries **no interrupt number**.
Which I/O APIC inputs a comparator may drive is a bitmask in `Tn_INT_ROUTE_CAP`, in
the device's own registers. So discovery (`acpi.parseHpet`) maps the block, reads the
mask, and records one concrete GSI as an `irq` resource. The driver then programs
`Tn_INT_ROUTE_CNF` to raise exactly that line — and the kernel will only bind the one
it recorded. Hardware that describes itself at runtime still has to fit through a
static capability.
## Publishing children: `device_register`
A device that *contains other devices* — a PCI bridge, a USB hub, or the HPET's block
of comparators — needs a driver that enumerates it and tells the kernel what it found.
That's `device_register`, and it makes the device table a tree rather than a list
(`DeviceDesc.parent`).
```zig
var child = std.mem.zeroes(dev.DeviceDesc);
child.class = @intFromEnum(dev.DeviceClass.timer);
child.resource_count = 1;
child.resources[0] = .{ .kind = memory, .start = bus_base + 0x100, .len = 0x20 };
const child_id = dev.register(bus_id, &child).?;
```
The child is left **unclaimed**, which is the whole point: another process claims it and
`mmio_map`s it, and sees only that 0x20-byte window.
The rule the kernel enforces is **containment**: every resource of a child must lie
inside a resource of the same kind on its parent. Ranges must nest; an IRQ must match
exactly. This isn't bureaucracy — a `DeviceDesc` is a licence to map physical memory, so
without containment `device_register` would be a syscall for mapping any page you like. A
bus driver may only ever subdivide what it already owns.
A device with **no resources** is legal and common. A USB device is reached through its
controller, not by MMIO, so it gets `resource_count = 0`.
See [`sbin/busd.zig`](../sbin/busd.zig) for a complete one, and
[driver-model.md](driver-model.md) for how bus drivers, class drivers and host
controller drivers fit together.
## What the kernel does not do for you
- **It does not quiet your device.** That's the whole reason `irq_ack` exists.
- **It does not know your registers.** `mmio_map` hands you a base address; every
offset in this document came from the HPET spec, not from danos.
- **It does not serialise your driver.** Two clients calling one driver endpoint are
serialised by `replyWait`, but nothing stops your driver from being preempted.
## Limits, today
Worth knowing before you write the second driver:
- **Ring 3 has no port I/O.** The TSS I/O permission bitmap is absent
(`tss.zig`: `iomap_base = @sizeOf(Tss)`), and IOPL is never raised, so `in`/`out`
from a driver is a #GP. That rules out a user-space 16550 UART (`0x3F8`), PS/2
(`0x60`/`0x64`), and legacy PCI config (`0xCF8`/`0xCFC`). Everything must be MMIO.
`io_port` resources are recorded by discovery and then ignored.
- **Page granularity.** `mmio_map` rounds to 4 KiB. Two devices sharing a page means
granting one grants the other. A `device_register`ed child's *resource* can be narrower
than a page, but its *mapping* can't.
- **No DMA memory.** `mmap` gives you writeback-cached, non-contiguous pages and never
tells you their physical address, so you cannot build a descriptor ring. Any driver
for a bus-mastering device is blocked on this.
- **No memory barriers.** There are none in the tree, and `volatile` is not one — it
won't stop the compiler sinking an ordinary store (your DMA descriptor) past a
volatile MMIO store (your doorbell). On x86 you mostly get away with it; on ARM you
will not. See [driver-model.md](driver-model.md#m14).
- **DMA is not contained.** A driver that can program a bus-mastering device can make
that device write to *any* physical address — page tables don't sit between a device
and RAM; an IOMMU does. Until VT-d/DMAR is programmed, `device_claim` on a DMA-capable
device is effectively equivalent to granting ring 0. This is the largest gap between
the design's promise and what it delivers.
- **One endpoint per GSI**, so shared legacy PCI INTx lines can't be split between two
drivers. MSI/MSI-X — one vector per device, edge-triggered, unshared — is the real
answer, and QEMU's HPET doesn't offer it (`Tn_FSB_INT_DEL_CAP = 0`).
- **Polarity is hardcoded** active-high in `irq.bind`. A device whose MADT override
says active-low needs that threaded through from discovery.
- **14 device vectors** (33–46) and **24 GSIs**, bounded by the stubs `isr.s` emits and
by a single I/O APIC.
- **Don't bind more than 8 GSIs to one endpoint.** The notify ring is 8 deep and drops
on overflow. With one GSI per endpoint that's unreachable — the line is masked from
the ISR until `irq_ack`, so at most one badge is ever outstanding. Bind nine devices
to one endpoint, though, and a dropped badge leaves that line masked with nobody
left to ack it.
- **A faulting driver still kills the machine.** There is no per-process kill path: a
ring-3 page fault halts the kernel, so `releaseIrqs` runs only on a voluntary
`exit`. Fault isolation is the whole premise ([vision](vision.md)) and it is
[not built yet](resilience.md).
- **A dead driver's device is not reclaimed.** `releaseIrqs` unbinds and masks the
line on exit, but the claim is never released — restart is
[not built](resilience.md).
- **On real hardware, the mask/EOI cycle may need a remote-IRR flush.** Masking a
level-triggered redirection entry with remote-IRR set doesn't clear it on some
chipsets, and the line never fires again. QEMU clears it on EOI regardless, so the
tests can't see this. Linux flushes remote-IRR by toggling the entry to edge and
back. See the note at the top of `src/kernel/irq.zig`.
## Verifying it
The `hpet` test spawns `hpetd` from the initrd and watches the serial log. The driver
prints `hpetd: ok` only after being woken five times, and its loop's only exit is
through `replyWait` returning a notification — it cannot reach that line by polling.
The last check doesn't trust the driver's self-report at all: the kernel reads the I/O
APIC redirection entry back and asserts the line really is routed to a device vector,
really is level-triggered, and really was left unmasked by the driver's final
`irq_ack`.
```
$ python3 test/qemu_test.py hpet irqfree iopass
hpet ... PASS (matched 'DANOS-TEST-RESULT: PASS')
irqfree ... PASS (matched 'DANOS-TEST-RESULT: PASS')
iopass ... PASS (matched 'DANOS-TEST-RESULT: PASS')
```
Two companions cover what `hpetd` can't, because it never exits:
- **`irqfree`** — the teardown path. Binds two owners to one shared endpoint, releases
one, and reads the I/O APIC back: the departing owner's line is masked, the sibling's
is not. That second half is why bindings are keyed on the owning *task* and not on
the endpoint pointer — endpoints are shared, so releasing "everything pointing at
this endpoint" would silently mask a live driver's device.
- **`iopass`** — the `device_grant` teardown rule, so destroying a driver's address
space never returns MMIO frames to the RAM pool.
## What's next (not done here)
The big ones — capability passing (class drivers), DMA + barriers and MSI (host
controller drivers), and the IOMMU — have proposed signatures in
[driver-model.md](driver-model.md). Smaller items:
- **Port I/O grants**, so a PS/2 or 16550 driver is possible: either a per-device TSS
I/O permission bitmap swapped on context switch, or `io_in`/`io_out` syscalls gated
by the same claim. The legacy devices that need it are all low-rate, so the syscall
is likely fast enough.
- **Releasing a claim.** There is no `dev_release`, and `device_service` never drops a claim on
exit — only IRQ bindings are released. A dead driver's device stays owned forever,
which blocks restart.
- **Unregistering children.** `device_register` only appends. A USB device that is
unplugged cannot be removed, and a bus driver in a loop can exhaust the 64-entry
table.
- **Restart.** A driver that dies should release its claim, have its device quiesced,
and be respawned by a supervisor. Some pieces (`releaseIrqs`, `device_grant`
teardown, the claim table) exist; the policy doesn't.
- **Interrupt priority / threaded IRQ latency.** `notifyFromIsr` enqueues the woken
driver but doesn't preempt (`wakeLocked` deliberately leaves that to the caller), so
a woken driver waits for the next scheduling point.