danos/docs/drivers.md

# Writing a driver

In a monolithic kernel a driver is a function call away from everything: it runs in
ring 0, dereferences any physical address, and its interrupt handler *is* the ISR. In
danos a driver is **an ordinary ring-3 process**. It has its own address space, it
can crash without taking the kernel with it, and — the point of this document — it
can be restarted ([resilience](resilience.md)).

That leaves three questions the kernel has to answer, because a process can't answer
them for itself:

1. **What hardware exists?** → `device_enumerate`, over the device table discovery built
   ([discovery](discovery.md), [acpi](acpi.md)).
2. **How do I touch its registers?** → `device_claim` + `mmio_map`: the kernel maps the
   device's physical MMIO window into your address space, and from then on it's plain
   memory. No syscall per register access.
3. **How do I find out it wants something?** → `irq_bind`: the interrupt is delivered
   to you as an IPC notification. You block; the hardware wakes you.

A driver is, in one sentence, *a process that sleeps until its device has something to
say.*

## How a driver gets started: discover, match, spawn

Nothing in the kernel decides that the PCI host bridge needs the `pci-bus` driver — that
is policy, and policy lives in user space. Boot brings user space up as a three-level
supervision hierarchy, each level owning one job:

```
kernel  ──spawns──►  init (PID 1)  ──spawns──►  device-manager  ──spawns──►  pci-bus
  |                     |                          |
  spawns only init,     the service supervisor:    the driver supervisor: enumerates
  publishes the         starts the system          /system/devices, matches each device
  initial-ramdisk       services (vfs, the         to a driver, and system_spawn's it
  so user space can     device-manager). Its
  system_spawn from it  list is init policy.
```

The kernel launches exactly one process — `init` — and hands it nothing but the raw
ability to start more (`system_spawn(name, arguments)`, which loads a binary bundled
in the initial-ramdisk as a fresh ring-3 process — `name` becoming its argv[0],
the optional arguments its argv[1..], on a SysV entry stack, see sysv.md). Everything else is a user-space decision:

- **init** ([system/services/init](system/services/init/init.zig)) is the **service
  supervisor**. It spawns the system services danos brings up at boot — today `vfs` and
  the `device-manager` — from a small list. Drivers are deliberately *not* its job.
- **device-manager** ([system/services/device-manager](system/services/device-manager/device-manager.zig))
  is the **driver supervisor**. It does the three steps a monolithic kernel would do in
  its probe path, entirely from ring 3:
  1. **Discover** — `device_enumerate` snapshots the device table the kernel built from
     ACPI/PCI ([discovery](discovery.md)).
  2. **Match** — for each device it looks up a driver by `DeviceClass`. The match policy
     is a table (`driverFor`): today a static `timer → hpet` map; a fuller system reads
     what each driver *binds* (a manifest under `/system/drivers`, or the driver
     describing its own match).
  3. **Spawn** — `system_spawn(driver_name, arguments)` starts the matched driver (the
     arguments can carry *which* device it matched), which then claims
     its device and runs the event loop below.

So "how is a driver discovered and configured" has two halves: **discovery** is the
kernel's device table, read by anyone; **configuration** is two user-space policies —
init's service list and the device-manager's match table. Both are hardcoded in their
respective programs today; the natural next step is to move them into `/etc` (see the
milestone notes in [driver-model.md](driver-model.md)). `system_spawn` is currently
ungated — any process may spawn any bundled binary — because there is no spawn
capability yet.

## The capability: claim before touch

The driver syscall numbers (`system/abi.zig`) with the device types they carry
(`system/devices/device-abi.zig`), dispatched in `system/kernel/process.zig`:

| # | Call | Meaning |
|---|------|---------|
| 11 | `device_enumerate(buf, max) -> total` | Snapshot the device table |
| 12 | `device_claim(id) -> ok` | Take **exclusive** ownership |
| 13 | `mmio_map(id, res_idx) -> vaddr` | Map a claimed device's register window |
| 14 | `irq_bind(id, res_idx, endpoint)` | Deliver that device's IRQ as a notification |
| 15 | `irq_ack(id, res_idx)` | Re-arm the IRQ after servicing the device |
| 16 | `device_register(parent_id, desc) -> id` | Publish a child of a device you claimed |

Notice that **nothing takes a physical address or an interrupt number.** Every call
names a device by id and a resource by index. That indirection is the entire security
model. If `mmio_map` took a physical address, any process could map the kernel's
memory; if `irq_bind` took a GSI, any process could bind the keyboard's line and
silently intercept it. Instead the kernel checks two things (`process.ownedGsi`, and
the same check at the top of `sysMmioMap`):

- `devices_broker.ownerOf(dev_id) == me` — you claimed it, and claims are exclusive
- the resource at `res_idx` is of the right *kind* — `memory` for `mmio_map`, `irq`
  for `irq_bind`

The claim is the capability. Everything else follows from it.

## Registers: `mmio_map`

`mmio_map` walks the caller's page tables and installs the device's physical frames
with `present | user | writable | nx | pcd | pwt`
(`arch/x86_64/paging.zig:mapUserDeviceInto`). Two of those bits are load-bearing:

- **`pcd | pwt`** — strong-uncacheable. A device register is not memory; a cached read
  would return a stale value and a write might never leave the CPU.
- **`device_grant`** (bit 9, one of the PTE's available bits) — marks the leaf as MMIO
  rather than RAM, so `freeSubtree` skips `pmm.free` on it when the address space is
  destroyed. Without this, killing a driver would hand the HPET's registers back to
  the frame allocator as if they were free RAM. The `iopass` test guards it.

Grants land in their own arena, `0x0000_7100_0000_0000` (PML4[226]), so device pages
never widen an existing mapping.

Then you just… use it:

```zig
const base = dev.mmioMap(dev_id, mmio_res) orelse return;
const counter: *volatile u64 = @ptrFromInt(base + 0xF0);
const now = counter.*;   // a load, straight to the hardware. no kernel involved.
```

## Interrupts: the cycle, and why it has that shape

An interrupt handler in a microkernel has a problem. The code that knows how to quiet
the device is in ring 3, in another address space, and it will not run for
microseconds or milliseconds — after a context switch, when the scheduler gets to it.
But the CPU wants an EOI *now*, and a **level-triggered** line stays asserted until
the device is quieted. EOI a still-asserted line and the I/O APIC redelivers
immediately. Forever. The driver never gets to run at all.

The way out is to mask the line before acknowledging it:

```
kernel ISR   irqMask(gsi)      // line still asserted; stop it reaching a CPU
             irqEoi()          // now safe to tell the LAPIC we're done
             notifyFromIsr()   // wake the driver — it runs much later

driver       replyWait() -> badge with the notify bit set
             <clear the device's status register>   // NOW the line deasserts
             irq_ack(dev, res) // kernel unmasks: quiet, so it can't refire

```

`irq_ack` is not bookkeeping you could skip. **It is the unmask.** Forget it and the
interrupt fires exactly once, ever; call it before the device is quiet and you get an
interrupt storm. That single fact explains why `irq_bind` and `irq_ack` are two
syscalls and not one.

This is also why `interruptDispatch` (`arch/x86_64/idt.zig`) no longer issues the EOI
itself. It used to, before running the handler — correct for the LAPIC timer, and
impossible for a routed device line. Each handler now owns its EOI, because only the
handler knows which discipline its source needs.

### The driver side is an event loop, not a callback

`IPC_ReplyWait` returns *either* a client request *or* a notification, told apart by
the top bit of the badge (`ipc_sync.notify_badge_bit`). So a driver is one
single-threaded loop over both of its event sources:

```zig
while (true) {
    const r = ipc.replyWait(endpoint, reply, &recv);
    if (r.isNotification()) {            // r.source() is the GSI
        service_device();                // clear the status register
        _ = dev.irqAck(id, irq_res);     // re-arm
    } else {
        handle_client_request(recv[0..r.len]);
    }
}
```

No reentrancy, no "what am I allowed to call from an interrupt handler", no shared
state between ISR and task context. The interrupt is just a message.

Two properties worth knowing:

- **An interrupt taken while you're elsewhere is not lost.** If the driver is off in
  an `ipc_call` to another server when the IRQ fires, `wakeLocked` finds nobody
  waiting, but the badge is already on the endpoint's notify ring. The next
  `replyWait` pops it (`ipc_sync.replyWait` checks `popNotify` before the sender FIFO).
- **Notifications coalesce, they don't count.** The ring is 8 deep and drops on
  overflow. That's correct: an IRQ notification is a *level* ("the device wants
  attention"), not a tally. Re-read the device's status register; never assume one
  notification means exactly one event. Because the ISR masks the line until you
  `irq_ack`, at most one badge per GSI can be outstanding — so the ring can only
  overflow if you bind more than eight GSIs to a single endpoint. Don't.

## A whole driver

A minimal leaf driver is only ~150 lines and does all of it. danos ships **no such
example binary** — the driver model is proven by the real drivers (`pci-bus`, `ps2-bus`,
`usb-xhci-bus`), and a teaching example belongs here, in the docs, rather than as a
compiled program nobody runs. Illustrated with a hypothetical HPET timer driver, the
shape is:

```zig
const hpet = findHpet(buf) orelse return;      // device_enumerate, look for
                                               //   class=timer with memory + irq
_ = dev.claim(hpet.dev_id);                    // the capability
const base = dev.mmioMap(hpet.dev_id, hpet.mmio).?;
const endpoint = ipc.createIpcEndpoint().?;

// program the hardware over the mapping we were just handed
reg(base, 0x100).* = level | int_enb | (hpet.gsi << 9);   // timer 0 config
reg(base, 0x108).* = reg(base, 0xF0).* + period;          // comparator
reg(base, 0x010).* |= 1;                                  // ENABLE

_ = dev.irqBind(hpet.dev_id, hpet.irq, endpoint);

while (...) {
    const r = ipc.replyWait(endpoint, &.{}, &recv);       // blocked. not polling.
    if (r.badge & notify_bit == 0) continue;
    reg(base, 0x020).* = 1;                               // clear status -> deassert
    reg(base, 0x108).* = reg(base, 0xF0).* + period;      // re-arm
    _ = dev.irqAck(hpet.dev_id, hpet.irq);                // unmask
}
```

The HPET makes a good illustration for a reason that isn't obvious. Its *counter* is a
clocksource — the only way to use it is to read it, so it exercises `mmio_map` without
needing interrupts at all. Its *comparators* are a clockevent, and can be configured
**level-triggered** (`Tn_INT_TYPE_CNF`), which asserts a bit in `GENERAL_INT_STATUS`
that the driver must write-1-to-clear. That's a genuine deassert step, so the full
mask/ack cycle above is exercised for real rather than being decoration on an
edge-triggered line that would have been fine without it.

One wrinkle it also demonstrates: the ACPI HPET table carries **no interrupt number**.
Which I/O APIC inputs a comparator may drive is a bitmask in `Tn_INT_ROUTE_CAP`, in
the device's own registers. So discovery (`acpi.parseHpet`) maps the block, reads the
mask, and records one concrete GSI as an `irq` resource. The driver then programs
`Tn_INT_ROUTE_CNF` to raise exactly that line — and the kernel will only bind the one
it recorded. Hardware that describes itself at runtime still has to fit through a
static capability.

## Publishing children: `device_register`

A device that *contains other devices* — a PCI bridge, a USB hub, or the HPET's block
of comparators — needs a driver that enumerates it and tells the kernel what it found.
That's `device_register`, and it makes the device table a tree rather than a list
(`DeviceDesc.parent`).

```zig
var child = std.mem.zeroes(dev.DeviceDesc);
child.class = @intFromEnum(dev.DeviceClass.timer);
child.resource_count = 1;
child.resources[0] = .{ .kind = memory, .start = bus_base + 0x100, .len = 0x20 };
const child_id = dev.register(bus_id, &child).?;
```

The child is left **unclaimed**, which is the whole point: another process claims it and
`mmio_map`s it, and sees only that 0x20-byte window.

The rule the kernel enforces is **containment**: every resource of a child must lie
inside a resource of the same kind on its parent. Ranges must nest; an IRQ must match
exactly. This isn't bureaucracy — a `DeviceDesc` is a licence to map physical memory, so
without containment `device_register` would be a syscall for mapping any page you like. A
bus driver may only ever subdivide what it already owns.

A device with **no resources** is legal and common. A USB device is reached through its
controller, not by MMIO, so it gets `resource_count = 0`.

See [`system/drivers/pci-bus/pci-bus.zig`](../system/drivers/pci-bus/pci-bus.zig) for a
real one — it claims a PCI host bridge, maps its ECAM window, and publishes each function
it finds as a child — and [driver-model.md](driver-model.md) for how bus drivers, class
drivers and host controller drivers fit together.

## What the kernel does not do for you

- **It does not quiet your device.** That's the whole reason `irq_ack` exists.
- **It does not know your registers.** `mmio_map` hands you a base address; every
  offset in this document came from the HPET spec, not from danos.
- **It does not serialise your driver.** Two clients calling one driver endpoint are
  serialised by `replyWait`, but nothing stops your driver from being preempted.

## Limits, today

Worth knowing before you write the second driver:

Several things this list used to warn about are now available (see
[driver-model.md](driver-model.md)): **port I/O** (`io_read`/`io_write`, claim-gated by
the device's `io_port` resource — direct ring-3 `in`/`out` is still a #GP, so a PS/2 or
16550 driver goes through these), **DMA memory** (`dma_alloc`: contiguous, pinned,
uncacheable, physical address exposed), and **memory barriers** (`/lib/mmio`'s
`mb`/`rmb`/`wmb`). What remains:

- **Page granularity.** `mmio_map` rounds to 4 KiB. Two devices sharing a page means
  granting one grants the other. A `device_register`ed child's *resource* can be narrower
  than a page, but its *mapping* can't.
- **DMA is not contained.** A driver that can program a bus-mastering device can make
  that device write to *any* physical address — page tables don't sit between a device
  and RAM; an IOMMU does. The IOMMU is now *detected* (M16), but no translation domains
  are programmed, so `device_claim` on a DMA-capable device is still effectively
  equivalent to granting ring 0. This is the largest gap between the design's promise and
  what it delivers; enforcement lands with the first DMA driver.
- **No `dev_release`.** A claim is never dropped (only IRQ/MSI bindings are, on exit), so
  a device stays owned for the life of its driver — which blocks restart.
- **One endpoint per GSI**, so shared legacy PCI INTx lines can't be split between two
  drivers. MSI/MSI-X — one vector per device, edge-triggered, unshared — is the real
  answer, and QEMU's HPET doesn't offer it (`Tn_FSB_INT_DEL_CAP = 0`).
- **Polarity is hardcoded** active-high in `irq.bind`. A device whose MADT override
  says active-low needs that threaded through from discovery.
- **14 device vectors** (33–46) and **24 GSIs**, bounded by the stubs `isr.s` emits and
  by a single I/O APIC.
- **Don't bind more than 8 GSIs to one endpoint.** The notify ring is 8 deep and drops
  on overflow. With one GSI per endpoint that's unreachable — the line is masked from
  the ISR until `irq_ack`, so at most one badge is ever outstanding. Bind nine devices
  to one endpoint, though, and a dropped badge leaves that line masked with nobody
  left to ack it.
- **A faulting driver still kills the machine.** There is no per-process kill path: a
  ring-3 page fault halts the kernel, so `releaseIrqs` runs only on a voluntary
  `exit`. Fault isolation is the whole premise ([vision](vision.md)) and it is
  [not built yet](resilience.md).
- **A dead driver's device is not reclaimed.** `releaseIrqs` unbinds and masks the
  line on exit, but the claim is never released — restart is
  [not built](resilience.md).
- **On real hardware, the mask/EOI cycle may need a remote-IRR flush.** Masking a
  level-triggered redirection entry with remote-IRR set doesn't clear it on some
  chipsets, and the line never fires again. QEMU clears it on EOI regardless, so the
  tests can't see this. Linux flushes remote-IRR by toggling the entry to edge and
  back. See the note at the top of `system/kernel/irq.zig`.

## Verifying it

No demo driver ships to prove this end to end; the *real* drivers do, so the tests
target them and the kernel primitives directly:

- **`device-manager`** — boots only the device manager, which discovers the PCI host
  bridge, matches `pci-bus`, and `system_spawn`s it. The test reads kernel state — the
  process table and the device tree — to confirm pci-bus came up and registered the
  functions it enumerated: the whole discover → match → spawn → driver-up chain.
- **`acpi-ps2`** — a user-space driver (`ps2-bus`) is woken by its device's IRQ,
  delivered as an IPC notification, and attaches the keyboard: IRQ-as-IPC, end to end.
- **`pci-scan`** — a user-space driver (`pci-bus`) maps its device's MMIO (the ECAM
  window) and walks it: `mmio_map`, end to end.
- **`containment`** — the kernel refuses a `device_register` whose child window escapes
  the parent's grant (else it would be a syscall for mapping arbitrary memory), while an
  identical re-register stays idempotent. Asserted in-kernel, straight against the broker.
- **`irqfree`** — the teardown path. Binds two owners to one shared endpoint, releases
  one, and reads the I/O APIC back: the departing owner's line is masked, the sibling's
  is not. That second half is why bindings are keyed on the owning *task* and not on the
  endpoint pointer — endpoints are shared, so releasing "everything pointing at this
  endpoint" would silently mask a live driver's device.
- **`iopass`** — the `device_grant` teardown rule, so destroying a driver's address
  space never returns MMIO frames to the RAM pool.

```
$ python3 test/qemu_test.py device-manager acpi-ps2 pci-scan containment irqfree iopass
  device-manager ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
  acpi-ps2     ... PASS
  pci-scan     ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
  containment  ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
  irqfree      ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
  iopass       ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
```

## What's next (not done here)

The big driver-model pieces — capability passing (class drivers), DMA + barriers, MSI,
and IOMMU detection — are **now done** ([driver-model.md](driver-model.md), M13–M16), as
is **port I/O** (`io_read`/`io_write`, the claim-gated syscalls that make a PS/2 or 16550
driver possible). What's left is IOMMU *enforcement* (per-device domains — it waits on
the first DMA driver to protect and test against) and these smaller items:

- **Releasing a claim.** There is no `dev_release`, and `devices_broker` never drops a claim on
  exit — only IRQ bindings are released. A dead driver's device stays owned forever,
  which blocks restart.
- **Unregistering children.** `device_register` only appends. A USB device that is
  unplugged cannot be removed, and a bus driver in a loop can exhaust the 64-entry
  table.
- **Restart.** A supervisor that *spawns* drivers now exists — the device-manager starts
  them with `system_spawn` — but a supervisor that *restarts* them does not. A driver that
  dies should release its claim, have its device quiesced, and be respawned; today nothing
  notices the death. Some pieces (`releaseIrqs`, `device_grant` teardown, the claim table)
  exist, and `dev_release` (below) is the missing mechanism; the restart policy is the
  resilience track ([resilience.md](resilience.md)).
- **Interrupt priority / threaded IRQ latency.** `notifyFromIsr` enqueues the woken
  driver but doesn't preempt (`wakeLocked` deliberately leaves that to the caller), so
  a woken driver waits for the next scheduling point.

## The driver contract (M17–M18)

Claiming and mapping is half of being a danos driver; the other half is the
**lifecycle and protocol contract**, and the runtime makes it nearly free:

- Build on `runtime.service.run` — one replyWait loop folding protocol
  requests, signals, and notifications into callbacks. The harness answers the
  universal zero-length ping and turns `terminate` into a clean exit for you
  ([process-lifecycle.md](process-lifecycle.md)).
- A driver spawned with an assignment (its device id as argv[1]) sends the
  versioned `hello` to the device manager inside the deadline, and a **bus**
  driver reports what it discovers with `child_added`
  ([device-manager.md](device-manager.md); usb-xhci-bus is the reference
  implementation).
- Crash freely — that is the design. The kernel releases your claims, IRQ
  bindings, and MSI vectors at death; the manager reads your exit reason,
  prunes what you reported, restarts you with backoff, and your fresh instance
  re-claims and re-reports. Never depend on your own cleanup running
  (iron rule 1).