danos/docs/device-driver-development/device-interrupts.md

# Device interrupts

CPU exceptions ([interrupts.md](../os-development/interrupts.md)) are the kernel reacting to its own
mistakes. **Device interrupts** are the opposite: hardware asking for attention —
a timer firing, a key pressed, a packet arriving. They share the IDT, but differ
in one fundamental way: an exception here is terminal (we report and halt), while a
device interrupt is *handled and returned from*, so the interrupted code resumes as
if nothing happened. This is danos's first code that takes an interrupt and comes
back — the same mechanism a scheduler will later use to preempt tasks.

The first device we bring up is the **timer**, because it's the simplest: it lives
entirely on the CPU's local interrupt controller, needing no external routing.
It's all x86_64-specific, behind the [architecture](../os-development/architecture.md) boundary.

## The APIC, not the PIC

Interrupt delivery on modern x86 goes through the **APIC**, not the legacy 8259
PIC. There are two halves; we only need one so far:

- The **Local APIC** (per-CPU, memory-mapped at physical `0xFEE00000`) handles the
  CPU's own timer and receives interrupts routed to it. `system/kernel/architecture/x86_64/apic.zig`.
- The **IO-APIC** routes *external* device lines (keyboard, etc.) to LAPIC vectors.
  Not needed for the timer — it'll arrive with the keyboard.

The old PIC has to be dealt with first, though: left alone it would deliver
interrupts on vectors `0x08-0x0F`, which **collide with the CPU exception
vectors** — a spurious IRQ would look like a double fault. So `init` remaps the
PIC's vectors to `0x20-0x2F` and masks every line, taking it out of the picture.

Then the LAPIC is enabled in two places: the `IA32_APIC_BASE` MSR's global-enable
bit, and the LAPIC's own spurious-vector register (bit 8 = software enable). The
spurious vector is `0x2F` — low nibble `F` by convention, and inside our gate
range so a stray spurious interrupt lands on a valid no-op.

## The timer

The LAPIC timer is three register writes (`initTimer`): a divide setting, then the
LVT-timer entry giving it a **vector** (32) and **periodic** mode, then an initial
count that becomes the reload value. From then on it fires vector 32 repeatedly, on
its own, forever.

The reload count isn't picked arbitrarily — it's **calibrated to real time**,
which the [real-time](../vision.md) scheduling guarantees depend on. Since the LAPIC
timer's raw rate is bus-clock dependent and unknown up front, `calibrate` runs the
LAPIC timer one-shot from its maximum count while a **reference clock** counts out a
known 10 ms, then sees how far the LAPIC got — its counts-per-millisecond, from which
`initTimer(hz)` computes the reload count for any target frequency. danos runs it at
**1000 Hz** (a 1 ms tick).

The reference clock is chosen in order of preference, so danos calibrates on
legacy-free **UEFI Class 3** hardware where the old 8254 PIT may be *absent* (polling
a missing PIT would hang the boot):

1. **CPUID leaf 0x15** — the CPU's TSC frequency directly, needing no external timer
   at all (the LAPIC is then measured against the TSC).
2. The **HPET**, discovered via ACPI (see [discovery](../os-development/discovery.md) / [acpi](../os-development/acpi.md)).
3. The **ACPI PM timer** (a fixed 3.579545 MHz counter from the FADT).
4. The **PIT** (legacy 8254, 1.193182 MHz) — last resort, and bounded so it can't hang.

All four yield the same rate; on QEMU (no CPUID crystal enumeration) it lands on the
HPET, matching the PIT numbers to within measurement jitter.

## The high-resolution clock (TSC)

The timer tick gives *scheduling* — a 1 ms quantum — but 1 ms is coarse for a
real-time system to *measure* with (interrupt latency, jitter, timeouts). So the
same calibration also measures the **TSC** (Time Stamp Counter): a per-core cycle
counter read with `rdtsc` in a couple of cycles, giving roughly **nanosecond**
resolution — a million times finer than the tick. We snapshot the TSC across the
same 10 ms calibration window to get its frequency (measured ~1 GHz under QEMU).

The monotonic clock is exposed as one function per resolution — `nanos()`,
`micros()`, `millis()` — each scaling the cycle delta directly at its unit (with a
128-bit intermediate so a long uptime doesn't overflow) rather than chaining
divisions. `millis()` is what the scheduler uses for `sleep` deadlines; `nanos()`
is there for fine measurement. Note the two clocks are distinct: the **tick** drives
preemption and wakeups (1 ms granularity); the **TSC** is the resolution you read
time at. Making `sleep` itself sub-millisecond would take a tickless one-shot
timer — a later step.

### Is the TSC trustworthy? Invariant, and synchronized

A cycle counter is only a valid *clock* if two things hold, and danos checks both,
because they decide whether we read time with a cheap `rdtsc` or fall back to the HPET.

**Invariant.** An old TSC counted core clock cycles, so it sped up and slowed down with
frequency scaling — useless as wall time. Modern CPUs (all of danos's targets) provide an
**invariant TSC**: a constant rate across P/C-states that never stops. The guarantee is a
CPUID bit — leaf `0x80000007`, EDX bit 8 — on both Intel *and* AMD. danos reads it in
`calibrate`, and a TSC that doesn't advertise it is demoted to the HPET clocksource —
provided a usable HPET exists (64-bit; a 32-bit one wraps too fast to stay monotonic).
With no such fallback the TSC stays, there being nothing steadier to switch to. AMD is
why this matters in practice: it doesn't populate the Intel leaf `0x15` that enumerates
the TSC *frequency*, so danos already measures AMD's rate against the HPET — but a
measured frequency without the invariance guarantee is not enough.

**Synchronized.** Each core has its own TSC. Even invariant ones can start at different
values (a second socket, some firmware), so a thread migrating from a core reading
`1_000_000` to one reading `999_000` would see time jump *backward*. danos runs a **warp
check** as each application processor comes online (`checkWarpSource`, adapted from
Linux's): the waking core and the BSP hammer a shared "highest seen" TSC under a lock,
and if either ever reads below it, the cores' TSCs are skewed. It's pairwise because APs
come up one at a time ([smp.md](../os-development/smp.md)).

**The fallback.** When the TSC fails either test — non-invariant (a bare VM such as the
default qemu64), or warped between cores — danos moves the monotonic clock onto the
**HPET** main counter: one fixed-rate counter, so it can neither skew between cores nor
drift with frequency. It costs a memory-mapped read instead of a register read, but it
keeps time *accurate*, which is the whole point. The switch preserves the current value,
so the clock never jumps. The boot log names the outcome:

```
/system/kernel: clocksource tsc  (TSC invariant: yes, synchronized: yes)   # real Intel/AMD
/system/kernel: clocksource hpet (TSC invariant: no,  synchronized: yes)   # a bare VM (TCG)
```

## Two kinds of vector, one dispatch

The IDT now installs gates `0-47`: the 32 exceptions plus the device range. Every
gate still funnels through the same stub tail (`isr_common`), which calls one
dispatcher that branches on the vector (`interruptDispatch` in `idt.zig`):

```zig
if (state.vector < 32) {
    on_fault(state);              // exception: report and halt (never returns)
} else if (handlers[state.vector]) |handler| {
    handler();                    // device: run the registered handler
}
// else: spurious/unhandled — deliberately no EOI
```

(A third branch has since joined for user mode, elided here:
`state.vector == system_call_vector` (128) hands the trap frame to the ring-3
syscall handler.)

Two things make device interrupts *return* where exceptions don't:

1. **The handler returns.** The timer handler just bumps a tick counter. Control
   flows back to `isr_common`, which restores every register it saved and executes
   `iretq` — resuming the interrupted instruction exactly. (This is why the stub
   saves *all* the general registers.)
2. **End-of-interrupt.** Somewhere in there we write the LAPIC's EOI register. Miss
   this and the LAPIC thinks we're still busy and never delivers the next
   interrupt. It's the single most common "my timer fired once and stopped" bug.

**Each handler issues its own EOI**, rather than the dispatcher doing it around the
call. That looks like a needless devolution while the timer is the only device, and
`apic.timerTick` indeed does nothing but `eoi()` before bumping its counter (early,
because the tick hook is the scheduler, which may switch tasks and not return
promptly — the LAPIC mustn't wait on it).

It stops looking needless with the second device. A *routed* interrupt — one arriving
through the I/O APIC from a real device line — must be **masked before it is
acknowledged**, because a level-triggered line is still asserted at EOI time and would
redeliver instantly, forever. Only the handler knows which discipline its source
needs, so only the handler can sequence it. See [drivers.md](drivers.md), where the
device is quieted by a driver in ring 3, long after the ISR has returned.

A device handler is a plain `fn () void` — a timer or keyboard handler doesn't need
the interrupted registers. (The stubs originally didn't save the SSE/vector
registers, so a handler couldn't use them; `isr_common` now does an
`fxsave`/`fxrstor` of the full SSE/x87 state around dispatch — see
[interrupts.md](../os-development/interrupts.md).)

## Turning them on

Exceptions can't be masked, which is why they worked all along. Maskable device
interrupts don't fire until the CPU's interrupt flag is set — so the final step is
`sti` (`arch.enableInterrupts()`), after the APIC and timer are configured. From
that instant the kernel has a heartbeat, and its idle `hlt` loop
([halting.md](../os-development/halting.md)) wakes on every tick and dozes off again.

## Verifying it

The `timer` test (see [testing.md](../testing.md)) is the proof that an interrupt both
*fires* and *returns*: it records the tick count, busy-waits, and checks the count
advanced on its own.

```
$ python3 test/qemu_test.py timer
  timer        ... PASS  (matched 'DANOS-TEST-RESULT: PASS')
```

If the APIC weren't enabled, or `sti` were missing, or EOI were forgotten, the
count would stay put and the test would fail. That it advances — while the CPU was
spinning in unrelated code — is the whole mechanism working end to end.

## Since (done elsewhere)

- **Preemption**: the timer handler is where the scheduler decides to switch — the
  reason a *returning* interrupt matters. See [scheduling.md](../os-development/scheduling.md).
- **`sleep()` / timeouts** built on the calibrated clock.
- **The I/O APIC, routed**: external device lines now reach a vector, and the
  interrupt is delivered onward to a *user-space* driver as an IPC message. See
  [drivers.md](drivers.md).
- **Uncacheable MMIO**: device grants are mapped `PCD|PWT` (strong-uncacheable) for
  user drivers — see [paging.md](../os-development/paging.md).

## What's next (partly done since)

- **The keyboard** — done, exactly as sketched: the PS/2 bus driver
  (`system/drivers/ps2-bus/`) claims the port-mapped 8042 controller through the
  claim-gated `io_read`/`io_write` syscalls ([drivers.md](drivers.md)), binds
  IRQ 1 (and the aux mouse's IRQ 12), reads scancodes from `0x60`, and decodes
  them into HID events for the [input service](input.md).
- **MSI-X** — still open: `msi_bind` gives one per-device edge-triggered vector
  (M15); MSI-X's multi-vector table (many queues per device, e.g. NVMe) is the
  remaining extension.
- **The LAPIC's own page** — still mapped writeback-cacheable like the rest of
  the identity map. QEMU tolerates it; real hardware wants it uncacheable.