Daniel Samson
452080e997
Time is a kernel concern in danos: the kernel owns the scheduling timer and already exposes monotonic time via the clock/sleep/timer_bind syscalls, so a userspace time service would be a redundant, slower path. This adds the generic runtime.time module over those syscalls, retires the two demonstration drivers, reorganizes the milestone docs, and makes the monotonic clock correct on Intel, AMD, and inside any VM. runtime.time (library/runtime/time.zig) - Instant/Duration interface: now, sleep, spin, after, monotonicNanos, available - a thin layer over system.clock/sleep/timerOnce; unit-tested arithmetic Remove the demo drivers hpet and bus (a teaching example belongs in the docs, not shipped in the tree) - system/drivers/ now holds only real drivers: pci-bus, ps2-bus, usb-xhci-bus - device-manager end-to-end test repointed to pci-bus (asserts on kernel state: the process table and the device tree, not a racy serial marker) - device_register containment moved to a new in-kernel `containment` test - the driver-model worked example moved inline into docs/drivers.md Reorganize milestone docs into topic docs - m17-m18 / m19-m20 / m21 plans dissolved into process-lifecycle, device-manager, discovery, and acpi docs; new docs/power.md and docs/timers.md; ~20 citations repointed; plan docs deleted TSC reliability (apic.zig, smp.zig, cpu.zig, kernel.zig) - check the invariant-TSC bit (CPUID 0x80000007 EDX[8]) on Intel and AMD - cross-core "warp" check at SMP bring-up, pairwise BSP<->AP as each core comes up - fall back to the HPET clocksource when the TSC is not invariant (a bare VM) or not synchronized (a warp), switched continuously so time never jumps - boot log reports the outcome; new tsc-sync test exercises the TSC + warp path Verified: zig build; zig build test; 60/60 QEMU cases (incl. new containment and tsc-sync).
166 lines
9.6 KiB
Markdown
166 lines
9.6 KiB
Markdown
# Resilience: fault isolation and live restart
|
||
|
||
Steps 1–4 of the ordering below are **built** (M17–M18, 2026-07-13): user-mode
|
||
isolation; fault → kill the process → keep the core (`onException`; the
|
||
`fault-recovery` test); the supervisor notification **with exit reasons**
|
||
([process-lifecycle.md](process-lifecycle.md) — clean exit, fault class, or
|
||
killed, recorded before the notice posts); and the **restart policy itself**
|
||
([device-manager.md](device-manager.md)): the device manager supervises every
|
||
driver, restarts crashes with backoff, caps crash loops, and re-claims work
|
||
because the kernel releases a dead process's claims. The `driver-restart` and
|
||
`usb-report` scenarios prove kill → release → respawn → re-claim → re-report
|
||
end to end. What remains of this document's ladder is scope, not mechanism:
|
||
more of the system moved into restartable processes (the discovery migration,
|
||
[discovery.md](discovery.md), is the next rung). This is the property danos is really chasing:
|
||
**if a part of the OS breaks, isolate it, and re-initialise it — without rebooting.**
|
||
A crashed driver gets restarted; a wedged service gets killed and brought back. It's
|
||
the reason the [microkernel](vision.md) shape was chosen, and it's a *separate* goal
|
||
from [real-time](smp.md#does-the-right-choice-depend-on-real-time-vs-resilience) —
|
||
one that's less pervasive to build (see [vision.md](vision.md)).
|
||
|
||
## The idea: "let it crash" + supervision
|
||
|
||
The philosophy is older than microkernels and shows up across systems: don't try to
|
||
make every component perfect — make failures **contained and recoverable**. Isolate
|
||
each component, watch it, and when it dies, restart it from a known-good state. A
|
||
small trusted core supervises a fleet of restartable, untrusted parts.
|
||
|
||
Prior art worth studying (see Further reading): **MINIX 3's reincarnation server** (a
|
||
driver crashes, a supervisor restarts it live — the closest thing to your goal),
|
||
**QNX** (restartable drivers on a message-passing microkernel), **Erlang/OTP
|
||
supervision trees** ("let it crash", not a kernel but the canonical design), and the
|
||
historical **Tandem NonStop** (fault-tolerant by process pairs).
|
||
|
||
## Why a microkernel makes this possible
|
||
|
||
The blast radius of a fault is the address space it happens in. In a monolith, a
|
||
driver bug can corrupt anything — the kernel *is* the driver. In a microkernel,
|
||
drivers and services are **isolated user-space processes**, so a fault is trapped by
|
||
the kernel and confined to that one process. The kernel — the one thing that *can't*
|
||
be restarted, because it's the trusted base — stays tiny, which is precisely why a
|
||
small kernel is a *more recoverable* kernel: less code that can take the whole system
|
||
down. **Keeping the kernel minimal is a resilience strategy, not just an aesthetic.**
|
||
|
||
## The building blocks
|
||
|
||
1. **Address-space isolation.** A fault in one component can't corrupt another or the
|
||
kernel. This is the [user-mode milestone](vision.md) (ring 3, per-process page
|
||
tables) — the shared prerequisite for *any* of this, and it's needed regardless.
|
||
2. **Fault detection** — how the system notices a component is dead or sick:
|
||
- **Crash**: a CPU fault in a user process (page fault, illegal instruction) traps
|
||
to the kernel, which kills the process and notifies the supervisor. The clean,
|
||
easy case — and danos already reports CPU faults (see
|
||
[interrupts.md](interrupts.md)); user mode turns "halt on fault" into "kill the
|
||
process and tell the supervisor."
|
||
- **Hang**: a livelocked or infinite-looping component needs a **watchdog /
|
||
heartbeat** and the ability to **preempt and kill** it. The preemptive scheduler
|
||
already built ([scheduling.md](scheduling.md)) is what makes a runaway component
|
||
killable — a nice case of a scheduling mechanism serving resilience without any
|
||
real-time *guarantee*.
|
||
- **Misbehaviour**: IPC timeouts, failed health checks.
|
||
3. **A supervisor / reincarnation server.** A user-space server holding the *policy*:
|
||
what components exist, their dependencies, and each one's restart strategy. When a
|
||
component dies, it decides whether/how to restart it. (MINIX 3 calls this the
|
||
reincarnation server; Erlang calls it a supervisor.)
|
||
4. **A resource model that supports clean teardown.** When a component dies, its
|
||
resources — memory, MMIO grants, IPC channels, IRQ routes — must be **reclaimed**,
|
||
and a restarted replacement must be able to **re-acquire** them. This is where a
|
||
**capability** model shines (seL4's reference design): a component holds
|
||
capabilities to its resources; killing it **revokes** them, which frees everything
|
||
in one clean sweep, and the supervisor hands the replacement fresh caps. A simpler
|
||
grant/ownership table can work too — capabilities are the principled version.
|
||
5. **Re-initialisable drivers.** A driver must start from a known state and
|
||
re-establish its hardware. Some hardware is easy to reset; some holds state that's
|
||
hard to recover — a real limit on what "just restart it" can fix.
|
||
|
||
## The hard part: restarting *correctly*
|
||
|
||
Detecting and killing is the easy half. The genuinely tricky questions are about the
|
||
*rest of the system* when a component dies:
|
||
|
||
- **In-flight IPC**: messages sent to the dead component, or replies its clients are
|
||
blocked waiting for. The channel has to break cleanly and unblock the waiters with
|
||
an error rather than hang them forever (a design constraint that reaches back into
|
||
[ipc.md](ipc.md) — channels need a "peer died" outcome).
|
||
- **Clients**: how does a client discover the service it was talking to is gone and
|
||
has been replaced? Options: capability revocation makes stale handles fail; or a
|
||
**name server** re-binds clients to the new instance; or clients retry through a
|
||
stable endpoint.
|
||
- **State**: the cheapest model is **stateless restart** — the replacement starts
|
||
fresh and clients re-establish whatever they need. Richer options (checkpointed
|
||
state, state handed to a standby) are more work and more failure modes. Start
|
||
stateless.
|
||
|
||
These are the constraints most worth *bumping into and researching* — they're where
|
||
resilience gets genuinely interesting.
|
||
|
||
## Kernel mechanism vs user-space policy
|
||
|
||
The microkernel split applies to fault management itself:
|
||
|
||
- **Kernel (mechanism):** isolation, trapping faults, enforcing capabilities/grants,
|
||
IPC, creating/destroying address spaces, granting/revoking resources, preempting a
|
||
runaway task.
|
||
- **User space (policy):** the supervisor decides *what* to restart, *when*, and
|
||
*how* — dependency order, retry limits, escalation. None of that belongs in the
|
||
kernel.
|
||
|
||
So the kernel gains a few primitives (kill an address space, reclaim its resources,
|
||
deliver a "child died" notification); everything smart lives in a user-space server.
|
||
|
||
## What's *not* recoverable this way
|
||
|
||
Honest boundaries:
|
||
|
||
- **The kernel itself.** It's the trusted base; if it faults, this mechanism can't
|
||
save it. The mitigation is to keep it tiny — the microkernel bet.
|
||
- **Corrupted hardware state.** Isolation limits the blast radius to one process, but
|
||
if a driver wedged the device itself, a restart may not un-wedge it.
|
||
- **Shared-resource corruption** that happened *before* the fault was detected. Clean
|
||
capability revocation limits this, but it's why fault *detection latency* matters.
|
||
|
||
## Suggested ordering
|
||
|
||
1. **User mode + address-space isolation** — the shared prerequisite (also on the
|
||
path for everything else). **Done.**
|
||
2. **Kernel: fault → kill process → notify.** Turn today's "halt on fault" into
|
||
"confine to the process and report it." **Done** (the kill and reclaim; the
|
||
supervisor notification waits for step 3's supervisor). A killed server's
|
||
pending client is unblocked with `-EPEER` rather than hung.
|
||
3. **A minimal supervisor server** that can (re)start a process.
|
||
4. **Resource cleanup on death** — reclaim memory/MMIO/IPC/IRQ, via caps or a grant
|
||
table.
|
||
5. **First restartable driver** — the keyboard — as the end-to-end proof: crash it on
|
||
purpose, watch it come back.
|
||
|
||
## Relationship to real-time
|
||
|
||
Resilience needs **structural** features (isolation + supervision + a resource
|
||
model); real-time needs a **pervasive** timing invariant. They're separable, and
|
||
resilience is the lighter commitment (see [smp.md](smp.md) and [vision.md](vision.md)).
|
||
Note the overlap, though: **preemptive scheduling** and **priorities** — already
|
||
built — serve resilience too (you can preempt and kill a misbehaving component, and
|
||
run the supervisor at high priority). So danos keeps the useful *mechanisms* of the
|
||
real-time work without owing anyone a timing *guarantee*.
|
||
|
||
## Further reading
|
||
|
||
- Herder, Bos, Gras, Homburg, Tanenbaum — the **MINIX 3** papers, esp. *"Construction
|
||
of a Highly Dependable Operating System"* and *"Fault Isolation for Device
|
||
Drivers"* — the reincarnation server, the closest match to danos's goal.
|
||
- **QNX** architecture — a shipping microkernel with restartable drivers.
|
||
- **Erlang/OTP** supervision trees and the *"let it crash"* philosophy — the design
|
||
pattern, distilled.
|
||
- **seL4** capability model — the principled basis for clean resource teardown.
|
||
- **Tandem NonStop** (historical) — fault tolerance via process pairs.
|
||
|
||
## Related
|
||
|
||
- [vision.md](vision.md) — the goals this serves (learning by doing; resilience over
|
||
hard real-time).
|
||
- [scheduling.md](scheduling.md) — preemption, which makes runaway components killable.
|
||
- [ipc.md](ipc.md) — channels that need a "peer died" outcome for clean restart.
|
||
- [interrupts.md](interrupts.md) — fault reporting that user mode turns into "kill and
|
||
restart" instead of "halt".
|
||
- [smp.md](smp.md) — the real-time-vs-resilience fork, in the SMP context.
|