danos/docs/resilience.md

# Resilience: fault isolation and live restart

Steps 1–4 of the ordering below are **built** (M17–M18, 2026-07-13): user-mode
isolation; fault → kill the process → keep the core (`onException`; the
`fault-recovery` test); the supervisor notification **with exit reasons**
([process-lifecycle.md](process-lifecycle.md) — clean exit, fault class, or
killed, recorded before the notice posts); and the **restart policy itself**
([device-manager.md](device-manager.md)): the device manager supervises every
driver, restarts crashes with backoff, caps crash loops, and re-claims work
because the kernel releases a dead process's claims. The `driver-restart` and
`usb-report` scenarios prove kill → release → respawn → re-claim → re-report
end to end. What remains of this document's ladder is scope, not mechanism:
more of the system moved into restartable processes (the discovery migration,
[m19-m20-plan.md](m19-m20-plan.md), is the next rung). This is the property danos is really chasing:
**if a part of the OS breaks, isolate it, and re-initialise it — without rebooting.**
A crashed driver gets restarted; a wedged service gets killed and brought back. It's
the reason the [microkernel](vision.md) shape was chosen, and it's a *separate* goal
from [real-time](smp.md#does-the-right-choice-depend-on-real-time-vs-resilience) —
one that's less pervasive to build (see [vision.md](vision.md)).

## The idea: "let it crash" + supervision

The philosophy is older than microkernels and shows up across systems: don't try to
make every component perfect — make failures **contained and recoverable**. Isolate
each component, watch it, and when it dies, restart it from a known-good state. A
small trusted core supervises a fleet of restartable, untrusted parts.

Prior art worth studying (see Further reading): **MINIX 3's reincarnation server** (a
driver crashes, a supervisor restarts it live — the closest thing to your goal),
**QNX** (restartable drivers on a message-passing microkernel), **Erlang/OTP
supervision trees** ("let it crash", not a kernel but the canonical design), and the
historical **Tandem NonStop** (fault-tolerant by process pairs).

## Why a microkernel makes this possible

The blast radius of a fault is the address space it happens in. In a monolith, a
driver bug can corrupt anything — the kernel *is* the driver. In a microkernel,
drivers and services are **isolated user-space processes**, so a fault is trapped by
the kernel and confined to that one process. The kernel — the one thing that *can't*
be restarted, because it's the trusted base — stays tiny, which is precisely why a
small kernel is a *more recoverable* kernel: less code that can take the whole system
down. **Keeping the kernel minimal is a resilience strategy, not just an aesthetic.**

## The building blocks

1. **Address-space isolation.** A fault in one component can't corrupt another or the
   kernel. This is the [user-mode milestone](vision.md) (ring 3, per-process page
   tables) — the shared prerequisite for *any* of this, and it's needed regardless.
2. **Fault detection** — how the system notices a component is dead or sick:
   - **Crash**: a CPU fault in a user process (page fault, illegal instruction) traps
     to the kernel, which kills the process and notifies the supervisor. The clean,
     easy case — and danos already reports CPU faults (see
     [interrupts.md](interrupts.md)); user mode turns "halt on fault" into "kill the
     process and tell the supervisor."
   - **Hang**: a livelocked or infinite-looping component needs a **watchdog /
     heartbeat** and the ability to **preempt and kill** it. The preemptive scheduler
     already built ([scheduling.md](scheduling.md)) is what makes a runaway component
     killable — a nice case of a scheduling mechanism serving resilience without any
     real-time *guarantee*.
   - **Misbehaviour**: IPC timeouts, failed health checks.
3. **A supervisor / reincarnation server.** A user-space server holding the *policy*:
   what components exist, their dependencies, and each one's restart strategy. When a
   component dies, it decides whether/how to restart it. (MINIX 3 calls this the
   reincarnation server; Erlang calls it a supervisor.)
4. **A resource model that supports clean teardown.** When a component dies, its
   resources — memory, MMIO grants, IPC channels, IRQ routes — must be **reclaimed**,
   and a restarted replacement must be able to **re-acquire** them. This is where a
   **capability** model shines (seL4's reference design): a component holds
   capabilities to its resources; killing it **revokes** them, which frees everything
   in one clean sweep, and the supervisor hands the replacement fresh caps. A simpler
   grant/ownership table can work too — capabilities are the principled version.
5. **Re-initialisable drivers.** A driver must start from a known state and
   re-establish its hardware. Some hardware is easy to reset; some holds state that's
   hard to recover — a real limit on what "just restart it" can fix.

## The hard part: restarting *correctly*

Detecting and killing is the easy half. The genuinely tricky questions are about the
*rest of the system* when a component dies:

- **In-flight IPC**: messages sent to the dead component, or replies its clients are
  blocked waiting for. The channel has to break cleanly and unblock the waiters with
  an error rather than hang them forever (a design constraint that reaches back into
  [ipc.md](ipc.md) — channels need a "peer died" outcome).
- **Clients**: how does a client discover the service it was talking to is gone and
  has been replaced? Options: capability revocation makes stale handles fail; or a
  **name server** re-binds clients to the new instance; or clients retry through a
  stable endpoint.
- **State**: the cheapest model is **stateless restart** — the replacement starts
  fresh and clients re-establish whatever they need. Richer options (checkpointed
  state, state handed to a standby) are more work and more failure modes. Start
  stateless.

These are the constraints most worth *bumping into and researching* — they're where
resilience gets genuinely interesting.

## Kernel mechanism vs user-space policy

The microkernel split applies to fault management itself:

- **Kernel (mechanism):** isolation, trapping faults, enforcing capabilities/grants,
  IPC, creating/destroying address spaces, granting/revoking resources, preempting a
  runaway task.
- **User space (policy):** the supervisor decides *what* to restart, *when*, and
  *how* — dependency order, retry limits, escalation. None of that belongs in the
  kernel.

So the kernel gains a few primitives (kill an address space, reclaim its resources,
deliver a "child died" notification); everything smart lives in a user-space server.

## What's *not* recoverable this way

Honest boundaries:

- **The kernel itself.** It's the trusted base; if it faults, this mechanism can't
  save it. The mitigation is to keep it tiny — the microkernel bet.
- **Corrupted hardware state.** Isolation limits the blast radius to one process, but
  if a driver wedged the device itself, a restart may not un-wedge it.
- **Shared-resource corruption** that happened *before* the fault was detected. Clean
  capability revocation limits this, but it's why fault *detection latency* matters.

## Suggested ordering

1. **User mode + address-space isolation** — the shared prerequisite (also on the
   path for everything else). **Done.**
2. **Kernel: fault → kill process → notify.** Turn today's "halt on fault" into
   "confine to the process and report it." **Done** (the kill and reclaim; the
   supervisor notification waits for step 3's supervisor). A killed server's
   pending client is unblocked with `-EPEER` rather than hung.
3. **A minimal supervisor server** that can (re)start a process.
4. **Resource cleanup on death** — reclaim memory/MMIO/IPC/IRQ, via caps or a grant
   table.
5. **First restartable driver** — the keyboard — as the end-to-end proof: crash it on
   purpose, watch it come back.

## Relationship to real-time

Resilience needs **structural** features (isolation + supervision + a resource
model); real-time needs a **pervasive** timing invariant. They're separable, and
resilience is the lighter commitment (see [smp.md](smp.md) and [vision.md](vision.md)).
Note the overlap, though: **preemptive scheduling** and **priorities** — already
built — serve resilience too (you can preempt and kill a misbehaving component, and
run the supervisor at high priority). So danos keeps the useful *mechanisms* of the
real-time work without owing anyone a timing *guarantee*.

## Further reading

- Herder, Bos, Gras, Homburg, Tanenbaum — the **MINIX 3** papers, esp. *"Construction
  of a Highly Dependable Operating System"* and *"Fault Isolation for Device
  Drivers"* — the reincarnation server, the closest match to danos's goal.
- **QNX** architecture — a shipping microkernel with restartable drivers.
- **Erlang/OTP** supervision trees and the *"let it crash"* philosophy — the design
  pattern, distilled.
- **seL4** capability model — the principled basis for clean resource teardown.
- **Tandem NonStop** (historical) — fault tolerance via process pairs.

## Related

- [vision.md](vision.md) — the goals this serves (learning by doing; resilience over
  hard real-time).
- [scheduling.md](scheduling.md) — preemption, which makes runaway components killable.
- [ipc.md](ipc.md) — channels that need a "peer died" outcome for clean restart.
- [interrupts.md](interrupts.md) — fault reporting that user mode turns into "kill and
  restart" instead of "halt".
- [smp.md](smp.md) — the real-time-vs-resilience fork, in the SMP context.