8.5 KiB

Raw Blame History

Resilience: fault isolation and live restart

A design/research note, not built yet. This is the property danos is really chasing: if a part of the OS breaks, isolate it, and re-initialise it — without rebooting. A crashed driver gets restarted; a wedged service gets killed and brought back. It's the reason the microkernel shape was chosen, and it's a separate goal from real-time — one that's less pervasive to build (see vision.md).

The idea: "let it crash" + supervision

The philosophy is older than microkernels and shows up across systems: don't try to make every component perfect — make failures contained and recoverable. Isolate each component, watch it, and when it dies, restart it from a known-good state. A small trusted core supervises a fleet of restartable, untrusted parts.

Prior art worth studying (see Further reading): MINIX 3's reincarnation server (a driver crashes, a supervisor restarts it live — the closest thing to your goal), QNX (restartable drivers on a message-passing microkernel), Erlang/OTP supervision trees ("let it crash", not a kernel but the canonical design), and the historical Tandem NonStop (fault-tolerant by process pairs).

Why a microkernel makes this possible

The blast radius of a fault is the address space it happens in. In a monolith, a driver bug can corrupt anything — the kernel is the driver. In a microkernel, drivers and services are isolated user-space processes, so a fault is trapped by the kernel and confined to that one process. The kernel — the one thing that can't be restarted, because it's the trusted base — stays tiny, which is precisely why a small kernel is a more recoverable kernel: less code that can take the whole system down. Keeping the kernel minimal is a resilience strategy, not just an aesthetic.

The building blocks

Address-space isolation. A fault in one component can't corrupt another or the kernel. This is the user-mode milestone (ring 3, per-process page tables) — the shared prerequisite for any of this, and it's needed regardless.
Fault detection — how the system notices a component is dead or sick:
- Crash: a CPU fault in a user process (page fault, illegal instruction) traps to the kernel, which kills the process and notifies the supervisor. The clean, easy case — and danos already reports CPU faults (see interrupts.md); user mode turns "halt on fault" into "kill the process and tell the supervisor."
- Hang: a livelocked or infinite-looping component needs a watchdog / heartbeat and the ability to preempt and kill it. The preemptive scheduler already built (scheduling.md) is what makes a runaway component killable — a nice case of a scheduling mechanism serving resilience without any real-time guarantee.
- Misbehaviour: IPC timeouts, failed health checks.
A supervisor / reincarnation server. A user-space server holding the policy: what components exist, their dependencies, and each one's restart strategy. When a component dies, it decides whether/how to restart it. (MINIX 3 calls this the reincarnation server; Erlang calls it a supervisor.)
A resource model that supports clean teardown. When a component dies, its resources — memory, MMIO grants, IPC channels, IRQ routes — must be reclaimed, and a restarted replacement must be able to re-acquire them. This is where a capability model shines (seL4's reference design): a component holds capabilities to its resources; killing it revokes them, which frees everything in one clean sweep, and the supervisor hands the replacement fresh caps. A simpler grant/ownership table can work too — capabilities are the principled version.
Re-initialisable drivers. A driver must start from a known state and re-establish its hardware. Some hardware is easy to reset; some holds state that's hard to recover — a real limit on what "just restart it" can fix.

The hard part: restarting correctly

Detecting and killing is the easy half. The genuinely tricky questions are about the rest of the system when a component dies:

In-flight IPC: messages sent to the dead component, or replies its clients are blocked waiting for. The channel has to break cleanly and unblock the waiters with an error rather than hang them forever (a design constraint that reaches back into ipc.md — channels need a "peer died" outcome).
Clients: how does a client discover the service it was talking to is gone and has been replaced? Options: capability revocation makes stale handles fail; or a name server re-binds clients to the new instance; or clients retry through a stable endpoint.
State: the cheapest model is stateless restart — the replacement starts fresh and clients re-establish whatever they need. Richer options (checkpointed state, state handed to a standby) are more work and more failure modes. Start stateless.

These are the constraints most worth bumping into and researching — they're where resilience gets genuinely interesting.

Kernel mechanism vs user-space policy

The microkernel split applies to fault management itself:

Kernel (mechanism): isolation, trapping faults, enforcing capabilities/grants, IPC, creating/destroying address spaces, granting/revoking resources, preempting a runaway task.
User space (policy): the supervisor decides what to restart, when, and how — dependency order, retry limits, escalation. None of that belongs in the kernel.

So the kernel gains a few primitives (kill an address space, reclaim its resources, deliver a "child died" notification); everything smart lives in a user-space server.

What's not recoverable this way

Honest boundaries:

The kernel itself. It's the trusted base; if it faults, this mechanism can't save it. The mitigation is to keep it tiny — the microkernel bet.
Corrupted hardware state. Isolation limits the blast radius to one process, but if a driver wedged the device itself, a restart may not un-wedge it.
Shared-resource corruption that happened before the fault was detected. Clean capability revocation limits this, but it's why fault detection latency matters.

Suggested ordering

User mode + address-space isolation — the shared prerequisite (also on the path for everything else).
Kernel: fault → kill process → notify. Turn today's "halt on fault" into "confine to the process and report it."
A minimal supervisor server that can (re)start a process.
Resource cleanup on death — reclaim memory/MMIO/IPC/IRQ, via caps or a grant table.
First restartable driver — the keyboard — as the end-to-end proof: crash it on purpose, watch it come back.

Relationship to real-time

Resilience needs structural features (isolation + supervision + a resource model); real-time needs a pervasive timing invariant. They're separable, and resilience is the lighter commitment (see smp.md and vision.md). Note the overlap, though: preemptive scheduling and priorities — already built — serve resilience too (you can preempt and kill a misbehaving component, and run the supervisor at high priority). So danos keeps the useful mechanisms of the real-time work without owing anyone a timing guarantee.

8.5 KiB Raw Blame History