159 lines
9.0 KiB
Markdown
159 lines
9.0 KiB
Markdown
# Resilience: fault isolation and live restart
|
||
|
||
Steps 1–2 of the ordering below are **built**: user-mode isolation, and fault →
|
||
kill the process → keep the core (`onException` in `system/kernel/kernel.zig`; the
|
||
`fault-recovery` test proves a crashing ring-3 process dies alone while the system
|
||
keeps running). The supervisor notification and restart policy (steps 3+) are
|
||
still design. This is the property danos is really chasing:
|
||
**if a part of the OS breaks, isolate it, and re-initialise it — without rebooting.**
|
||
A crashed driver gets restarted; a wedged service gets killed and brought back. It's
|
||
the reason the [microkernel](vision.md) shape was chosen, and it's a *separate* goal
|
||
from [real-time](smp.md#does-the-right-choice-depend-on-real-time-vs-resilience) —
|
||
one that's less pervasive to build (see [vision.md](vision.md)).
|
||
|
||
## The idea: "let it crash" + supervision
|
||
|
||
The philosophy is older than microkernels and shows up across systems: don't try to
|
||
make every component perfect — make failures **contained and recoverable**. Isolate
|
||
each component, watch it, and when it dies, restart it from a known-good state. A
|
||
small trusted core supervises a fleet of restartable, untrusted parts.
|
||
|
||
Prior art worth studying (see Further reading): **MINIX 3's reincarnation server** (a
|
||
driver crashes, a supervisor restarts it live — the closest thing to your goal),
|
||
**QNX** (restartable drivers on a message-passing microkernel), **Erlang/OTP
|
||
supervision trees** ("let it crash", not a kernel but the canonical design), and the
|
||
historical **Tandem NonStop** (fault-tolerant by process pairs).
|
||
|
||
## Why a microkernel makes this possible
|
||
|
||
The blast radius of a fault is the address space it happens in. In a monolith, a
|
||
driver bug can corrupt anything — the kernel *is* the driver. In a microkernel,
|
||
drivers and services are **isolated user-space processes**, so a fault is trapped by
|
||
the kernel and confined to that one process. The kernel — the one thing that *can't*
|
||
be restarted, because it's the trusted base — stays tiny, which is precisely why a
|
||
small kernel is a *more recoverable* kernel: less code that can take the whole system
|
||
down. **Keeping the kernel minimal is a resilience strategy, not just an aesthetic.**
|
||
|
||
## The building blocks
|
||
|
||
1. **Address-space isolation.** A fault in one component can't corrupt another or the
|
||
kernel. This is the [user-mode milestone](vision.md) (ring 3, per-process page
|
||
tables) — the shared prerequisite for *any* of this, and it's needed regardless.
|
||
2. **Fault detection** — how the system notices a component is dead or sick:
|
||
- **Crash**: a CPU fault in a user process (page fault, illegal instruction) traps
|
||
to the kernel, which kills the process and notifies the supervisor. The clean,
|
||
easy case — and danos already reports CPU faults (see
|
||
[interrupts.md](interrupts.md)); user mode turns "halt on fault" into "kill the
|
||
process and tell the supervisor."
|
||
- **Hang**: a livelocked or infinite-looping component needs a **watchdog /
|
||
heartbeat** and the ability to **preempt and kill** it. The preemptive scheduler
|
||
already built ([scheduling.md](scheduling.md)) is what makes a runaway component
|
||
killable — a nice case of a scheduling mechanism serving resilience without any
|
||
real-time *guarantee*.
|
||
- **Misbehaviour**: IPC timeouts, failed health checks.
|
||
3. **A supervisor / reincarnation server.** A user-space server holding the *policy*:
|
||
what components exist, their dependencies, and each one's restart strategy. When a
|
||
component dies, it decides whether/how to restart it. (MINIX 3 calls this the
|
||
reincarnation server; Erlang calls it a supervisor.)
|
||
4. **A resource model that supports clean teardown.** When a component dies, its
|
||
resources — memory, MMIO grants, IPC channels, IRQ routes — must be **reclaimed**,
|
||
and a restarted replacement must be able to **re-acquire** them. This is where a
|
||
**capability** model shines (seL4's reference design): a component holds
|
||
capabilities to its resources; killing it **revokes** them, which frees everything
|
||
in one clean sweep, and the supervisor hands the replacement fresh caps. A simpler
|
||
grant/ownership table can work too — capabilities are the principled version.
|
||
5. **Re-initialisable drivers.** A driver must start from a known state and
|
||
re-establish its hardware. Some hardware is easy to reset; some holds state that's
|
||
hard to recover — a real limit on what "just restart it" can fix.
|
||
|
||
## The hard part: restarting *correctly*
|
||
|
||
Detecting and killing is the easy half. The genuinely tricky questions are about the
|
||
*rest of the system* when a component dies:
|
||
|
||
- **In-flight IPC**: messages sent to the dead component, or replies its clients are
|
||
blocked waiting for. The channel has to break cleanly and unblock the waiters with
|
||
an error rather than hang them forever (a design constraint that reaches back into
|
||
[ipc.md](ipc.md) — channels need a "peer died" outcome).
|
||
- **Clients**: how does a client discover the service it was talking to is gone and
|
||
has been replaced? Options: capability revocation makes stale handles fail; or a
|
||
**name server** re-binds clients to the new instance; or clients retry through a
|
||
stable endpoint.
|
||
- **State**: the cheapest model is **stateless restart** — the replacement starts
|
||
fresh and clients re-establish whatever they need. Richer options (checkpointed
|
||
state, state handed to a standby) are more work and more failure modes. Start
|
||
stateless.
|
||
|
||
These are the constraints most worth *bumping into and researching* — they're where
|
||
resilience gets genuinely interesting.
|
||
|
||
## Kernel mechanism vs user-space policy
|
||
|
||
The microkernel split applies to fault management itself:
|
||
|
||
- **Kernel (mechanism):** isolation, trapping faults, enforcing capabilities/grants,
|
||
IPC, creating/destroying address spaces, granting/revoking resources, preempting a
|
||
runaway task.
|
||
- **User space (policy):** the supervisor decides *what* to restart, *when*, and
|
||
*how* — dependency order, retry limits, escalation. None of that belongs in the
|
||
kernel.
|
||
|
||
So the kernel gains a few primitives (kill an address space, reclaim its resources,
|
||
deliver a "child died" notification); everything smart lives in a user-space server.
|
||
|
||
## What's *not* recoverable this way
|
||
|
||
Honest boundaries:
|
||
|
||
- **The kernel itself.** It's the trusted base; if it faults, this mechanism can't
|
||
save it. The mitigation is to keep it tiny — the microkernel bet.
|
||
- **Corrupted hardware state.** Isolation limits the blast radius to one process, but
|
||
if a driver wedged the device itself, a restart may not un-wedge it.
|
||
- **Shared-resource corruption** that happened *before* the fault was detected. Clean
|
||
capability revocation limits this, but it's why fault *detection latency* matters.
|
||
|
||
## Suggested ordering
|
||
|
||
1. **User mode + address-space isolation** — the shared prerequisite (also on the
|
||
path for everything else). **Done.**
|
||
2. **Kernel: fault → kill process → notify.** Turn today's "halt on fault" into
|
||
"confine to the process and report it." **Done** (the kill and reclaim; the
|
||
supervisor notification waits for step 3's supervisor). A killed server's
|
||
pending client is unblocked with `-EPEER` rather than hung.
|
||
3. **A minimal supervisor server** that can (re)start a process.
|
||
4. **Resource cleanup on death** — reclaim memory/MMIO/IPC/IRQ, via caps or a grant
|
||
table.
|
||
5. **First restartable driver** — the keyboard — as the end-to-end proof: crash it on
|
||
purpose, watch it come back.
|
||
|
||
## Relationship to real-time
|
||
|
||
Resilience needs **structural** features (isolation + supervision + a resource
|
||
model); real-time needs a **pervasive** timing invariant. They're separable, and
|
||
resilience is the lighter commitment (see [smp.md](smp.md) and [vision.md](vision.md)).
|
||
Note the overlap, though: **preemptive scheduling** and **priorities** — already
|
||
built — serve resilience too (you can preempt and kill a misbehaving component, and
|
||
run the supervisor at high priority). So danos keeps the useful *mechanisms* of the
|
||
real-time work without owing anyone a timing *guarantee*.
|
||
|
||
## Further reading
|
||
|
||
- Herder, Bos, Gras, Homburg, Tanenbaum — the **MINIX 3** papers, esp. *"Construction
|
||
of a Highly Dependable Operating System"* and *"Fault Isolation for Device
|
||
Drivers"* — the reincarnation server, the closest match to danos's goal.
|
||
- **QNX** architecture — a shipping microkernel with restartable drivers.
|
||
- **Erlang/OTP** supervision trees and the *"let it crash"* philosophy — the design
|
||
pattern, distilled.
|
||
- **seL4** capability model — the principled basis for clean resource teardown.
|
||
- **Tandem NonStop** (historical) — fault tolerance via process pairs.
|
||
|
||
## Related
|
||
|
||
- [vision.md](vision.md) — the goals this serves (learning by doing; resilience over
|
||
hard real-time).
|
||
- [scheduling.md](scheduling.md) — preemption, which makes runaway components killable.
|
||
- [ipc.md](ipc.md) — channels that need a "peer died" outcome for clean restart.
|
||
- [interrupts.md](interrupts.md) — fault reporting that user mode turns into "kill and
|
||
restart" instead of "halt".
|
||
- [smp.md](smp.md) — the real-time-vs-resilience fork, in the SMP context.
|