153 lines
8.5 KiB
Markdown
153 lines
8.5 KiB
Markdown
# Resilience: fault isolation and live restart
|
|
|
|
A design/research note, not built yet. This is the property danos is really chasing:
|
|
**if a part of the OS breaks, isolate it, and re-initialise it — without rebooting.**
|
|
A crashed driver gets restarted; a wedged service gets killed and brought back. It's
|
|
the reason the [microkernel](vision.md) shape was chosen, and it's a *separate* goal
|
|
from [real-time](smp.md#does-the-right-choice-depend-on-real-time-vs-resilience) —
|
|
one that's less pervasive to build (see [vision.md](vision.md)).
|
|
|
|
## The idea: "let it crash" + supervision
|
|
|
|
The philosophy is older than microkernels and shows up across systems: don't try to
|
|
make every component perfect — make failures **contained and recoverable**. Isolate
|
|
each component, watch it, and when it dies, restart it from a known-good state. A
|
|
small trusted core supervises a fleet of restartable, untrusted parts.
|
|
|
|
Prior art worth studying (see Further reading): **MINIX 3's reincarnation server** (a
|
|
driver crashes, a supervisor restarts it live — the closest thing to your goal),
|
|
**QNX** (restartable drivers on a message-passing microkernel), **Erlang/OTP
|
|
supervision trees** ("let it crash", not a kernel but the canonical design), and the
|
|
historical **Tandem NonStop** (fault-tolerant by process pairs).
|
|
|
|
## Why a microkernel makes this possible
|
|
|
|
The blast radius of a fault is the address space it happens in. In a monolith, a
|
|
driver bug can corrupt anything — the kernel *is* the driver. In a microkernel,
|
|
drivers and services are **isolated user-space processes**, so a fault is trapped by
|
|
the kernel and confined to that one process. The kernel — the one thing that *can't*
|
|
be restarted, because it's the trusted base — stays tiny, which is precisely why a
|
|
small kernel is a *more recoverable* kernel: less code that can take the whole system
|
|
down. **Keeping the kernel minimal is a resilience strategy, not just an aesthetic.**
|
|
|
|
## The building blocks
|
|
|
|
1. **Address-space isolation.** A fault in one component can't corrupt another or the
|
|
kernel. This is the [user-mode milestone](vision.md) (ring 3, per-process page
|
|
tables) — the shared prerequisite for *any* of this, and it's needed regardless.
|
|
2. **Fault detection** — how the system notices a component is dead or sick:
|
|
- **Crash**: a CPU fault in a user process (page fault, illegal instruction) traps
|
|
to the kernel, which kills the process and notifies the supervisor. The clean,
|
|
easy case — and danos already reports CPU faults (see
|
|
[interrupts.md](interrupts.md)); user mode turns "halt on fault" into "kill the
|
|
process and tell the supervisor."
|
|
- **Hang**: a livelocked or infinite-looping component needs a **watchdog /
|
|
heartbeat** and the ability to **preempt and kill** it. The preemptive scheduler
|
|
already built ([scheduling.md](scheduling.md)) is what makes a runaway component
|
|
killable — a nice case of a scheduling mechanism serving resilience without any
|
|
real-time *guarantee*.
|
|
- **Misbehaviour**: IPC timeouts, failed health checks.
|
|
3. **A supervisor / reincarnation server.** A user-space server holding the *policy*:
|
|
what components exist, their dependencies, and each one's restart strategy. When a
|
|
component dies, it decides whether/how to restart it. (MINIX 3 calls this the
|
|
reincarnation server; Erlang calls it a supervisor.)
|
|
4. **A resource model that supports clean teardown.** When a component dies, its
|
|
resources — memory, MMIO grants, IPC channels, IRQ routes — must be **reclaimed**,
|
|
and a restarted replacement must be able to **re-acquire** them. This is where a
|
|
**capability** model shines (seL4's reference design): a component holds
|
|
capabilities to its resources; killing it **revokes** them, which frees everything
|
|
in one clean sweep, and the supervisor hands the replacement fresh caps. A simpler
|
|
grant/ownership table can work too — capabilities are the principled version.
|
|
5. **Re-initialisable drivers.** A driver must start from a known state and
|
|
re-establish its hardware. Some hardware is easy to reset; some holds state that's
|
|
hard to recover — a real limit on what "just restart it" can fix.
|
|
|
|
## The hard part: restarting *correctly*
|
|
|
|
Detecting and killing is the easy half. The genuinely tricky questions are about the
|
|
*rest of the system* when a component dies:
|
|
|
|
- **In-flight IPC**: messages sent to the dead component, or replies its clients are
|
|
blocked waiting for. The channel has to break cleanly and unblock the waiters with
|
|
an error rather than hang them forever (a design constraint that reaches back into
|
|
[ipc.md](ipc.md) — channels need a "peer died" outcome).
|
|
- **Clients**: how does a client discover the service it was talking to is gone and
|
|
has been replaced? Options: capability revocation makes stale handles fail; or a
|
|
**name server** re-binds clients to the new instance; or clients retry through a
|
|
stable endpoint.
|
|
- **State**: the cheapest model is **stateless restart** — the replacement starts
|
|
fresh and clients re-establish whatever they need. Richer options (checkpointed
|
|
state, state handed to a standby) are more work and more failure modes. Start
|
|
stateless.
|
|
|
|
These are the constraints most worth *bumping into and researching* — they're where
|
|
resilience gets genuinely interesting.
|
|
|
|
## Kernel mechanism vs user-space policy
|
|
|
|
The microkernel split applies to fault management itself:
|
|
|
|
- **Kernel (mechanism):** isolation, trapping faults, enforcing capabilities/grants,
|
|
IPC, creating/destroying address spaces, granting/revoking resources, preempting a
|
|
runaway task.
|
|
- **User space (policy):** the supervisor decides *what* to restart, *when*, and
|
|
*how* — dependency order, retry limits, escalation. None of that belongs in the
|
|
kernel.
|
|
|
|
So the kernel gains a few primitives (kill an address space, reclaim its resources,
|
|
deliver a "child died" notification); everything smart lives in a user-space server.
|
|
|
|
## What's *not* recoverable this way
|
|
|
|
Honest boundaries:
|
|
|
|
- **The kernel itself.** It's the trusted base; if it faults, this mechanism can't
|
|
save it. The mitigation is to keep it tiny — the microkernel bet.
|
|
- **Corrupted hardware state.** Isolation limits the blast radius to one process, but
|
|
if a driver wedged the device itself, a restart may not un-wedge it.
|
|
- **Shared-resource corruption** that happened *before* the fault was detected. Clean
|
|
capability revocation limits this, but it's why fault *detection latency* matters.
|
|
|
|
## Suggested ordering
|
|
|
|
1. **User mode + address-space isolation** — the shared prerequisite (also on the
|
|
path for everything else).
|
|
2. **Kernel: fault → kill process → notify.** Turn today's "halt on fault" into
|
|
"confine to the process and report it."
|
|
3. **A minimal supervisor server** that can (re)start a process.
|
|
4. **Resource cleanup on death** — reclaim memory/MMIO/IPC/IRQ, via caps or a grant
|
|
table.
|
|
5. **First restartable driver** — the keyboard — as the end-to-end proof: crash it on
|
|
purpose, watch it come back.
|
|
|
|
## Relationship to real-time
|
|
|
|
Resilience needs **structural** features (isolation + supervision + a resource
|
|
model); real-time needs a **pervasive** timing invariant. They're separable, and
|
|
resilience is the lighter commitment (see [smp.md](smp.md) and [vision.md](vision.md)).
|
|
Note the overlap, though: **preemptive scheduling** and **priorities** — already
|
|
built — serve resilience too (you can preempt and kill a misbehaving component, and
|
|
run the supervisor at high priority). So danos keeps the useful *mechanisms* of the
|
|
real-time work without owing anyone a timing *guarantee*.
|
|
|
|
## Further reading
|
|
|
|
- Herder, Bos, Gras, Homburg, Tanenbaum — the **MINIX 3** papers, esp. *"Construction
|
|
of a Highly Dependable Operating System"* and *"Fault Isolation for Device
|
|
Drivers"* — the reincarnation server, the closest match to danos's goal.
|
|
- **QNX** architecture — a shipping microkernel with restartable drivers.
|
|
- **Erlang/OTP** supervision trees and the *"let it crash"* philosophy — the design
|
|
pattern, distilled.
|
|
- **seL4** capability model — the principled basis for clean resource teardown.
|
|
- **Tandem NonStop** (historical) — fault tolerance via process pairs.
|
|
|
|
## Related
|
|
|
|
- [vision.md](vision.md) — the goals this serves (learning by doing; resilience over
|
|
hard real-time).
|
|
- [scheduling.md](scheduling.md) — preemption, which makes runaway components killable.
|
|
- [ipc.md](ipc.md) — channels that need a "peer died" outcome for clean restart.
|
|
- [interrupts.md](interrupts.md) — fault reporting that user mode turns into "kill and
|
|
restart" instead of "halt".
|
|
- [smp.md](smp.md) — the real-time-vs-resilience fork, in the SMP context.
|