Kill a faulting user process instead of halting the machine

A CPU exception raised in ring 3 by a scheduled process now kills that
process - IRQ bindings, IPC handles, and address space reclaimed, a
client it owed a reply to failed with the new -EPEER instead of hung -
and the core reschedules (docs/resilience.md step 2). Kernel-mode
faults, NMI, double fault, and machine check stay terminal, as does the
borrowed-thread isolation probe. Proven by the new fault-recovery QEMU
test: init keeps heartbeating after a process page-faults to death.

docs/interrupts.md

@@ -80,11 +80,22 @@ inline). `build.zig` adds `isr.s` to the arch module.
 ## Reporting a fault
 
 `isr_common` calls `exceptionHandler`, which forwards to a swappable `on_fault`
-hook. The generic kernel installs a reporter (`onException` in `main.zig`) that
-prints, in red, the exception name and vector, the error code, the faulting RIP
+hook. The generic kernel installs a reporter (`onException` in `kernel.zig`) that
+prints the exception name and vector, the error code, the faulting RIP
 and RSP, and — for a page fault (#PF, vector 14) — the faulting address from
-**CR2**. Then it halts. There's no fault *recovery* yet, so every exception is
-terminal; the point is that it's now **visible** instead of a silent reset.
+**CR2**. What happens next depends on where the fault came from:
+
+- **User mode (CPL 3): kill the process, keep the machine.** The kernel is intact
+  (the CPU trapped onto the task's kernel stack), so the faulting process is
+  killed — address space, IRQ bindings, and IPC handles reclaimed; a client it
+  owed a reply to is failed with `-EPEER` — and the core reschedules. A crashing
+  driver takes itself down, never the OS. This is fault recovery step 2 of
+  [resilience.md](resilience.md). NMI, double fault, and machine check are
+  excluded: they report machine trouble regardless of what was running.
+- **Kernel mode: halt this core.** The trusted base itself is broken, so there is
+  nothing safe to kill; the fault is still *contained* to the core (an
+  application-processor fault leaves the rest of the system running), and the
+  report makes it **visible** instead of a silent reset.
 
 The hook is set before `arch.init()` in `kmain`, so a fault during setup is still
 caught.

docs/resilience.md

@@ -1,6 +1,10 @@
 # Resilience: fault isolation and live restart
 
-A design/research note, not built yet. This is the property danos is really chasing:
+Steps 1–2 of the ordering below are **built**: user-mode isolation, and fault →
+kill the process → keep the core (`onException` in `system/kernel/kernel.zig`; the
+`fault-recovery` test proves a crashing ring-3 process dies alone while the system
+keeps running). The supervisor notification and restart policy (steps 3+) are
+still design. This is the property danos is really chasing:
 **if a part of the OS breaks, isolate it, and re-initialise it — without rebooting.**
 A crashed driver gets restarted; a wedged service gets killed and brought back. It's
 the reason the [microkernel](vision.md) shape was chosen, and it's a *separate* goal
@@ -111,9 +115,11 @@ Honest boundaries:
 ## Suggested ordering
 
 1. **User mode + address-space isolation** — the shared prerequisite (also on the
-   path for everything else).
+   path for everything else). **Done.**
 2. **Kernel: fault → kill process → notify.** Turn today's "halt on fault" into
-   "confine to the process and report it."
+   "confine to the process and report it." **Done** (the kill and reclaim; the
+   supervisor notification waits for step 3's supervisor). A killed server's
+   pending client is unblocked with `-EPEER` rather than hung.
 3. **A minimal supervisor server** that can (re)start a process.
 4. **Resource cleanup on death** — reclaim memory/MMIO/IPC/IRQ, via caps or a grant
    table.

system/kernel/architecture/x86_64/idt.zig

@@ -3,9 +3,11 @@
 //! triple-faults and silently resets the machine. With it, the CPU vectors into
 //! our stubs, which capture the register state and hand it to a dispatcher.
 //!
-//! Vectors split in two: 0-31 are CPU exceptions (terminal — reported and
-//! halted); 32+ are device interrupts (a registered handler runs, the APIC is
-//! acknowledged, and we return to the interrupted code).
+//! Vectors split in two: 0-31 are CPU exceptions, handed to the `on_fault` hook
+//! and never returned from (the kernel's handler kills a faulting user process
+//! and reschedules, or halts the core for a kernel-mode fault); 32+ are device
+//! interrupts (a registered handler runs, the APIC is acknowledged, and we
+//! return to the interrupted code).
 
 const gdt = @import("gdt.zig");
 const tss = @import("tss.zig");
@@ -162,8 +164,9 @@ pub fn loadOnThisCpu() void {
 }
 
 /// Called by isr_common (isr.s) with a pointer to the trap frame. Exported so the
-/// assembly stubs can `call` it by name. Exceptions are terminal; device
-/// interrupts run their handler, get acknowledged, and return.
+/// assembly stubs can `call` it by name. Exceptions never return here (on_fault
+/// kills the faulting process or halts the core); device interrupts run their
+/// handler, get acknowledged, and return.
 export fn interruptDispatch(state: *CpuState) callconv(.c) void {
     if (state.vector < 32) {
         on_fault(state); // CPU exception — never returns

system/kernel/ipc-synchronous.zig

@@ -46,6 +46,7 @@ pub const EFAULT: i64 = 3; // buffer unmapped / out of the user half
 pub const ENOENT: i64 = 4; // no such registered service
 pub const ENOSPC: i64 = 5; // handle table or registry full
 pub const ENOMEM: i64 = 6; // out of memory
+pub const EPEER: i64 = 7; // peer died before replying (its process exited or was killed)
 
 /// A badge with this bit set is an asynchronous notification (e.g. an IRQ), not a
 /// message from a client — there is no reply owed. The low bits carry the source

system/kernel/kernel.zig

@@ -385,13 +385,42 @@ fn kib(frames: u64) u64 {
     return frames * abi.page_size / (1024);
 }
 
-/// Report a CPU exception and halt **this core**. There's no fault recovery yet, so
-/// the faulting core is terminal — but the fault is *contained* to it: on an
-/// application processor only that core stops, and the rest of the system keeps
-/// running (full recovery — kill the task, keep the core — is the resilience track,
-/// see docs/resilience.md). The report names the core so an AP fault is attributed,
-/// and goes to every output sink plus a POST code and a persistent breadcrumb.
+/// Whether a ring-3 exception is attributable to the process that raised it — and
+/// therefore recoverable by killing that process. NMI (2), double fault (8), and
+/// machine check (18) report machine or kernel trouble even when they arrive with a
+/// user CS (an NMI interrupts whatever happens to be running), so they stay terminal.
+fn recoverableFault(vector: u64) bool {
+    return switch (vector) {
+        2, 8, 18 => false,
+        else => true,
+    };
+}
+
+/// Report a CPU exception. Two outcomes (docs/resilience.md):
+///
+/// **A fault taken in user mode kills the faulting process, not the machine.** The
+/// kernel is intact — the CPU trapped onto the task's kernel stack — so the process
+/// is killed, everything it held (address space, IRQ bindings, IPC handles, device
+/// grants' frames) is reclaimed, and the core reschedules. A crashing driver takes
+/// itself down, never the OS.
+///
+/// **Everything else halts this core.** A kernel-mode fault means the trusted base
+/// itself is broken — there is nothing safe to kill — and NMI/#DF/#MC report machine
+/// trouble regardless of CS (`recoverableFault`). Even then the fault is *contained*:
+/// on an application processor only that core stops and the rest keep running. The
+/// report names the core so an AP fault is attributed, and goes to every output sink
+/// plus a POST code and a persistent breadcrumb. (A ring-3 fault on a *borrowed*
+/// kernel thread — process.run, the user-pf isolation probe — also lands here: there
+/// is no scheduled process to kill.)
 fn onException(state: *const architecture.CpuState) noreturn {
+    if (architecture.fromUser(state) and scheduler.currentIsUserProcess() and recoverableFault(state.vector)) {
+        statusPrint("\ndanos: process {d} killed by {s} (vector {d}) on core {d}\n", .{ scheduler.currentId(), architecture.exceptionName(state.vector), state.vector, scheduler.currentCpuIndex() });
+        statusPrint("  error code : 0x{x}\n", .{state.error_code});
+        statusPrint("  IP         : 0x{x:0>16}\n", .{architecture.instructionPointer(state)});
+        if (architecture.faultAddress(state)) |address| statusPrint("  fault addr : 0x{x:0>16}\n", .{address});
+        process.killCurrentProcess(); // reclaims everything, reschedules; never returns
+    }
+
     log.checkpoint(cp_exception);
     const core = scheduler.currentCpuIndex();
     // A fault is user-facing enough to paint on screen too (via statusPrint), on

system/kernel/process.zig

@@ -120,18 +120,10 @@ fn system_call(state: *architecture.CpuState) void {
     switch (@as(SystemCall, @enumFromInt(architecture.systemCallNumber(state)))) {
         .exit => {
             exit_code = architecture.systemCallArg(state, 0);
-            // A scheduled process drops its endpoint references, frees its address
-            // space, and reschedules; a borrowed test thread unwinds back to the
-            // kernel that entered it.
+            // A scheduled process tears down fully (terminateCurrent); a borrowed
+            // test thread unwinds back to the kernel that entered it.
             if (scheduler.currentIsUserProcess()) {
-                // Unbind before closeHandles: dropping the last reference destroys the
-                // Endpoint, and a still-bound GSI would have an ISR call
-                // notifyFromIsr on freed memory the next time the device fired.
-                // unbindAll also leaves the line masked, so a dead driver's device
-                // goes quiet rather than storming.
-                releaseIrqs(scheduler.current());
-                ipc.closeHandles(scheduler.current());
-                scheduler.exitUser();
+                terminateCurrent();
             } else architecture.userExit();
         },
         .yield => {
@@ -428,12 +420,44 @@ fn systemSpawn(state: *architecture.CpuState) void {
     fail(state); // no bundled binary by that name
 }
 
-/// Drop every IRQ binding `t` made. Called on exit, before the handle table is closed
-/// (which is what frees the endpoints an ISR would otherwise notify into).
-fn releaseIrqs(t: *scheduler.Task) void {
-    const flags = sync.enter();
-    defer sync.leave(flags);
-    irq.releaseOwner(t.id);
+/// Processes killed by a CPU fault rather than a clean exit. Evidence for the
+/// fault-recovery test, and a health signal a supervisor can consult later.
+pub var fault_kill_count: u64 = 0;
+
+/// Tear down the current user process and reschedule; never returns. Shared by the
+/// exit system call and the fault path (`killCurrentProcess`). The order matters:
+/// - IRQ bindings are dropped before the handle table closes: dropping the last
+///   endpoint reference destroys the Endpoint, and a still-bound GSI would have an
+///   ISR call notifyFromIsr on freed memory the next time the device fired.
+///   `releaseOwner` also leaves the line masked, so a dead driver's device goes
+///   quiet rather than storming.
+/// - A client this task still owes a reply to (it died between receive and reply)
+///   is failed with -EPEER rather than left blocked forever — a dead server must
+///   not hang its callers.
+pub fn terminateCurrent() noreturn {
+    const t = scheduler.current();
+    {
+        const flags = sync.enter();
+        defer sync.leave(flags);
+        irq.releaseOwner(t.id);
+        if (t.ipc_client) |client| {
+            t.ipc_client = null;
+            client.ipc_status = -ipc.EPEER;
+            scheduler.readyLocked(client); // its blocked `call` now returns the error
+        }
+        ipc.closeHandles(t);
+    }
+    scheduler.exitUser();
+}
+
+/// Kill the current user process in response to a CPU fault it raised in ring 3.
+/// The fault is confined to the process — the kernel trapped it on the task's own
+/// kernel stack and is intact — so everything the process held is reclaimed and the
+/// core reschedules. The system keeps running; only the faulting process dies
+/// (docs/resilience.md: fault -> kill -> continue).
+pub fn killCurrentProcess() noreturn {
+    fault_kill_count += 1;
+    terminateCurrent();
 }
 
 /// Resolve `(device_id, resource_index)` to a GSI this process is entitled to bind, or null.

system/kernel/tests.zig

@@ -118,6 +118,8 @@ pub fn run(case: []const u8, boot_information: *const BootInformation) void {
         userMemTest();
     } else if (eql(case, "user-pf")) {
         userPfTest();
+    } else if (eql(case, "fault-recovery")) {
+        faultRecoveryTest(boot_information);
     } else if (eql(case, "init")) {
         initTest(boot_information);
     } else if (eql(case, "process")) {
@@ -1190,6 +1192,87 @@ fn userPfTest() void {
     log("DANOS-TEST-RESULT: FAIL (user read of kernel memory did not fault)\n", .{});
 }
 
+/// Spawn a real scheduled ring-3 process whose body is the user-pf blob (a read of
+/// a kernel-only page, then a spin), so its first instruction raises #PF. Built by
+/// hand — address space, code page RO+X, stack page RW+NX — because the blob is a
+/// raw code fragment, not an ELF `spawnProcess` could load. Returns false if any
+/// allocation fails.
+fn spawnFaultingProcess() bool {
+    const blob = process.pfBlob();
+    const flags = sync.enter();
+    defer sync.leave(flags);
+
+    const aspace = architecture.createAddressSpace() orelse return false;
+    const code_frame = pmm.alloc() orelse {
+        architecture.destroyAddressSpace(aspace);
+        return false;
+    };
+    // Fill through the physmap (the user mapping is read-only); pad with int3 so a
+    // stray jump traps instead of sliding.
+    const code: [*]u8 = @ptrFromInt(boot_handoff.physicalToVirtual(code_frame));
+    @memset(code[0..abi.page_size], 0xCC);
+    @memcpy(code[0..blob.len], blob);
+    architecture.mapUserPageInto(aspace, process.code_virtual, code_frame, false, true); // RO + X
+
+    const stack_frame = pmm.alloc() orelse {
+        architecture.destroyAddressSpace(aspace); // frees code_frame too — it's mapped
+        return false;
+    };
+    architecture.mapUserPageInto(aspace, process.stack_virtual, stack_frame, true, false); // RW + NX
+
+    if (!scheduler.spawnUserLocked(aspace, process.code_virtual, process.stack_virtual + abi.page_size, 4)) {
+        architecture.destroyAddressSpace(aspace);
+        return false;
+    }
+    return true;
+}
+
+/// Fault recovery (docs/resilience.md step 2): a scheduled ring-3 process that
+/// faults must be killed — counted, resources reclaimed — while the rest of the
+/// system keeps running. init heartbeats before and after the kill are the proof
+/// the OS survived; the old behaviour (halt the core) would freeze the beat and
+/// time the harness out.
+fn faultRecoveryTest(boot_information: *const BootInformation) void {
+    log("DANOS-TEST-BEGIN: fault-recovery\n", .{});
+    check("bootloader handed over /system/services/init", boot_information.init_len != 0);
+    if (boot_information.init_len == 0) {
+        result();
+        return;
+    }
+    const image = @as([*]const u8, @ptrFromInt(boot_handoff.physicalToVirtual(boot_information.init_base)))[0..boot_information.init_len];
+
+    process.write_count = 0;
+    process.fault_kill_count = 0;
+    const spawned = if (process.spawnProcess(image, 4)) true else |_| false;
+    check("init spawned as the surviving process", spawned);
+
+    // A first heartbeat proves init runs before the fault.
+    scheduler.setPriority(1); // drop below the processes so they get the core
+    var deadline = architecture.millis() + 8000;
+    while (process.write_count < 1 and architecture.millis() < deadline) scheduler.yield();
+    scheduler.setPriority(4);
+    check("init heartbeat before the fault", process.write_count >= 1);
+
+    check("faulting process spawned", spawnFaultingProcess());
+
+    // The kill: the faulting process #PFs on its first instruction and the kernel
+    // reaps it instead of halting.
+    scheduler.setPriority(1);
+    deadline = architecture.millis() + 5000;
+    while (process.fault_kill_count < 1 and architecture.millis() < deadline) scheduler.yield();
+    scheduler.setPriority(4);
+    check("faulting process was killed (not the machine)", process.fault_kill_count == 1);
+
+    // Life after the kill: init must keep beating on the same core.
+    const beats_at_kill = process.write_count;
+    scheduler.setPriority(1);
+    deadline = architecture.millis() + 8000;
+    while (process.write_count < beats_at_kill + 2 and architecture.millis() < deadline) scheduler.yield();
+    scheduler.setPriority(4);
+    check("init kept heartbeating after the kill", process.write_count >= beats_at_kill + 2);
+    result();
+}
+
 /// The full PID-1 path: the bootloader read /system/services/init off the boot volume and
 /// handed it over; load it as a user ELF and spawn it as a real ring-3 process
 /// — the same call the normal boot path makes — then confirm it beats. init

test/qemu_test.py

@@ -199,6 +199,12 @@ CASES = [
     {"name": "user-pf",
      "expect": r"page fault \(vector 14\)[\s\S]*error code : 0x5[\s\S]*IP\s*: 0x00007000000000",
      "fail":   r"DANOS-TEST-RESULT: FAIL"},
+    # Fault recovery: a scheduled ring-3 process that faults is killed — resources
+    # reclaimed, core kept — while init's heartbeat proves the OS survived.
+    {"name": "fault-recovery",
+     "timeout": 60,
+     "expect": r"DANOS-TEST-RESULT: PASS",
+     "fail":   r"DANOS-TEST-RESULT: FAIL"},
     # The real user binary: the bootloader ships /system/services/init off the ESP, the
     # kernel loads the ELF and runs it in ring 3, and it writes + exits cleanly.
     {"name": "init",