danos/docs/os-development/shared-fate-plan.md

Shared fate: whole-process death (plan)

Status: implemented 2026-07-22 (branch shared-fate), M1–M4 all landed; leader thread_exit → -EPERM as decided. One scope addition forced by M4: the per-task DMA/shared-memory arena cursors moved to the per-space object (the shm-mapping-ref test could not distinguish corruption-by-remap from corruption-by-free while sibling threads overlapped the arena) — the same move the mmap/MMIO cursors made in threading M7.

threading.md promises that a process dies whole — a fault in any thread, or a kill, takes down every thread. The kernel doesn't do that yet: every death path (exit, thread_exit, a ring-3 fault, process_kill) tears down exactly one Task, and the address-space refcount keeps the space alive for the siblings — so a faulting worker orphans its threads, which keep running in the possibly-corrupted address space (process.zig killCurrentProcess; scheduler.zig exitUserLocked). This plan closes that gap the way Linux, Windows, and Fuchsia all did: the process is the unit of fate; only a voluntary thread_exit is per-thread. The supervisor contract — one exit notification, then process_exit_reason — is deliberately unchanged.

The contract

Event	Who dies	Reason the supervisor reads (leader's record)
CPU fault in any thread (recoverable vector)	the whole group	the fault class (segmentation_fault, …)
process_kill on any member id	the whole group	killed
exit(code) from any thread	the whole group	exited (0) / aborted (≠0)
thread_exit from a worker	that worker only	— (per-task record: exited)
thread_exit from the leader	nobody — refused, -EPERM (decided, below)	—
NMI, double fault, machine check	the core halts (unchanged)	—

exit gaining group semantics is the exit_group lesson from Linux: the runtime's main-return path calls exit, and a process whose main returned must not leave workers running. thread_exit (what the worker trampoline calls) keeps today's per-thread behavior, refcount and all.

The leader-thread_exit rule (decided: refuse). The syscall is reachable from the leader even though the runtime never does it. Options weighed: (i) refuse with -EPERM — cheapest and honest; the group ends only through exit/fault/kill; (ii) escalate to exit(0) — Linux-flavored, but silently turns a (buggy) library call into process death; (iii) a Linux-style zombie leader whose slot survives until the group ends — the most faithful, and by far the most machinery. (i) chosen at sign-off; rows and tests below follow it.

Group identity: a leader id

Nothing on Task names a process today — threads are tied to their process only by an equal address_space, their name is "thread", and their supervisor is whichever task spawned them (possibly another worker), so supervision links form a chain, not a group. Scanning by address_space is also fragile during teardown, because both death paths zero it.

So: add leader: u32 to Task — the Linux tgid, in danos clothes. spawnProcessSupervised sets leader = own id; spawnThreadSupervised copies the caller's leader; kernel tasks keep leader = 0, which is never followed. The leader id is exactly the id system_spawn returned to the supervisor, so the outside world already speaks it. Group membership = equal leader, where a live member means state ∈ {.ready, .blocked, .running} — the same filter taskByIdLocked applies; .reaping corpses are excluded. While the field is being introduced, add leader to ProcessDescriptor too (the ABI is private, so this is cheap now and lets process_enumerate consumers group threads).

process_kill re-derives its authority through the leader — with the existing guard order preserved: a kernel task (address_space == 0) is -ESRCH before any leader resolution (the kernel test asserts exactly that). Then: resolve the target, follow target.leader, require leader.supervisor == caller. The kill capability becomes per-process, aimed at any member id, and the odd today-behavior where a worker can be individually killed by its spawning thread disappears. (Audited: nothing in-tree kills a worker tid or relies on thread-supervisor kill semantics.)

The group-dying latch

AddressSpaceRef — one per space, refcounted by its member tasks, recycled with a full struct re-init — gains the group-death state:

dying: bool = false,           // set by the first trigger; never cleared
group_reason: abi.ExitReason,  // what the leader's record will say
exit_endpoint: ?*ipc.Endpoint, // the leader's counted ref, moved here
leader: u32,

The latch answers three attacks the red team confirmed against a latch-free design:

• The spawn gate. A member already inside thread_spawn on another core when the fan-out runs (it passed the syscall-entry kill_pending check, then spun on the BKL) would otherwise complete the spawn after the fan-out's lock hold ends — a fresh, uncondemned member that escapes the kill and, worse, holds a space reference that keeps the group-death hook from ever firing. Fix: retainAddressSpace (equivalently spawnUserLocked) refuses a dying space; the in-flight thread_spawn fails with -ESRCH under the same lock that would have created the member.
• Concurrent triggers. A second member faulting (or exiting) on another core while the first fan-out runs must not re-run the fan-out, double-bump fault_kill_count, or re-stamp reasons. Every kill path checks the latch first: already dying → skip straight to terminateCurrentLocked, no stamp, no count. First trigger wins, deterministically. exit_reason and fault_kill_count writes move under the BKL as part of this.
• Notification ownership. The leader's exit_endpoint is a counted birth-to-death reference dropped at notify time. The stamp pass moves that reference onto the AddressSpaceRef and nulls Task.exit_endpoint in the same hold, so the leader's own releaseTaskResourcesLocked sees null (no early notify, no double drop); the group-death hook notifies and drops exactly once.

The fan-out: killGroupLocked

One new function in process.zig, running under a single BKL hold (built from the *Locked primitives — the lock is non-recursive, and terminateCurrentLocked never returns, which forces the shape):

killGroupLocked(leader: u32, reason: ExitReason, trigger: ?*Task)
  0. Latch: AddressSpaceRef.dying = true, stash {reason, leader,
     leader's exit_endpoint (moved)}.
  1. Stamp pass: the LEADER's exit_reason = reason — the leader's
     record is the one the supervisor can read, so it carries the
     group reason even when the trigger is a worker. The trigger
     also keeps `reason` (its own record tells the truth); every
     other live member gets .killed. All members get kill_pending.
     Stamping precedes any teardown, because recordExitLocked
     snapshots the reason first thing.
  2. Reap pass, to fixpoint: reap every member in .ready or .blocked
     via reapTaskLocked, re-reading Task.state each iteration — a
     member's teardown can wake another member (-EPEER wakes, joiner
     wakes), flipping it .blocked → .ready behind the scan cursor.
     Terminates in ≤ one pass per member: the scrub calls in
     releaseTaskResourcesLocked (abandonSenderLocked,
     removeFromWaitQueueLocked, forgetIpcClientLocked,
     killOwnedEndpointsLocked) run before destroy, so no wake path
     holds a pointer to a reaped member.
  3. Members .running on other cores stay condemned (kill_pending);
     a condemned member dies at its next syscall entry, at its own
     core's next tick while in user mode, or — once it blocks or is
     preempted — at any core's next tick reap. There is no kill IPI.
     (The entry check reads kill_pending unlocked; benign on
     x86-TSO — a missed read is caught by the next delivery point —
     but make the field atomic when touching it.)
  4. If the current task is a member (fault, exit, in-group kill):
     terminateCurrentLocked, last, because it switches away and the
     reap paths free the kernel stack being stood on.
     If the caller is outside the group (supervisor kill): return.

The invariants this preserves, each load-bearing today:

• Only .ready/.blocked tasks are reaped synchronously. A member running on another core can only be condemned — it tears itself down after switching CR3 off the dying page tables (the stack it stands on is freed later by the reap list), and its address-space reference protects the page tables its CR3 still points at. Force-destroying the space under a running sibling is the one unrecoverable mistake available here.
• The refcount decides when the space dies. Reaping N members drops N references; the last drop — possibly on a condemned sibling's core, a tick later — destroys the space. No path forces it.
• fault_kill_count bumps once per group, not per member (fault-recovery asserts == 1 exactly); the latch is what enforces this under racing faults.
• Per-tid resource sweeps stay per-tid. Each member's releaseTaskResourcesLocked releases what that tid owns — claims, GSI/MSI bindings, registered endpoints, handles. That keying is correct under shared fate (and is today's hazard: a lone worker death already yanks its claims out from under live siblings). A worker that does carry an exit_endpoint (the ABI allows it; the runtime passes no_cap) keeps today's per-task posting at its own teardown — only the leader's notification moves.

When is the group dead? The notification

Today each task posts its own exit notification as the last step of its release, so a supervisor observes a fully-released child. For a group that guarantee must hold for the whole group: if the leader's notification fires while a condemned sibling still runs on another core, the device manager can respawn the driver into a claim conflict with a not-yet-dead sibling.

The clean fix falls out of the refcount: the group is dead exactly when the address space is destroyed. releaseAddressSpace's last-drop path calls a new group_exit_hook (the scheduler already calls up through hooks — terminate_current_hook — precisely to keep this layering), which:

1. re-stamps the leader's exit record with the stashed group_reason — the record is written (again) at group-death time, so "the reason is recorded before the notification posts" stays true and a supervisor can never be notified and then read -ESRCH because the burst evicted an old record;
2. posts the leader's exit notification (and subscriber broadcast) from the stashed endpoint, and drops that reference — exactly once.

Both releaseAddressSpace call sites (exitUserLocked, destroyTaskLocked) run under the BKL, so the hook does too; its wakes are safe at both (verified). For a single-threaded process the behavior is externally indistinguishable from today — the order of notify vs. destroy inverts, but both sit inside one lock hold, so no other core can observe the space destroyed but the notification unposted, or vice versa. That sentence is the correctness argument; it is also the first invariant to re-examine if the BKL is ever split, along with killGroupLocked's single-hold atomicity. (Hand-built spaces that were never retained take the immediate-destroy path and are out of the hook's scope.)

Workers' exit_subscribers broadcasts still fire per task — the FAT server's dead-client sweep is keyed by tid and needs those.

Signals. signal_bind is per-task and the service harness binds on the main thread, so signals address the leader in practice; that stays. During a group death, process_signal may return 0 (accepted by a condemned member — never delivered, every delivery point kills first) or -ESRCH (member already reaped); init's stop sequence already tolerates both, and its timer escalation to process_kill covers the gap. process_signal follows process_kill's leader re-key for consistency.

The shared-memory frame hazard

dropSharedMemoryReference frees a region's physical frames when the last handle reference drops, but mappings die only with the address space. If the last handle lived in a torn-down member while any task still has the region mapped, that task holds a live mapping onto freed frames — and the red team showed this is not group-specific: a plain thread_exit of the handle-holding thread, or a last-ref drop by a task outside the dying group during the condemned window, hits the same use-after-free.

So the fix is a property of the object, not the dropper: give SharedMemoryObject a per-mapping reference — shared_memory_map (and create's self-map) retains; each space's destruction releases. "Last reference" then means no handles and no mappings, both hazard paths collapse into the existing refcount, and no group-kill special case is needed at all.

Deliberately unchanged

• Worker thread_exit: per-thread, full per-tid resource sweep, refcount drop.
• The condemned-but-running window: a member on another core can finish its in-flight syscall and run user code for up to a tick before dying — identical to today's single-task process_kill semantics ("prompt but asynchronous, like a Unix signal"). A kill IPI would shrink it; it is not part of this plan.
• thread_join returns 0 for a killed thread; joiners inside a dying group are woken and then reaped like any member.
• The .reaping state, reap lists, and stack reaper.

Accepted limits (documented, not fixed here)

• Notify-ring overflow: a group death posts one subscriber badge per member into 8-slot rings; a >8-member group can drop badges. Group size is bounded by the 48-task table; today's largest production group is 2 (display) and thread-test already reaches 5.
• Exit-record ring pressure: one 64-entry ring, one record per member — made harmless for the supervisor by the hook's group-death re-stamp.
• Enumerate shows a partial group mid-death: reaped members vanish at once, condemned members linger up to a tick (audited: no in-tree consumer misbehaves; the leader field in ProcessDescriptor lets future consumers group correctly).
• Pre-existing reap race, not widened: a task preempted mid-syscall is .ready with in_system_call = true, and the tick's reap loop will reap it — an existing hazard the fan-out inherits but must not add new instances of. Filed to investigate separately.
• Per-task DMA/shm cursors — fixed during M4 after all: the shm-mapping-ref test tripped the overlap (the sibling's churn regions mapped over the worker's region), so both cursors moved to the AddressSpaceRef like the mmap/MMIO cursors before them. The post-implementation review then found the other half: the shm/DMA page-table walks and their pmm/heap calls ran outside the big kernel lock — pre-existing, but fatal once siblings were invited to race them (and a plausible root for the long-standing intermittent AP ring-3 fault at the shm base). All three paths now follow the mmap discipline: metadata and allocation under one hold, the map itself per-page under brief holds.
• Mapping-record slots are never recycled: 16 per space, one per shared_memory_create/map, freed only at space destruction (there is no shm unmap). A long-lived compositor that churns surfaces will hit the cap; the failure is a clean refused create, and slot recycling can ride whatever adds shared_memory_unmap.
• Two properties lack direct tests: the spawn gate (an in-flight thread_spawn racing the fan-out — inherently nondeterministic to arrange; covered by code inspection and the -ESRCH path) and the process_signal leader re-key (exercised only implicitly by the signals case).

Milestones

• M1 — the leader id. Task.leader (kernel tasks: 0, never followed), set on both spawn paths; leader added to ProcessDescriptor; process_kill and process_signal re-keyed (kernel-task -ESRCH guard before leader resolution). No fan-out yet. Existing tests must pass untouched.
• M2 — the latch + fan-out. AddressSpaceRef.dying + stash; retainAddressSpace refuses dying spaces; killGroupLocked; wire the fault path, exit, and process_kill into it; leader thread_exit → -EPERM; exit_reason/fault_kill_count writes under the BKL; kill_pending made atomic. Group notification via the group_exit_hook re-stamp + post.
• M3 — shared-memory mapping refs. SharedMemoryObject counts mappings; space destruction releases them; frames free only at zero handles and zero mappings.
• M4 — tests + docs. New -Dtest-cases (all smp: 4 where cross-core matters), driving thread-test with new argv modes:
  • thread-fault-group: a worker faults; assert both tasks gone from enumerate, fault_kill_count == 1, process_exit_reason(leader) == segmentation_fault, address-space and stack-bytes counters return to base.
  • kill-threaded-group: process_kill(leader) with a worker spinning on another core; assert the worker dies by the deferred path, exactly one exit badge, delivered only after both members are dead, and process_exit_reason(leader) == .killed.
  • kill-via-worker-tid: process_kill(worker) kills the whole group; -EPERM for a non-supervisor aiming at the worker.
  • racing-triggers: two members fault/exit simultaneously on different cores; assert a deterministic leader reason and fault_kill_count == 1.
  • exit-group: a worker calls exit(3); group dies, leader reason .aborted.
  • leader-thread-exit: asserts the chosen rule (-EPERM, workers unaffected).
  • thread-exit-solo: regression — worker thread_exit still leaves siblings running.
  • group-claim-release: a member claims a device; assert the claim is free and the leader notification arrives only after every member is dead.
  • shm-mapping-ref: last handle dropped by a dying thread; sibling's mapping stays valid until space death (M3 regression). Then update threading.md (the shared-fate gap note), process-lifecycle.md, process-management.md, and ipc.md/drivers.md mentions.