danos/docs/threading-plan.md

Threading — build plan (runtime.Thread over a private thread ABI)

The ordered, checkpointable build-out for threading.md. Each milestone lands on its own and ends in a verifiable gate — shaped for a /loop run, like display-v2-plan.md. Read threading.md first for the why.

Locked decisions (do not relitigate)

• runtime.Thread mirrors std.Thread's API; the implementation is danos-native. Not literal std.Thread — that would break the private ABI.
• Threads are a narrow, per-binary opt-in. Default concurrency stays process + IPC (resilience.md); only a service that asks is built single_threaded = false.
• Blocking is futex-backed, never spin-backed — waiters park in the kernel so an idle core still halts (halting.md).
• New syscalls are private: extend abi.zig SystemCall after shm_physical = 36 (thread_spawn = 37, thread_exit = 38, futex_wait = 39, futex_wake = 40) + a library/runtime wrapper; user code never names a number.
• Restart granularity stays the process — a faulting thread kills its process; the supervisor restarts the process, which respawns its threads.

Conventions

Follow coding-standards.md: spell out non-acronym abbreviations, kebab-case file names, no Co-Authored-By trailers. New user binaries go through addUserBinary (with the new threaded flag where a binary spawns threads) and get packed into the initial-ramdisk; new syscalls extend abi.zig SystemCall + a library/runtime wrapper; test services live beside the code they exercise and register a ServiceId if they must be looked up.

How to verify along the way

Every gate is serial-checkable — no screenshots (this plan runs unattended). A thread proves it ran by writing to shared memory the parent reads back, and proves parallelism by stamping the core index it ran on (like the smp/affinity cases).

• zig build test — host unit tests (closure packing, mutex state machine, futex wrapper encodings).
• python3 test/qemu_test.py <case> — boots the kernel in QEMU; asserts on serial markers. Thread cases set smp: true (real parallelism) and bump mem (they boot the process/scheduler stack); each milestone adds its case to CASES so its gate is runnable.
• Guardrail every milestone: the concurrency-sensitive existing cases stay green — smoke, sched, priority, smp, affinity, process, process-kill, supervision, fault-recovery, vfs-client-death, ipc/ipc-cap, display-service. A threading change that regresses those is rejected.

Unattended execution (the loop contract)

This plan runs to completion without human input. Every design choice is already fixed in Locked decisions; the checkboxes are the only state. A loop iteration must:

1. Resume at the first milestone that still has an unchecked - [ ]. (All earlier milestones are done — do not revisit them.)
2. Work on a branch. On the first iteration, branch off main (e.g. threading); never commit threading work to main. All work stays local — do not push.
3. Implement every unchecked item in that milestone, including adding its -Dtest-case to CASES in test/qemu_test.py (with smp: true / a mem bump where noted) so the gate is runnable.
4. Run the gate: python3 test/qemu_test.py <case>, then the full guardrail set, then zig build (clean) and zig build test (green).
5. Decide, do not ask:
   • Green = the milestone's case prints its stated marker(s) and reports PASS, the whole guardrail set passes, zig build is clean, and host tests are green. → tick this milestone's boxes and its **Gate:**-referenced case, git commit (threads(M<n>): <summary>, no Co-Authored-By trailer per coding-standards.md), and continue to the next milestone in the same iteration if budget remains; otherwise let the loop re-fire.
   • Red = anything above fails. Diagnose from the captured serial log (zig-out/qemu-test/<case>-failed-serial.log) and fix in place, then re-run — up to 3 fix attempts for that gate. A concurrency case that fails then passes on a bare re-run is flaky, not green: re-run it twice more and treat green only if it passes all; otherwise fix the race (a real threading bug), don't paper over it.
6. A genuinely ambiguous fork is not a stop. Pick the option most consistent with threading.md's Locked decisions, note the choice in the commit message, and continue. Do not pause for confirmation on in-scope, reversible work — this plan is that authorization.

The only stop conditions:

• Done — every milestone box is checked (M1–M6), zig build clean, whole thread-* suite + guardrail green. Update threading.md's status line to "built" (that is M6's own task) and stop.
• Blocked — a gate is still red after 3 fix attempts, or a step needs something outside the repo (a toolchain change, new hardware, a decision no locked decision covers). Append > **BLOCKED (M<n>):** <what failed, what was tried, the serial marker missing> under that milestone, commit the WIP on the branch, and stop. Do not thrash further and do not silently skip the milestone.

Nothing else warrants stopping — not "should I proceed?", not "is this right?". The checkboxes + git history are the resumable record; the next iteration picks up from the first unchecked box.

---

M1 — Address-space refcount (kernel foundation, no API, no behaviour change) ✅

The one invariant change threads require, landed and proven before anything shares an address space. Today aspace is 1:1 with a task and teardown destroys it on any user task's exit; make destruction happen on the last exit.

• A refcount keyed by the address-space root, held in scheduler.zig (aspace_refs): retainAspace takes a reference in spawnUserLocked (on the success path, after the slot + stack are secured), all under the big kernel lock.
• Both task-teardown paths (scheduler.zig: exitUserLocked and destroyTaskLocked) call releaseAspace, which decrements and only destroyAddressSpaces at zero; an unretained space (hand-built test spaces) is destroyed directly, preserving prior behaviour.
• -Dtest-case=aspace-refcount: spawn and reap several ring-3 processes in sequence and assert (via test-observable liveAspaceCount/aspaceDestroyCount) that the live-space count returns to baseline and destructions advance by exactly that many — each space destroyed exactly once, no leak, no double-free. (Refcount observables, not raw frame counts, since kernel stacks are still leaked on exit.)

Gate (met): python3 test/qemu_test.py aspace-refcount passes (aspace-refcount: spaces released to baseline ok → DANOS-TEST-RESULT: PASS), and the full guardrail set passes unchanged — 13/13 (smoke, sched, priority, smp, affinity, process, process-kill, supervision, fault-recovery, vfs-client-death, ipc, ipc-cap, display-service); default zig build clean, zig build test green. The reframing is invisible until an aspace is actually shared.

M2 — thread_spawn + thread_exit: a thread runs in the shared address space ✅

Spawn only — no join yet. Prove a second task executes in the caller's address space and exits cleanly.

• abi.zig: thread_spawn = 37, thread_exit = 38. Handlers in process.zig; thread_spawn calls scheduler.spawnThread (shares the caller's aspace, retainAspace); thread_exit ends the task like a process exit(0) (terminateCurrent → releaseAspace). The closure pointer is delivered in the new thread's rdi via a new jump_to_user_arg asm path (t.user_arg, 0 for a process) — no naked runtime asm.
• library/runtime/thread.zig (barrel-exported as runtime.Thread): spawn maps a stack (mmap), heap-allocates the {args} closure, and calls thread_spawn(&Closure.entry, stack_top, closure); Closure.entry (a plain C-ABI Zig fn, closure in rdi) runs the function and calls thread_exit. Stack top is 16-aligned-minus-8 for the C entry.
• A threaded flag on the user-binary recipe (addThreadedUserBinary → single_threaded = false); thread-test is the first opt-in binary.
• -Dtest-case=thread-spawn: thread-test spawns a worker that writes a sentinel to a shared global and release-stores done; the main thread acquire-polls done and asserts the shared global holds the sentinel — proof the worker ran in the same address space.

Gate (met): python3 test/qemu_test.py thread-spawn passes (thread-test: child ran in shared aspace ok → DANOS-TEST-RESULT: PASS); guardrail set 16/16 green (incl. args/init/process, which exercise the new jump_to_user_arg process path with arg 0) plus aspace-refcount; zig build clean, zig build test green.

Note (deferred to M3+): the mmap arena is per-task (heap_next), so two threads in one aspace that both mmap would collide. Fine for M2 (only the parent maps, for the child's stack); make the arena per-aspace and the runtime heap thread-safe alongside the Mutex work (M5).

M3 — join + detach + real parallelism

• join over the existing exit-notification path (process-lifecycle.md): spawn passes a per-thread exit_endpoint; join blocks in ipc_reply_wait until the child-exit notice for that tid, then munmaps the stack. detach relinquishes the join right; the reaper reclaims a detached thread's stack + slot on exit.
• runtime.Thread.join / detach / getCurrentId (id = kernel task id).
• -Dtest-case=thread-join (smp: true): the parent spawns N threads that each do K @atomicRmw-increments on a shared counter and stamp the core index they ran on; the parent joins all N and asserts counter == N*K and distinct cores > 1 (genuine cross-core parallelism). A detached thread sub-check confirms no leak.

Gate: python3 test/qemu_test.py thread-join logs thread: N joined, counter=N*K, cores=<>1>; guardrail set green (esp. smp, affinity, process-kill).

M4 — Futex: the one blocking primitive

• abi.zig: futex_wait = 39, futex_wake = 40. Kernel wait-queue keyed by (aspace_root, vaddr); futex_wait(addr, expected, timeout) parks the task iff *addr == expected (re-checked under the lock) and returns on wake or timeout; futex_wake(addr, count) moves up to count waiters back to ready. No spinning — a parked waiter leaves its core free to hlt.
• runtime.Thread.Futex (wait / timedWait / wake) over the wrappers.
• -Dtest-case=thread-futex: thread A prints waiting, futex_waits on a word; thread B sets the word and futex_wakes; A prints woke. Serial order waiting → waking → woke proves the kernel handoff (not a spin). A second check confirms timedWait returns error.Timeout when nobody wakes it.

Gate: python3 test/qemu_test.py thread-futex logs thread: futex handoff ordered ok and thread: futex timeout ok; guardrail set green.

M5 — Mutex + Condition + Semaphore

• runtime.Thread.Mutex (atomic fast path, futex_wait/wake slow path), Condition (wait/timedWait/signal/broadcast), Semaphore — the same state machines std.Thread uses, ported onto our Futex. Host unit tests for the lock/unlock/wait state transitions.
• Migrate join to a futex completion word (the std shape) — drops the per-thread endpoint from M3.
• -Dtest-case=thread-mutex (smp: true): a bounded producer/consumer over Mutex + Condition moves K items across cores; assert the final tally is exactly K with no lost wakeups (consumer never misses an item, producer never overruns the bound), and the consumer reaches a parked marker (it blocked, it didn't spin).

Gate: python3 test/qemu_test.py thread-mutex logs thread: produced/consumed K, no lost wakeups; zig build test covers the mutex/condition state machines; guardrail set green.

M6 — TLS, getCurrentId polish, docs, and CI wiring

• Per-thread TLS: thread_spawn sets fs.base to a runtime-allocated per-thread block; enough for getCurrentId and a threadlocal slot. Only expanded if a consumer needs full threadlocal.
• RwLock / WaitGroup if a consumer wants them (else deferred).
• All thread-* cases wired into test/qemu_test.py; threading.md status line updated to "built"; a short worked example in the doc.
• -Dtest-case=thread-tls: two threads read getCurrentId (distinct) and each writes its own threadlocal slot without cross-talk.

Gate: python3 test/qemu_test.py thread-tls logs thread: per-thread id + tls ok; the whole thread-* suite plus the full guardrail set pass; default zig build is clean and zig build test is green.

---

Deferred (explicitly not in this plan)

• Cross-process shared-memory futex — the (aspace, vaddr) key can become a physical-address key so two processes share a futex through an shm region. Not needed for intra-process threads.
• Per-thread priorities / affinity distinct from the process — threads inherit the process priority (scheduling.md); revisit only if it earns its keep.
• Per-thread signal delivery — signals stay process-scoped (process-lifecycle.md).
• A pthread/POSIX surface — the API is std.Thread-shaped Zig, nothing more.
• A real std.Thread backend — arrives with self-hosting (zig-self-hosting.md); it sits on these same primitives, so it swaps the impl under runtime.Thread, not the call sites.