danos/docs/threading-plan.md

# Threading — build plan (`runtime.Thread` over a private thread ABI)

The ordered, checkpointable build-out for [threading.md](threading.md). Each milestone
lands on its own and ends in a **verifiable gate** — shaped for a `/loop` run, like
[display-v2-plan.md](display-v2-plan.md). Read threading.md first for the *why*.

## Locked decisions (do not relitigate)

- **`runtime.Thread` mirrors `std.Thread`'s API; the implementation is danos-native.**
  Not literal `std.Thread` — that would break the [private ABI](syscall.md).
- **Threads are a narrow, per-binary opt-in.** Default concurrency stays process + IPC
  ([resilience.md](resilience.md)); only a service that asks is built
  `single_threaded = false`.
- **Blocking is futex-backed, never spin-backed** — waiters park in the kernel so an
  idle core still halts ([halting.md](halting.md)).
- **New syscalls are private**: extend [abi.zig](../system/abi.zig) `SystemCall` after
  `shm_physical = 36` (`thread_spawn = 37`, `thread_exit = 38`, `current_core = 39`,
  `futex_wait = 40`, `futex_wake = 41`) + a `library/runtime` wrapper; user code never names a number.
- **Restart granularity stays the process** — a faulting thread kills its process; the
  supervisor restarts the process, which respawns its threads.

## Conventions

Follow [coding-standards.md](coding-standards.md): spell out non-acronym abbreviations,
kebab-case file names, no `Co-Authored-By` trailers. New user binaries go through
`addUserBinary` (with the new `threaded` flag where a binary spawns threads) and get
packed into the initial-ramdisk; new syscalls extend [abi.zig](../system/abi.zig)
`SystemCall` + a `library/runtime` wrapper; test services live beside the code they
exercise and register a `ServiceId` if they must be looked up.

## How to verify along the way

**Every gate is serial-checkable — no screenshots** (this plan runs unattended). A
thread proves it ran by writing to **shared memory** the parent reads back, and proves
parallelism by stamping the **core index** it ran on (like the `smp`/`affinity` cases).

- `zig build test` — host unit tests (closure packing, mutex state machine, futex
  wrapper encodings).
- `python3 test/qemu_test.py <case>` — boots the kernel in QEMU; asserts on serial
  markers. Thread cases set `smp: true` (real parallelism) and bump `mem` (they boot
  the process/scheduler stack); each milestone **adds its case to `CASES`** so its gate
  is runnable.
- **Guardrail every milestone:** the concurrency-sensitive existing cases stay green —
  `smoke`, `sched`, `priority`, `smp`, `affinity`, `process`, `process-kill`,
  `supervision`, `fault-recovery`, `vfs-client-death`, `ipc`/`ipc-cap`,
  `display-service`. A threading change that regresses those is rejected.

## Unattended execution (the loop contract)

This plan runs to completion **without human input**. Every design choice is already
fixed in *Locked decisions*; the checkboxes are the only state. A loop iteration must:

1. **Resume** at the first milestone that still has an unchecked `- [ ]`. (All earlier
   milestones are done — do not revisit them.)
2. **Work on a branch.** On the first iteration, branch off `main` (e.g. `threading`);
   never commit threading work to `main`. All work stays local — **do not push**.
3. **Implement** every unchecked item in that milestone, including adding its
   `-Dtest-case` to `CASES` in [test/qemu_test.py](../test/qemu_test.py) (with
   `smp: true` / a `mem` bump where noted) so the gate is runnable.
4. **Run the gate**: `python3 test/qemu_test.py <case>`, then the full **guardrail
   set**, then `zig build` (clean) and `zig build test` (green).
5. **Decide, do not ask:**
   - **Green** = the milestone's case prints its stated marker(s) and reports `PASS`,
     the whole guardrail set passes, `zig build` is clean, and host tests are green.
     → tick this milestone's boxes **and** its `**Gate:**`-referenced case, `git commit`
     (`threads(M<n>): <summary>`, no `Co-Authored-By` trailer per
     [coding-standards.md](coding-standards.md)), and continue to the next milestone in
     the same iteration if budget remains; otherwise let the loop re-fire.
   - **Red** = anything above fails. Diagnose from the captured serial log
     (`zig-out/qemu-test/<case>-failed-serial.log`) and fix in place, then re-run — up to
     **3 fix attempts** for that gate. A concurrency case that fails then passes on a
     bare re-run is **flaky, not green**: re-run it **twice more** and treat green only
     if it passes all; otherwise fix the race (a real threading bug), don't paper over
     it.
6. **A genuinely ambiguous fork is not a stop.** Pick the option most consistent with
   [threading.md](threading.md)'s *Locked decisions*, note the choice in the commit
   message, and continue. Do not pause for confirmation on in-scope, reversible work —
   this plan is that authorization.

**The only stop conditions:**

- **Done** — every milestone box is checked (M1–M6), `zig build` clean, whole
  `thread-*` suite + guardrail green. Update threading.md's status line to "built" (that
  is M6's own task) and stop.
- **Blocked** — a gate is still red after 3 fix attempts, or a step needs something
  outside the repo (a toolchain change, new hardware, a decision no locked decision
  covers). Append `> **BLOCKED (M<n>):** <what failed, what was tried, the serial
  marker missing>` under that milestone, commit the WIP on the branch, and stop. Do not
  thrash further and do not silently skip the milestone.

Nothing else warrants stopping — not "should I proceed?", not "is this right?". The
checkboxes + git history are the resumable record; the next iteration picks up from the
first unchecked box.

---

## M1 — Address-space refcount (kernel foundation, no API, no behaviour change) ✅

The one invariant change threads require, landed and proven **before** anything shares
an address space. Today aspace is 1:1 with a task and teardown destroys it on any user
task's exit; make destruction happen on the **last** exit.

- [x] A refcount keyed by the address-space root, held in `scheduler.zig`
      (`aspace_refs`): `retainAspace` takes a reference in `spawnUserLocked` (on the
      success path, after the slot + stack are secured), all under the big kernel lock.
- [x] Both task-teardown paths ([scheduler.zig](../system/kernel/scheduler.zig):
      `exitUserLocked` and `destroyTaskLocked`) call `releaseAspace`, which decrements
      and only `destroyAddressSpace`s at **zero**; an unretained space (hand-built test
      spaces) is destroyed directly, preserving prior behaviour.
- [x] `-Dtest-case=aspace-refcount`: spawn and reap several ring-3 processes in sequence
      and assert (via test-observable `liveAspaceCount`/`aspaceDestroyCount`) that the
      live-space count returns to **baseline** and destructions advance by exactly that
      many — each space destroyed exactly once, no leak, no double-free. (Refcount
      observables, not raw frame counts, since kernel stacks are still leaked on exit.)

**Gate (met):** `python3 test/qemu_test.py aspace-refcount` passes
(`aspace-refcount: spaces released to baseline ok` → `DANOS-TEST-RESULT: PASS`), and the
full guardrail set passes unchanged — 13/13 (`smoke`, `sched`, `priority`, `smp`,
`affinity`, `process`, `process-kill`, `supervision`, `fault-recovery`,
`vfs-client-death`, `ipc`, `ipc-cap`, `display-service`); default `zig build` clean,
`zig build test` green. The reframing is invisible until an aspace is actually shared.

## M2 — `thread_spawn` + `thread_exit`: a thread runs in the shared address space ✅

Spawn only — no join yet. Prove a second task executes in the **caller's** address
space and exits cleanly.

- [x] [abi.zig](../system/abi.zig): `thread_spawn = 37`, `thread_exit = 38`. Handlers in
      process.zig; `thread_spawn` calls `scheduler.spawnThread` (shares the caller's
      aspace, `retainAspace`); `thread_exit` ends the task like a process `exit(0)`
      (`terminateCurrent` → `releaseAspace`). The closure pointer is delivered in the new
      thread's **rdi** via a new `jump_to_user_arg` asm path (`t.user_arg`, 0 for a
      process) — no naked runtime asm.
- [x] `library/runtime/thread.zig` (barrel-exported as `runtime.Thread`): `spawn` maps a
      stack (`mmap`), heap-allocates the `{args}` closure, and calls
      `thread_spawn(&Closure.entry, stack_top, closure)`; `Closure.entry` (a plain C-ABI
      Zig fn, closure in rdi) runs the function and calls `thread_exit`. Stack top is
      16-aligned-minus-8 for the C entry.
- [x] A `threaded` flag on the user-binary recipe (`addThreadedUserBinary` →
      `single_threaded = false`); `thread-test` is the first opt-in binary.
- [x] `-Dtest-case=thread-spawn`: `thread-test` spawns a worker that writes a sentinel to
      a **shared** global and release-stores `done`; the main thread acquire-polls `done`
      and asserts the shared global holds the sentinel — proof the worker ran in the same
      address space.

**Gate (met):** `python3 test/qemu_test.py thread-spawn` passes
(`thread-test: child ran in shared aspace ok` → `DANOS-TEST-RESULT: PASS`); guardrail set
16/16 green (incl. `args`/`init`/`process`, which exercise the new `jump_to_user_arg`
process path with arg 0) plus `aspace-refcount`; `zig build` clean, `zig build test`
green.

> **Note (deferred to M3+):** the mmap arena is per-*task* (`heap_next`), so two threads
> in one aspace that both `mmap` would collide. Fine for M2 (only the parent maps, for the
> child's stack); make the arena per-aspace and the runtime heap thread-safe alongside the
> `Mutex` work (M5).

## M3 — `join` + `detach` + real parallelism ✅

- [x] `join` over the existing exit-notification path
      ([process-lifecycle.md](process-lifecycle.md)): `thread_spawn` gained a 4th arg, an
      `exit_endpoint` handle (resolved + refcounted like `spawnProcessSupervised`, via
      `spawnThreadSupervised`); `join` blocks in `ipc_reply_wait` on that endpoint until
      the child-exit notice for its `tid`, then `munmap`s the stack. `detach` relinquishes
      the join right (its stack is reclaimed at process exit — kernel-reaper reclaim for
      detached threads is deferred; see note).
- [x] `runtime.Thread.join` / `detach`, plus `Thread.currentCore()` (a new `current_core`
      = 39 syscall) for the parallelism proof. `getCurrentId` deferred to M6 (TLS), where
      a lighter self-id fits. The closure now rides the **thread's own stack** (not the
      heap) — private per thread, so spawn/join touch no shared heap.
- [x] `-Dtest-case=thread-join` (`smp: 4`): `thread-test` join mode spawns N=4 workers
      that each do K=100k `@atomicRmw`-increments on a shared counter and stamp the core
      they ran on; the main thread joins all N and asserts `counter == N*K` **and**
      `@popCount(cores_seen) > 1` (genuine cross-core parallelism), then a detached worker
      proves `detach` runs without a join.

**Gate (met):** `python3 test/qemu_test.py thread-join` passes (`thread-test: join ok` →
`DANOS-TEST-RESULT: PASS`), robust across 4 runs; guardrail 17/17 green (incl. `smp`,
`affinity`, `process-kill`, and `args`/`init`/`process` on the exit-endpoint spawn path)
plus `aspace-refcount`/`thread-spawn`; `zig build` clean, `zig build test` green.

> **Note (deferred):** a detached thread's stack is freed only at process exit (not by the
> reaper on thread exit) — kernel user-stack tracking + reclaim is a later refinement. And
> the runtime heap is still not thread-safe: threads that both allocate concurrently would
> race (the thread *machinery* avoids the heap, but worker code sharing an allocator does
> not). Both fold into the M5 `Mutex`/allocator work.

## M4 — Futex: the one blocking primitive ✅

- [x] [abi.zig](../system/abi.zig): `futex_wait = 40`, `futex_wake = 41`. A waiter is a
      `.blocked` task tagged with `Task.futex_addr` (no queue linkage);
      `futex_wait(addr, expected, timeout_ns)` reads the user word under the big lock,
      parks iff `*addr == expected`, and returns on wake or timeout; `futex_wake(addr,
      count)` scans the task table and readies up to `count` matching waiters (same
      address space). No spinning — a parked waiter leaves its core free to `hlt`. A
      timed wait also sets `wake_at`, so the timer's `wakeExpired` wakes it; `futex_addr`
      staying non-zero (only `futex_wake` clears it) is how the waiter tells timeout from
      a real wake.
- [x] `runtime.Thread.Futex` (`wait` / `timedWait` / `wake`) over the syscall wrappers.
- [x] `-Dtest-case=thread-futex` (`smp: 4`): a waiter thread prints `waiting` and
      `futex_wait`s on a word; the main thread publishes it, prints `waking`, and
      `futex_wake`s; the waiter prints `woke`. Then a `timedWait` on an unwoken word
      reports `error.Timeout`.

**Gate (met):** `python3 test/qemu_test.py thread-futex` passes, robust across 3 runs —
the case's **ordered** regex asserts `waiting → waking → woke → PASS` on the serial
stream (the handoff proof), and `thread-futex: timeout ok` confirms the timeout.
Guardrail 18/18 green (incl. `sleep`/`event`/`ipc` blocking paths) + `aspace-refcount`,
`thread-spawn`, `thread-join`; `zig build` clean, `zig build test` green.

> **Note:** the kernel test checks only the freshest verdict marker via `bufferHas` (the
> in-memory log ring buffer evicts older lines); ordering is asserted against the full
> serial stream by the qemu regex instead.

## M5 — `Mutex` + `Condition` + `Semaphore` ✅

- [x] `runtime.Thread.Mutex` (three-state futex mutex: CAS fast path, `futex_wait`/`wake`
      slow path), `Condition` (`wait`/`timedWait`/`signal`/`broadcast`, a futex sequence
      counter), `Semaphore` (permits over `Mutex`+`Condition`) — the same state machines
      `std.Thread` uses, ported onto our `Futex`.
- [x] `-Dtest-case=thread-mutex` (`smp: 4`): a bounded producer/consumer — 2 producers +
      2 consumers over one `Mutex` and two `Condition`s move N=2000 unique items through
      an 8-slot ring; the consumed checksum and tally match exactly (no lost/duplicated
      item, no overrun) under real cross-core contention. The small ring forces producers
      to block on full and consumers on empty, exercising `Condition.wait`.

**Gate (met):** `python3 test/qemu_test.py thread-mutex` passes (`thread-mutex: ok` →
`DANOS-TEST-RESULT: PASS`), robust across 3 runs; guardrail 17/17 green (incl.
`sleep`/`event`/`ipc`) + all M1–M4 thread cases; `zig build` clean, `zig build test`
green.

> **Deferred (with rationale):**
> - **`join` → futex completion word** — the exit-endpoint join (M3) is correct and
>   tested. A futex-completion join needs the *kernel* to clear+wake a word after the
>   thread is fully off its stack (a CLONE_CHILD_CLEARTID-style mechanism); doing it in
>   the thread's own trampoline would let `join` `munmap` the stack while the thread still
>   runs on it (use-after-free). Left on the exit-endpoint path; the kernel clear-on-exit
>   is a later, separate refinement.
> - **Host unit tests for the state machines** — `Mutex`/`Condition` bottom out in the
>   `futex_*` syscalls, unavailable on the host without a mockable `Futex` seam. The QEMU
>   `thread-mutex` gate exercises them under real concurrency instead; a host-side mock is
>   future work.

## M6 — TLS, `getCurrentId` polish, docs, and CI wiring

- [ ] Per-thread TLS: `thread_spawn` sets `fs.base` to a runtime-allocated per-thread
      block; enough for `getCurrentId` and a `threadlocal` slot. Only expanded if a
      consumer needs full `threadlocal`.
- [ ] `RwLock` / `WaitGroup` if a consumer wants them (else deferred).
- [ ] All `thread-*` cases wired into [test/qemu_test.py](../test/qemu_test.py);
      threading.md status line updated to "built"; a short worked example in the doc.
- [ ] `-Dtest-case=thread-tls`: two threads read `getCurrentId` (distinct) and each
      writes its own `threadlocal` slot without cross-talk.

**Gate:** `python3 test/qemu_test.py thread-tls` logs `thread: per-thread id + tls ok`;
the whole `thread-*` suite plus the full guardrail set pass; default `zig build` is
clean and `zig build test` is green.

---

## Deferred (explicitly not in this plan)

- **Cross-process shared-memory futex** — the `(aspace, vaddr)` key can become a
  physical-address key so two processes share a futex through an [shm](display-v2.md)
  region. Not needed for intra-process threads.
- **Per-thread priorities / affinity distinct from the process** — threads inherit the
  process priority ([scheduling.md](scheduling.md)); revisit only if it earns its keep.
- **Per-thread signal delivery** — signals stay process-scoped
  ([process-lifecycle.md](process-lifecycle.md)).
- **A `pthread`/POSIX surface** — the API is `std.Thread`-shaped Zig, nothing more.
- **A real `std.Thread` backend** — arrives with self-hosting
  ([zig-self-hosting.md](zig-self-hosting.md)); it sits on these same primitives, so it
  swaps the impl under `runtime.Thread`, not the call sites.