danos/docs/os-development/threading.md

# Threading: `Thread`, a std-shaped API over a private thread ABI

A note on danos **threads** — several tasks sharing one address space — provided by a
`Thread` type that mirrors the shape of Zig's `std.Thread` while keeping every
kernel entry behind the [runtime](../../library/kernel). **Built** (M1–M11, see
[threading-plan.md](threading-plan.md)): `spawn`/`join`/`detach`, cross-core parallelism,
a futex, `Mutex`/`Condition`/`Semaphore`/`RwLock`/`WaitGroup`, `getCurrentId`/`currentCore`,
per-thread thread-pointer TLS, thread-safe allocation, and a task reaper that reclaims dead
tasks' kernel stacks. Deferred by design (no consumer yet): the Zig `threadlocal`
*compiler* layer (the per-thread thread pointer is in place, so it's runtime+linker work on top) and
detached-thread user-stack reclaim — see the plan's M9/M10 notes. The analysis is against
**Zig 0.16** (the pinned toolchain); `std.Thread`'s internals move between releases, so
treat upstream shapes as "0.16.x."

## The win condition

A danos service can write

```zig
const t = try Thread.spawn(.{}, worker, .{ctx});
// ... do other work concurrently ...
t.join();
```

and get real parallelism across cores — with `Thread.Mutex`,
`Thread.Condition`, and `Thread.Semaphore` available for
coordination — **without any code path reaching the kernel except through the
runtime**. The call sites read exactly like `std.Thread`, so the day danos becomes a
real Zig target (see [self-hosting](#the-self-hosting-endgame)) we swap the
implementation underneath, not the API above.

## Locked decisions (do not relitigate)

- **We build `Thread`, not literal `std.Thread`.** It mirrors std's *API and
  features*; the implementation underneath is danos-native. See
  [Why not literal std.Thread](#why-not-literal-stdthread).
- **Threads are a narrow, opt-in capability — not the default concurrency tool.** The
  default for resilience stays **process + IPC** ([resilience.md](resilience.md),
  [ipc.md](../device-driver-development/ipc.md)). See [Where threads fit](#where-threads-fit-the-resilience-tension).
- **Blocking synchronization is futex-backed, never spin-backed.** Waiters sleep in
  the kernel so an idle core still halts ([halting.md](halting.md)).
- **Per-binary opt-in to multi-threaded codegen.** Only a service that asks for
  threads is built `single_threaded = false`; the rest stay lean and single-threaded.
- **The thread ABI is private.** New syscalls extend [abi.zig](../../system/abi.zig)
  `SystemCall` and are reached only through `library/kernel` wrappers, exactly like
  every other danos syscall ([syscall.md](syscall.md)) — numbers stay renumberable.

## Why not literal `std.Thread`

danos's ABI invariant is that the **runtime is the sole holder of the syscall ABI**,
and that ABI is private and renumberable ([syscall.md](syscall.md) — "unstable
private ABI"). That is a security and evolvability asset: no compiled binary can
hardcode a syscall number, and the kernel can renumber freely because only the
runtime — rebuilt in lockstep — knows the mapping.

`std.Thread` is incompatible with that invariant on two counts:

1. **It selects its backend from `builtin.os.tag`, and issues syscalls directly.**
   danos targets `.os_tag = .freestanding` ([build.zig](../../build.zig)), for which
   `std.Thread` resolves to an unsupported stub that `@compileError`s. Adding a real
   backend would either bake danos syscall numbers into std (breaking ABI privacy and
   renumbering) or fork std to route back through the runtime — a permanent rebase
   cost that buys nothing the native type doesn't.
2. **Our user binaries are built `single_threaded = true`** (the shared recipe in
   [build-support/build.zig](../../build-support/build.zig)), which compiles threading
   out entirely and makes atomics and TLS single-threaded. Threads need this flipped
   per binary regardless.

So we take the *shape* of `std.Thread`, not the *type*. The cost of replicating the
surface (spawn/join/Mutex/Condition) is small; the cost of the std type is the ABI
invariant.

## Where threads fit: the resilience tension

Threads are in genuine tension with a resilience-first microkernel, and it is worth
being explicit so we do not reach for them by reflex.

The reason danos pays for a microkernel is **fault isolation**
([resilience.md](resilience.md)): a component corrupts its own address space, faults,
and is **restarted** without touching anyone else — because the boundary *is* the
address space. Threads deliberately remove that boundary *within* a process:

- Threads share one address space, so one thread's stray write corrupts them all —
  there is no isolation **between** threads.
- Threads share fate — by contract: a fault in any thread, or a "kill the process"
  decision, takes down **all** of them, so restartability lives at the process level,
  not the thread level. (The kernel does not yet enforce this fan-out — see the
  Lifecycle note under
  [Interaction with the rest of the kernel](#interaction-with-the-rest-of-the-kernel).)
- Shared mutable state reintroduces data races — the failure class the
  isolate-and-message model was chosen to avoid.

**Therefore:** the default answer to "make X concurrent" stays *another process over
IPC* (isolated, independently restartable) or a single event loop with several
message sources. Reach for a thread only inside **one** service that needs genuine
**shared-memory, low-latency parallelism** and can accept intra-service fate-sharing —
e.g. a compositor splitting tile compositing across cores, where per-tile IPC would be
too chatty. "Input on one thread, display on another" is *not* that case; it wants two
processes. The isolation boundary stays at process granularity.

## The API surface (mirrors `std.Thread`)

Lives in `library/kernel/thread.zig`, re-exported as `Thread`.

```zig
pub const Thread = struct {
    pub const Id = u32;                       // the kernel task id
    pub const SpawnConfig = struct {
        stack_size: usize = default_stack_size, // no allocator: the closure lives at the top of the thread's own stack
    };
    pub const SpawnError = error{SystemResources};

    pub fn spawn(config: SpawnConfig, comptime function: anytype, args: anytype) SpawnError!Thread;
    pub fn join(self: Thread) void;           // block until the thread ends, reclaim its stack
    pub fn detach(self: Thread) void;         // give up the right to join; stack reclaimed at process exit
    pub fn getCurrentId() Id;
    pub fn currentCore() Id;                   // danos extension: the calling core's dense index

    pub const Mutex     = struct { pub fn lock(*Mutex) void; pub fn tryLock(*Mutex) bool; pub fn unlock(*Mutex) void; };
    pub const Condition = struct { pub fn wait(*Condition, *Mutex) void; pub fn timedWait(*Condition, *Mutex, u64) error{Timeout}!void; pub fn signal(*Condition) void; pub fn broadcast(*Condition) void; };
    pub const Semaphore = struct { pub fn wait(*Semaphore) void; pub fn post(*Semaphore) void; };
    pub const Futex     = struct { pub fn wait(*const atomic.Value(u32), u32) void; pub fn timedWait(...) error{Timeout}!void; pub fn wake(*const atomic.Value(u32), u32) void; };
    // RwLock / ResetEvent / WaitGroup follow the same pattern, added as needed.
};
```

Deviations from `std.Thread`, called out honestly:

- **The thread function's return value is discarded** (as `std.Thread.join` returns
  `void`). Return data through shared state or a `Semaphore`/`Condition`, not the
  return.
- No `getCpuCount()` (a service rarely needs it) and no `Thread.yield()` — `yield`
  lives in the `process` module. Instead `currentCore()` exposes the calling core's dense
  index ([smp.md](smp.md)), used to observe genuine cross-core parallelism.

## Kernel primitives (new private syscalls)

Five core entries extend [abi.zig](../../system/abi.zig) `SystemCall` after
`shared_memory_physical = 36` (plus small helpers `current_core`, `thread_self`, and
`set_thread_pointer`), each with a `library/kernel` wrapper:

| Syscall | Signature | Purpose |
|---|---|---|
| `thread_spawn` | `(entry, stack_top, arg, exit_endpoint) -> tid` | create a task sharing the **caller's** address space; the runtime passes `no_cap` for `exit_endpoint` (join is a syscall, not an endpoint) |
| `thread_exit`  | `()` | end the calling thread; its stack is reclaimed by the joiner's `munmap`, not the kernel |
| `thread_join`  | `(tid) -> 0` | block until the thread with id `tid` has exited |
| `futex_wait`   | `(addr, expected, timeout_ns) -> status` | block if `*addr == expected`, until woken or timeout |
| `futex_wake`   | `(addr, count) -> woken` | wake up to `count` waiters on `addr` |

Plus one invariant change with no new syscall: **address-space reference counting**.

## Mechanics

### Address-space reference counting

Before this work an address space was 1:1 with a task: `spawnUserLocked` records
`address_space` on the Task (as it still does), and teardown did
`destroyAddressSpace(t.address_space)` when **any** user task exited
([scheduler.zig](../../system/kernel/scheduler.zig)). With threads, several tasks share
one `address_space`, so the first to exit would rip the address space out from under its
siblings.

Fix: a small refcount keyed by the address-space root, kept in
[scheduler.zig](../../system/kernel/scheduler.zig): `retainAddressSpace` takes a
reference for every user task `spawnUserLocked` starts (count 1 on the first take, so
a thread sharing the caller's space increments it); task teardown calls
`releaseAddressSpace`, which only calls `destroyAddressSpace` at **zero**. All of
this is already under the big kernel lock, so no new locking. This is the one piece
that must land and be proven before anything shares an address space.

### `thread_spawn` and the trampoline

The scheduler already accepts an arbitrary `address_space` and does **not** smuggle
values through scratch registers — `startUserTask` reads the entry/stack (and the
thread's closure arg, delivered in `rdi` via `jumpToUserArg`) from the Task
([scheduler.zig](../../system/kernel/scheduler.zig)). That makes the thread path clean:

1. The runtime's `spawn` `mmap`s a stack (syscall `4`) and writes the closure —
   `{ tls_base, args }`, the std "Instance" pattern — at the **top of the new stack
   itself** (no heap allocation), with a small per-thread TLS block just below it.
2. It calls `thread_spawn(entry = &Closure.entry, stack_top, arg = closure_ptr,
   exit_endpoint = no_cap)`. The kernel calls the same `spawnUserLocked` path with the
   **caller's address space** (refcount++), `entry`, and `user_sp = stack_top`.
3. `Closure.entry` (a small runtime shim) receives the closure pointer in `rdi` — the
   kernel delivers `arg` as the entry's first C-ABI argument — sets the thread
   pointer, calls the user function, then calls `thread_exit`.

Unlike a process start, there is **no** System V argc/argv/auxv block
([sysv.md](sysv.md)) — a thread stack carries only the closure and its TLS block.

### Lifetime: exit, join, detach, stack reclaim

- **`thread_exit`** (no arguments) marks the task dead. The kernel releases the
  task's resources, decrements the address-space refcount, and frees the task slot —
  the user stack is not the kernel's to unmap; the joiner reclaims it.
- **`join`** is a dedicated `thread_join(tid)` syscall: the caller blocks in the
  kernel (`joinThreadLocked`, woken by `wakeJoinersLocked` when the thread exits),
  then `munmap`s the stack. (The plan staged join over a per-thread `exit_endpoint`
  first, with a futex `completion` word as a Stage-2 refinement; neither shipped — the
  dedicated syscall replaced both. `thread_spawn` still accepts an `exit_endpoint`
  argument, which the runtime passes as `no_cap`.)
- **`detach`** relinquishes the join right: no one waits for the thread, and its
  stack is reclaimed at process exit — kernel-side reclaim of a detached thread's
  user stack stays deferred (as the intro notes), since `thread_exit` passes no stack
  range.

### Futex, and the sync primitives on top

`futex_wait`/`futex_wake` are the one blocking primitive; `Mutex`, `Condition`, and
`Semaphore` are ordinary user-space state machines over an `atomic.Value(u32)` that
call the futex wrappers on the slow path — the same construction `std.Thread` uses,
so the algorithms port directly.

Keying: threads share an address space, so a **virtual address within that address space**
identifies a futex uniquely; the kernel keys its wait queue by `(address_space_root, virtual_address)`.
Keying by the **physical** address instead (translate `virtual_address -> physical_address` on entry) is a
deliberate forward door: it lets two *processes* share a futex through an
[shared-memory](../device-driver-development/display-v2.md) region later, without changing the API. We start with the
private-per-address-space key and note the physical-key upgrade.

No spinning: a contended lock parks the task in the kernel and the core is free to run
other work or `hlt` ([halting.md](halting.md)). This is why futex is a locked
decision, not a "maybe later."

### TLS and `getCurrentId`

Per-thread thread-pointer TLS is in place (the `threadlocal` *compiler* layer is not —
see the intro). Two scoped pieces, as built:

- **`getCurrentId`** returns the kernel task id via the trivial `thread_self` syscall.
- **The thread pointer** is per-thread: `spawn` carves a small TLS block (an `fs:0`
  self-pointer plus scratch) from the top of the thread's own stack, the trampoline
  calls `set_thread_pointer` before any user code runs, and the scheduler saves and
  restores the pointer per task across context switches. Full `threadlocal` support is
  runtime+linker work on top of this, only if a consumer needs it. Nothing in the core
  spawn/join/mutex path requires `threadlocal`.

### Build: multi-threaded codegen, opt-in

A binary opts in with `.threaded = true` in its package's
`build_support.userBinary` call — the shared recipe in build-support then builds it
`single_threaded = false` — so atomics
and (later) TLS are real. Threads and atomics are unsound in a `single_threaded` image,
so a binary must opt in **before** it may call `Thread.spawn`. Everyone else
stays single-threaded and lean.

## Interaction with the rest of the kernel

- **Scheduler / SMP** ([scheduling.md](scheduling.md), [smp.md](smp.md)): a thread is
  just another `Task` with an `address_space` shared with its siblings; the existing
  per-core ready queues, priorities, and affinity apply unchanged. Threads of one
  process can run on different cores simultaneously — that is the point.
- **Halting** ([halting.md](halting.md)): futex-parked waiters keep the "idle core
  halts" property intact under lock contention — no busy-wait.
- **Lifecycle** ([process-lifecycle.md](process-lifecycle.md)): the contract is that
  killing a process kills *all* its threads and only then drops the last address-space
  ref — and the kernel now implements exactly that
  ([shared-fate-plan.md](shared-fate-plan.md)): every death path (`exit` from any
  thread, a fault, `process_kill` aimed at any member id) fans out through the whole
  group via a `dying` latch on the address space; the supervisor's one exit
  notification — badged with the leader — fires only when the last member is gone.
  A worker's voluntary `thread_exit` stays per-thread; the leader's is refused
  (`-EPERM`).
- **Resilience** ([resilience.md](resilience.md)): by the same contract, a faulting
  thread kills its whole process (shared fate); the supervisor restarts the
  **process**, which respawns its threads from a known-good state — restart
  granularity stays the process. The leader's recorded exit reason carries the fault
  class even when a worker faulted, so restart policy is unchanged.
- **IPC — two consequences threads forced ([ipc.md](../device-driver-development/ipc.md)):**
  - *Handles do not cross threads.* The handle table lives on the `Task`
    ([scheduler.zig](../../system/kernel/scheduler.zig)), so a handle number is meaningful
    only to the thread that created it — thread A's endpoint handle `3` is not thread B's.
    A thread that needs to reach an endpoint another thread owns opens the name
    (`channel.openEndpoint("display")`) to install its **own** handle to the same
    underlying endpoint — an ordinary client open, with no special mechanism for the
    fact that the provider happens to be this process. This is how the display's
    mouse-listener thread reaches the compositor loop's endpoint to poke it awake
    (docs/display.md).
  - *IPC syscalls that touch shared kernel state now serialize under the big kernel lock.*
    `create_ipc_endpoint` allocates from the kernel heap and
    mutates endpoint refcounts and handle tables. Those paths
    were unlocked because a single-threaded process could not race itself; a multi-threaded
    one can, from two cores at once. They now take `sync.enter()` like `call`/`reply_wait`/
    `send` already did — the kernel heap has no lock of its own (heap.zig: "every kernel
    entry takes the big kernel lock"), so the big lock is what keeps its callers serialized.

## Build-out plan (staged, each gate serial-checkable)

The ordered, `/loop`-runnable milestones live in
**[threading-plan.md](threading-plan.md)** (shaped like
[display-v2-plan.md](../device-driver-development/display-v2-plan.md)): every milestone lands on its own and ends in
a verifiable gate (`python3 test/qemu_test.py <case>`, asserting serial markers;
`zig build test` for host unit tests). The stages below are the shape it expands.

- **Stage 0 — address-space refcount.** Refcount on the address-space root; teardown destroys
  at zero. No API yet; nothing shares an address space, so refcount is 1 everywhere.
  *Gate:* the full QEMU suite stays green (no regression) — proves the reframing is
  invisible until used.
- **Stage 1 — spawn / join / detach.** `thread_spawn` + `thread_exit`, the trampoline,
  stacks via `mmap`, join over the exit-endpoint (as built, join became the dedicated
  `thread_join` syscall instead), the `addThreadedUserBinary` build opt-in.
  *Gate:* two cases as built — `-Dtest-case=thread-spawn`, where a worker thread runs
  in the caller's address space (a shared-memory write, observed by the main thread),
  and `-Dtest-case=thread-join`, where N workers each atomically increment a shared
  counter K times, the parent joins all N and asserts the total is exactly N × K —
  the join case running multi-core (`smp` 4) to prove real parallelism.
- **Stage 2 — blocking synchronization.** `futex_wait`/`futex_wake` + `Futex`,
  `Mutex`, `Condition`, `Semaphore`; optionally migrate join to a futex completion
  word. *Gate:* `-Dtest-case=thread-mutex` — a bounded producer/consumer over a
  `Mutex` + `Condition` moves K items with no lost wakeups and no busy-wait (assert
  the consumer blocked, e.g. via a low idle tick count).
- **Stage 3 — polish.** Per-thread TLS / thread pointer and `threadlocal` (only if a
  consumer needs it), `RwLock`/`WaitGroup` as demanded, and this doc's cases wired
  into [test/qemu_test.py](../../test/qemu_test.py).

## Conventions

Follow [coding-standards.md](../coding-standards.md): spell out non-acronym
abbreviations, kebab-case file names, no `Co-Authored-By` trailers. New syscalls
extend [abi.zig](../../system/abi.zig) `SystemCall` + a `library/kernel` wrapper
([syscall.md](syscall.md)). `Thread` is a first-class runtime module, the same
way `process` ([process-lifecycle.md](process-lifecycle.md)) and `ipc`
are — user code never names a syscall.

## Non-goals

- **No preemptive user-space signals delivered to a specific thread.** Signals stay
  process-scoped ([process-lifecycle.md](process-lifecycle.md)).
- **No thread priorities distinct from the process.** Threads inherit the process
  priority; per-thread priority is a later question if it ever earns its keep.
- **No cross-process shared-memory futex yet** — the physical-address key leaves the
  door open, but the first cut is private-per-address-space.
- **No `pthread`/POSIX surface.** The API is `std.Thread`-shaped Zig, nothing more.

## The self-hosting endgame

When danos becomes a real Zig target and we (eventually) add a danos backend to std
([zig-self-hosting.md](../zig-self-hosting.md)), `std.Thread` can sit *on top of* these
same kernel primitives — the danos `std.Thread.Impl` would call the very
`thread_spawn`/`futex_*` wrappers `Thread` already uses. Because
`Thread` was built API-compatible from day one, that transition swaps the
implementation, not a single call site. Designing to the std shape now is what makes
the later self-hosting lift cheap.

## Further reading

- [scheduling.md](scheduling.md), [smp.md](smp.md) — the task model these threads join.
- [resilience.md](resilience.md), [vision.md](../vision.md) — why isolation is the default
  and threads are the exception.
- [syscall.md](syscall.md), [ipc.md](../device-driver-development/ipc.md) — the private ABI and the messaging model
  threads sit beside.
- [halting.md](halting.md) — the idle/halt property futex-backed blocking preserves.
- [zig-self-hosting.md](../zig-self-hosting.md) — the target this bends toward.