danos/docs/os-development/threading.md

Threading: Thread, a std-shaped API over a private thread ABI

A note on danos threads — several tasks sharing one address space — provided by a Thread type that mirrors the shape of Zig's std.Thread while keeping every kernel entry behind the runtime. Built (M1–M11, see threading-plan.md): spawn/join/detach, cross-core parallelism, a futex, Mutex/Condition/Semaphore/RwLock/WaitGroup, getCurrentId/currentCore, per-thread thread-pointer TLS, thread-safe allocation, and a task reaper that reclaims dead tasks' kernel stacks. Deferred by design (no consumer yet): the Zig threadlocal compiler layer (the per-thread thread pointer is in place, so it's runtime+linker work on top) and detached-thread user-stack reclaim — see the plan's M9/M10 notes. The analysis is against Zig 0.16 (the pinned toolchain); std.Thread's internals move between releases, so treat upstream shapes as "0.16.x."

The win condition

A danos service can write

const t = try Thread.spawn(.{}, worker, .{ctx});
// ... do other work concurrently ...
t.join();

and get real parallelism across cores — with Thread.Mutex, Thread.Condition, and Thread.Semaphore available for coordination — without any code path reaching the kernel except through the runtime. The call sites read exactly like std.Thread, so the day danos becomes a real Zig target (see self-hosting) we swap the implementation underneath, not the API above.

Locked decisions (do not relitigate)

• We build Thread, not literal std.Thread. It mirrors std's API and features; the implementation underneath is danos-native. See Why not literal std.Thread.
• Threads are a narrow, opt-in capability — not the default concurrency tool. The default for resilience stays process + IPC (resilience.md, ipc.md). See Where threads fit.
• Blocking synchronization is futex-backed, never spin-backed. Waiters sleep in the kernel so an idle core still halts (halting.md).
• Per-binary opt-in to multi-threaded codegen. Only a service that asks for threads is built single_threaded = false; the rest stay lean and single-threaded.
• The thread ABI is private. New syscalls extend abi.zig SystemCall and are reached only through library/kernel wrappers, exactly like every other danos syscall (syscall.md) — numbers stay renumberable.

Why not literal std.Thread

danos's ABI invariant is that the runtime is the sole holder of the syscall ABI, and that ABI is private and renumberable (syscall.md — "unstable private ABI"). That is a security and evolvability asset: no compiled binary can hardcode a syscall number, and the kernel can renumber freely because only the runtime — rebuilt in lockstep — knows the mapping.

std.Thread is incompatible with that invariant on two counts:

1. It selects its backend from builtin.os.tag, and issues syscalls directly. danos targets .os_tag = .freestanding (build.zig), for which std.Thread resolves to an unsupported stub that @compileErrors. Adding a real backend would either bake danos syscall numbers into std (breaking ABI privacy and renumbering) or fork std to route back through the runtime — a permanent rebase cost that buys nothing the native type doesn't.
2. Our user binaries are built single_threaded = true (the shared recipe in build-support/build.zig), which compiles threading out entirely and makes atomics and TLS single-threaded. Threads need this flipped per binary regardless.

So we take the shape of std.Thread, not the type. The cost of replicating the surface (spawn/join/Mutex/Condition) is small; the cost of the std type is the ABI invariant.

Where threads fit: the resilience tension

Threads are in genuine tension with a resilience-first microkernel, and it is worth being explicit so we do not reach for them by reflex.

The reason danos pays for a microkernel is fault isolation (resilience.md): a component corrupts its own address space, faults, and is restarted without touching anyone else — because the boundary is the address space. Threads deliberately remove that boundary within a process:

• Threads share one address space, so one thread's stray write corrupts them all — there is no isolation between threads.
• Threads share fate — by contract: a fault in any thread, or a "kill the process" decision, takes down all of them, so restartability lives at the process level, not the thread level. (The kernel does not yet enforce this fan-out — see the Lifecycle note under Interaction with the rest of the kernel.)
• Shared mutable state reintroduces data races — the failure class the isolate-and-message model was chosen to avoid.

Therefore: the default answer to "make X concurrent" stays another process over IPC (isolated, independently restartable) or a single event loop with several message sources. Reach for a thread only inside one service that needs genuine shared-memory, low-latency parallelism and can accept intra-service fate-sharing — e.g. a compositor splitting tile compositing across cores, where per-tile IPC would be too chatty. "Input on one thread, display on another" is not that case; it wants two processes. The isolation boundary stays at process granularity.

The API surface (mirrors std.Thread)

Lives in library/kernel/thread.zig, re-exported as Thread.

pub const Thread = struct {
    pub const Id = u32;                       // the kernel task id
    pub const SpawnConfig = struct {
        stack_size: usize = default_stack_size, // no allocator: the closure lives at the top of the thread's own stack
    };
    pub const SpawnError = error{SystemResources};

    pub fn spawn(config: SpawnConfig, comptime function: anytype, args: anytype) SpawnError!Thread;
    pub fn join(self: Thread) void;           // block until the thread ends, reclaim its stack
    pub fn detach(self: Thread) void;         // give up the right to join; stack reclaimed at process exit
    pub fn getCurrentId() Id;
    pub fn currentCore() Id;                   // danos extension: the calling core's dense index

    pub const Mutex     = struct { pub fn lock(*Mutex) void; pub fn tryLock(*Mutex) bool; pub fn unlock(*Mutex) void; };
    pub const Condition = struct { pub fn wait(*Condition, *Mutex) void; pub fn timedWait(*Condition, *Mutex, u64) error{Timeout}!void; pub fn signal(*Condition) void; pub fn broadcast(*Condition) void; };
    pub const Semaphore = struct { pub fn wait(*Semaphore) void; pub fn post(*Semaphore) void; };
    pub const Futex     = struct { pub fn wait(*const atomic.Value(u32), u32) void; pub fn timedWait(...) error{Timeout}!void; pub fn wake(*const atomic.Value(u32), u32) void; };
    // RwLock / ResetEvent / WaitGroup follow the same pattern, added as needed.
};

Deviations from std.Thread, called out honestly:

• The thread function's return value is discarded (as std.Thread.join returns void). Return data through shared state or a Semaphore/Condition, not the return.
• No getCpuCount() (a service rarely needs it) and no Thread.yield() — yield lives in the process module. Instead currentCore() exposes the calling core's dense index (smp.md), used to observe genuine cross-core parallelism.

Kernel primitives (new private syscalls)

Five core entries extend abi.zig SystemCall after shared_memory_physical = 36 (plus small helpers current_core, thread_self, and set_thread_pointer), each with a library/kernel wrapper:

Syscall	Signature	Purpose
thread_spawn	(entry, stack_top, arg, exit_endpoint) -> tid	create a task sharing the caller's address space; the runtime passes no_cap for exit_endpoint (join is a syscall, not an endpoint)
thread_exit	()	end the calling thread; its stack is reclaimed by the joiner's munmap, not the kernel
thread_join	(tid) -> 0	block until the thread with id tid has exited
futex_wait	(addr, expected, timeout_ns) -> status	block if *addr == expected, until woken or timeout
futex_wake	(addr, count) -> woken	wake up to count waiters on addr

Plus one invariant change with no new syscall: address-space reference counting.

Mechanics

Address-space reference counting

Before this work an address space was 1:1 with a task: spawnUserLocked records address_space on the Task (as it still does), and teardown did destroyAddressSpace(t.address_space) when any user task exited (scheduler.zig). With threads, several tasks share one address_space, so the first to exit would rip the address space out from under its siblings.

Fix: a small refcount keyed by the address-space root, kept in scheduler.zig: retainAddressSpace takes a reference for every user task spawnUserLocked starts (count 1 on the first take, so a thread sharing the caller's space increments it); task teardown calls releaseAddressSpace, which only calls destroyAddressSpace at zero. All of this is already under the big kernel lock, so no new locking. This is the one piece that must land and be proven before anything shares an address space.

thread_spawn and the trampoline

The scheduler already accepts an arbitrary address_space and does not smuggle values through scratch registers — startUserTask reads the entry/stack (and the thread's closure arg, delivered in rdi via jumpToUserArg) from the Task (scheduler.zig). That makes the thread path clean:

1. The runtime's spawn mmaps a stack (syscall 4) and writes the closure — { tls_base, args }, the std "Instance" pattern — at the top of the new stack itself (no heap allocation), with a small per-thread TLS block just below it.
2. It calls thread_spawn(entry = &Closure.entry, stack_top, arg = closure_ptr, exit_endpoint = no_cap). The kernel calls the same spawnUserLocked path with the caller's address space (refcount++), entry, and user_sp = stack_top.
3. Closure.entry (a small runtime shim) receives the closure pointer in rdi — the kernel delivers arg as the entry's first C-ABI argument — sets the thread pointer, calls the user function, then calls thread_exit.

Unlike a process start, there is no System V argc/argv/auxv block (sysv.md) — a thread stack carries only the closure and its TLS block.

Lifetime: exit, join, detach, stack reclaim

• thread_exit (no arguments) marks the task dead. The kernel releases the task's resources, decrements the address-space refcount, and frees the task slot — the user stack is not the kernel's to unmap; the joiner reclaims it.
• join is a dedicated thread_join(tid) syscall: the caller blocks in the kernel (joinThreadLocked, woken by wakeJoinersLocked when the thread exits), then munmaps the stack. (The plan staged join over a per-thread exit_endpoint first, with a futex completion word as a Stage-2 refinement; neither shipped — the dedicated syscall replaced both. thread_spawn still accepts an exit_endpoint argument, which the runtime passes as no_cap.)
• detach relinquishes the join right: no one waits for the thread, and its stack is reclaimed at process exit — kernel-side reclaim of a detached thread's user stack stays deferred (as the intro notes), since thread_exit passes no stack range.

Futex, and the sync primitives on top

futex_wait/futex_wake are the one blocking primitive; Mutex, Condition, and Semaphore are ordinary user-space state machines over an atomic.Value(u32) that call the futex wrappers on the slow path — the same construction std.Thread uses, so the algorithms port directly.

Keying: threads share an address space, so a virtual address within that address space identifies a futex uniquely; the kernel keys its wait queue by (address_space_root, virtual_address). Keying by the physical address instead (translate virtual_address -> physical_address on entry) is a deliberate forward door: it lets two processes share a futex through an shared-memory region later, without changing the API. We start with the private-per-address-space key and note the physical-key upgrade.

No spinning: a contended lock parks the task in the kernel and the core is free to run other work or hlt (halting.md). This is why futex is a locked decision, not a "maybe later."

TLS and getCurrentId

Per-thread thread-pointer TLS is in place (the threadlocal compiler layer is not — see the intro). Two scoped pieces, as built:

• getCurrentId returns the kernel task id via the trivial thread_self syscall.
• The thread pointer is per-thread: spawn carves a small TLS block (an fs:0 self-pointer plus scratch) from the top of the thread's own stack, the trampoline calls set_thread_pointer before any user code runs, and the scheduler saves and restores the pointer per task across context switches. Full threadlocal support is runtime+linker work on top of this, only if a consumer needs it. Nothing in the core spawn/join/mutex path requires threadlocal.

Build: multi-threaded codegen, opt-in

A binary opts in with .threaded = true in its package's build_support.userBinary call — the shared recipe in build-support then builds it single_threaded = false — so atomics and (later) TLS are real. Threads and atomics are unsound in a single_threaded image, so a binary must opt in before it may call Thread.spawn. Everyone else stays single-threaded and lean.

Interaction with the rest of the kernel

• Scheduler / SMP (scheduling.md, smp.md): a thread is just another Task with an address_space shared with its siblings; the existing per-core ready queues, priorities, and affinity apply unchanged. Threads of one process can run on different cores simultaneously — that is the point.
• Halting (halting.md): futex-parked waiters keep the "idle core halts" property intact under lock contention — no busy-wait.
• Lifecycle (process-lifecycle.md): the contract is that killing a process kills all its threads and only then drops the last address-space ref — and the kernel now implements exactly that (shared-fate-plan.md): every death path (exit from any thread, a fault, process_kill aimed at any member id) fans out through the whole group via a dying latch on the address space; the supervisor's one exit notification — badged with the leader — fires only when the last member is gone. A worker's voluntary thread_exit stays per-thread; the leader's is refused (-EPERM).
• Resilience (resilience.md): by the same contract, a faulting thread kills its whole process (shared fate); the supervisor restarts the process, which respawns its threads from a known-good state — restart granularity stays the process. The leader's recorded exit reason carries the fault class even when a worker faulted, so restart policy is unchanged.
• IPC — two consequences threads forced (ipc.md):
  • Handles do not cross threads. The handle table lives on the Task (scheduler.zig), so a handle number is meaningful only to the thread that created it — thread A's endpoint handle 3 is not thread B's. A thread that needs to reach an endpoint another thread owns looks it up (ipc.lookup(service)) to install its own handle to the same underlying endpoint. This is how the display's mouse-listener thread reaches the compositor loop's endpoint to poke it awake (docs/display.md).
  • IPC syscalls that touch shared kernel state now serialize under the big kernel lock. create_ipc_endpoint/ipc_register/ipc_lookup allocate from the kernel heap and mutate the global service registry, endpoint refcounts, and handle tables. Those paths were unlocked because a single-threaded process could not race itself; a multi-threaded one can, from two cores at once. They now take sync.enter() like call/reply_wait/ send already did — the kernel heap has no lock of its own (heap.zig: "every kernel entry takes the big kernel lock"), so the big lock is what keeps its callers serialized.

Build-out plan (staged, each gate serial-checkable)

The ordered, /loop-runnable milestones live in threading-plan.md (shaped like display-v2-plan.md): every milestone lands on its own and ends in a verifiable gate (python3 test/qemu_test.py <case>, asserting serial markers; zig build test for host unit tests). The stages below are the shape it expands.

• Stage 0 — address-space refcount. Refcount on the address-space root; teardown destroys at zero. No API yet; nothing shares an address space, so refcount is 1 everywhere. Gate: the full QEMU suite stays green (no regression) — proves the reframing is invisible until used.
• Stage 1 — spawn / join / detach. thread_spawn + thread_exit, the trampoline, stacks via mmap, join over the exit-endpoint (as built, join became the dedicated thread_join syscall instead), the addThreadedUserBinary build opt-in. Gate: two cases as built — -Dtest-case=thread-spawn, where a worker thread runs in the caller's address space (a shared-memory write, observed by the main thread), and -Dtest-case=thread-join, where N workers each atomically increment a shared counter K times, the parent joins all N and asserts the total is exactly N × K — the join case running multi-core (smp 4) to prove real parallelism.
• Stage 2 — blocking synchronization. futex_wait/futex_wake + Futex, Mutex, Condition, Semaphore; optionally migrate join to a futex completion word. Gate: -Dtest-case=thread-mutex — a bounded producer/consumer over a Mutex + Condition moves K items with no lost wakeups and no busy-wait (assert the consumer blocked, e.g. via a low idle tick count).
• Stage 3 — polish. Per-thread TLS / thread pointer and threadlocal (only if a consumer needs it), RwLock/WaitGroup as demanded, and this doc's cases wired into test/qemu_test.py.

Conventions

Follow coding-standards.md: spell out non-acronym abbreviations, kebab-case file names, no Co-Authored-By trailers. New syscalls extend abi.zig SystemCall + a library/kernel wrapper (syscall.md). Thread is a first-class runtime module, the same way process (process-lifecycle.md) and ipc are — user code never names a syscall.

Non-goals

• No preemptive user-space signals delivered to a specific thread. Signals stay process-scoped (process-lifecycle.md).
• No thread priorities distinct from the process. Threads inherit the process priority; per-thread priority is a later question if it ever earns its keep.
• No cross-process shared-memory futex yet — the physical-address key leaves the door open, but the first cut is private-per-address-space.
• No pthread/POSIX surface. The API is std.Thread-shaped Zig, nothing more.

The self-hosting endgame

When danos becomes a real Zig target and we (eventually) add a danos backend to std (zig-self-hosting.md), std.Thread can sit on top of these same kernel primitives — the danos std.Thread.Impl would call the very thread_spawn/futex_* wrappers Thread already uses. Because Thread was built API-compatible from day one, that transition swaps the implementation, not a single call site. Designing to the std shape now is what makes the later self-hosting lift cheap.

Further reading

• scheduling.md, smp.md — the task model these threads join.
• resilience.md, vision.md — why isolation is the default and threads are the exception.
• syscall.md, ipc.md — the private ABI and the messaging model threads sit beside.
• halting.md — the idle/halt property futex-backed blocking preserves.
• zig-self-hosting.md — the target this bends toward.