The display service claiming the framebuffer suppresses the on-screen
boot transcript — correctly in normal operation, but on a serial-less
machine being debugged, the timeline vanishes just when it matters. A
diagnose image (zig build -Ddiagnose=true) has init skip the display
service and demo: the timestamped transcript stays on screen
indefinitely, and the power button still runs the orderly shutdown (so
the logger's files land when storage works).
First use immediately caught a real bug IN QEMU: an intermittent (~1
in 3) usb-storage READ CAPACITY failure after a ~21 s stall — after
which usb-storage and fat both exit cleanly and nothing retries: one
transient early-boot USB failure leaves the system permanently without
storage (and therefore without logs). That no-retry policy is a prime
suspect for real hardware never mounting /var, and is Track B's first
work item.
Real-firmware finding: the per-file /system tree walk boots in seconds
under OVMF but stalls for MINUTES on real firmware — the cost is not
bytes (USB 3 moves the ~4 MB instantly) but firmware filesystem
OPERATIONS: ~30 opens, each an uncached directory-chain walk in an
unoptimized firmware FAT driver. This is why every real OS loader
(winload, GRUB) reads many files through its own filesystem code over
Block I/O rather than the firmware's file protocol.
The loader now reads boot/system.img — the bundled binaries packed into
ONE v2 initial_ramdisk (tools/pack-system-image.py, derived from the
same bundled list in the same build graph, so tree and capsule cannot
drift) — with a single open + sequential read, the one firmware file
I/O shape that is fast everywhere. The manifest (open each listed path
by name) and the tree walk remain as fallbacks, so a hand-assembled
stick without the capsule still boots. The running system is identical
in all three cases: the kernel receives the same in-RAM table.
The load phase now brackets itself with unconditional on-screen
breadcrumbs ('EFI: loading the system...' / '...starting the kernel'),
because this phase stalling behind a silent black screen — kernel
status is serial-only by design — already cost a real-hardware
debugging session.
Direction (settled with the user): this capsule becomes the BOOTSTRAP
capsule — kernel + init + the storage-bring-up set — once a
spawn-from-memory syscall lets init and the device manager load
everything else from the stick's real file tree at runtime through
danos's own storage stack: file-granular updates (rebuild one binary,
copy one file), the initramfs shape.
Per docs/coding-standards.md (no Unix-abbreviation exception): syscalls
shared_memory_create/map/physical, kernel SharedMemoryObject + handlers,
runtime.shared_memory (library/runtime/shared-memory.zig), the
shared-memory-server/-client test services, the shared-memory QEMU case,
and docs incl. vdso.md's danos_shared_memory_*. 87/87 QEMU tests pass.
runtime.fs now routes every path through fs_resolve: kernel-served
/system nodes are read via fs_node (tokens, no open state); everything
under a userspace mount goes straight to the owning backend's endpoint
with the kernel-rewritten mount-relative path — one syscall of naming,
then the unchanged vfs-protocol rendezvous, public API untouched. mkdir/
unlink/rename resolve-then-forward (rename checks both paths land on
the SAME backend); mount is the fs_mount syscall.
The fat server mounts twice — /mnt/usb from the volume root and /var
from its /var subtree — so the logger now writes the FHS path
/var/log/<boot-stamp>/... and swapping the persistent medium later
touches only fat's two mount calls. With clients holding fat's node ids
directly, fat records each handle's owner, checks it, and sweeps a dead
client's handles via the published exit events (the old router's
pattern, now where the state actually lives).
The userspace vfs server and its router die; ServiceId.vfs=1 stays
reserved-retired; protocol.zig moves to system/vfs-protocol.zig (the
wire contract is backend-only now). vfs-test becomes the ring-3 proof
of the kernel VFS (own-binary ELF magic through /system, read-only
refusals, listing); vfs-client-death becomes the fat sweep test over
the full storage chain, with a ring-scanning check (the last-write
buffer is too racy under a chattering tree).
The logger service drains the tagged ring every 250 ms and demultiplexes
it into one file per process on the flash volume:
/mnt/usb/var/log/2026-07-21T150434Z/system/services/fat.log
[ 0.214] mounted FAT (fat32, 128992 clusters, partition lba 0)
The boot stamp is the wall-clock anchor from klog_status (FAT-safe, no
colons); the kernel's records go to kernel.log; each line carries the
record's monotonic timestamp and level. Storage is best-effort and
late — the ring buffers a whole boot until the mount appears, then the
first drain writes the backlog, storage-stack records included. Lost
records surface as '-- N records lost --' from sequence gaps. Files
close (= fat's device cache flush) after a 2 s quiet period, bounding
data-at-risk without per-record flush thrash. The logger announces
itself once — a periodic status line would feed the stream it drains.
init spawns the logger last, so the reverse-order shutdown stops it
first and its final drain runs over a live storage chain; log-flush and
init's own DANOS.LOG shutdown flush retire (superseded). runtime.fs
makePath treats components at or above a mount point as router names —
create is best-effort per prefix, the final verdict is exists(path).
Replace the linear keep-earliest RAM buffer with a 512 KiB ring of
framed records (log-ring.zig, host-tested): every debug_write becomes
one record per payload line, stamped by the kernel with the sender's
pid, task name (its binary path), level, per-boot sequence number, and
monotonic timestamp. Attribution is structural — a payload cannot forge
another sender's tag, and newline injection lands inside the forger's
own next record. Oldest records are overwritten when full; sequence
gaps make the loss countable.
debug_write gains a level argument (err/warn/info/debug/raw; old
two-arg callers clamp to raw). klog_read becomes a stream-offset read
that fails once the cursor falls behind the ring's tail; the new
klog_status (#45) returns the cursors plus the boot wall-clock anchor —
what the logger service will name per-boot log directories with.
The log now guards itself with a dedicated spinlock (BKL -> log lock
order, never the reverse); panic paths use a bounded try-acquire and
fall back to sinks-only. Serial rendering keeps the historical
transcript byte-identical for kernel and legacy raw output; leveled
records get a kernel-rendered name prefix. log-flush/init's interim
drains start at the ring tail and write framed records until the logger
service replaces them.
Retire the build-time ramdisk packer and the packed initial-ramdisk.img.
make-fat-image.py now lays every user binary out at its FHS path on the
boot volume (system/services, system/drivers, system/tests), and the EFI
loader walks \system at boot, packing what it finds into an in-RAM v2
initial_ramdisk whose entry names are full FHS paths. init rides the
table like every other binary: its dedicated handoff fields are gone and
the kernel spawns PID 1 via the same lookup as everyone else
(process.spawnBundled).
system_spawn resolves names by exact path first, then unique basename,
and normalizes argv[0] to the stored path — so task names (and, next,
the tagged log ring's attribution) are honest binary paths everywhere.
initrd v2 rejects the old magic so a stale image fails loudly.
Groundwork for per-process logging (/var/log/<boot-stamp>/<binary-path>.log)
and the kernel-VFS /system mount.
The virtio-gpu present fence completes when the device has consumed the
frame: real completion feedback, and tear-freedom by snapshot semantics.
It is not a vblank — base virtio-gpu 2D has no display-refresh event at
all (Linux fakes one with a timer), so nothing paces presents to the
monitor. The code and docs claimed vsync anyway; now they don't.
- backend.hasVsync -> hasFencedPresent, with an honest doc comment
- marker 'display: vsync present ok' -> 'display: fenced present ok'
(display-modeset test expectation updated, passes)
- display-v2.md gains a 'Fenced is not vsync' note; the vsync claims in
both v2 docs are corrected
- true vsync arrives with a native driver's vblank IRQ, or approximated
by a compositor frame clock
Boots the usual VGA/GOP floor plus a virtio-gpu-pci adapter, so the
device-manager stack spawns the driver and the compositor upgrades to the
native fenced-present backend interactively. QEMU shows one head per
adapter: the live output is the virtio-gpu console in the View menu (the
VGA head freezes at the moment of upgrade). 512M, matching display-native.
The display's first use of threads (docs/threading.md, docs/display.md). The
compositor stays the single owner of the framebuffer — only the main
service.run loop touches the backend and layer stack — and a dedicated
mouse-listener thread runs beside it:
- Listener: blocks on input.subscribeMouse(), accumulates relative dx/dy
into an absolute cursor position clamped to the screen, and hands it to
the compositor. It never touches the compositor, so no lock guards the
framebuffer; a parked next() lets the core halt.
- CursorChannel: a single-slot latest-value cell under a Thread.Mutex (the
renderer wants where the cursor is now, not a replay of deltas), with a
coalesced self-ipc.send poke that wakes the main loop — parked in
replyWait — as a message-notification. At most one poke is queued while
the last is undrained, so a fast mouse can't flood the endpoint.
- Render: the cursor is a top-z compositor layer; on the poke the main loop
moves it via configure + present (which damages old + new footprints).
The display binary opts into threads (addThreadedUserBinary), and
input-source gains a "mouse" mode that publishes pure motion to drive it.
Two kernel-level findings this surfaced, both fixed:
1. IPC handles do not cross threads. The handle table lives on the Task, so
the listener can't reuse the main loop's endpoint handle — it
ipc.lookup(.display)s its own handle to the same endpoint to poke through.
2. Concurrent IPC from two threads raced unlocked kernel state. The display
is the first process issuing IPC syscalls from two threads at once, which
exposed a data race (flaky #GP in installEntry): create_ipc_endpoint /
ipc_register / ipc_lookup allocate from the kernel heap and mutate the
global registry, endpoint refcounts, and handle tables without the big
kernel lock. They were safe only while a process couldn't race itself.
They now sync.enter() like call/reply_wait/send already did (the kernel
heap has no lock of its own yet — heap.zig: "a lock comes with
threads/SMP" — so the big lock keeps its callers serialized).
Test: -Dtest-case=display-cursor (smp:4) spawns the input service, the
threaded display, and input-source in mouse mode; asserts the display's
"cursor tracking mouse ok" marker once the cursor has tracked a run of motion
end to end. Verified green 6/6 under stress (the race hit ~1-in-4 before the
lock fix) and in the full 29-case QEMU guardrail suite; zig build test clean.
runtime.Thread.RwLock (reader-preferring, lock/tryLock/unlock +
lockShared/tryLockShared/unlockShared) and WaitGroup (start/finish/wait), both on
the existing Mutex/Condition.
A compile-time Futex seam gated on builtin.os.tag: the futex syscalls on danos, a
spin+yield mock off-target (Zig 0.16 has no std.Thread.Futex; wake is a no-op
since the state machines re-check). thread.zig is wired into zig build test, so
Mutex/RwLock/WaitGroup run as host unit tests with real std.Thread threads (test
blocks compile only under test, so std.Thread there is fine on freestanding).
thread-rwlock QEMU case: 2 writers set both halves of a value under the exclusive
lock while 3 readers check they match under the shared lock; zero half-write
observations across ~150k reads.
Marks Phase 2 (M7-M11) built. threading.md/threading-plan.md status updated.
Gate: host zig build test covers the sync primitives; thread-rwlock PASS (3x);
full Done gate 26/26 (whole thread-* suite + guardrail); build clean.
A thread is a task sharing the caller's address space. New private syscalls
thread_spawn(entry, stack_top, arg)=37 and thread_exit=38: thread_spawn goes
through scheduler.spawnThread (retains the shared aspace), thread_exit ends the
task like a process exit(0) (terminateCurrent -> releaseAspace, so the space
survives while siblings hold it). The closure pointer reaches the new thread in
rdi via a new jump_to_user_arg asm path and a per-task user_arg (0 for a normal
process, whose _start ignores it) - so the runtime trampoline is a plain C-ABI
Zig function, no naked asm.
runtime.Thread (library/runtime/thread.zig) mirrors std.Thread.spawn: mmap a
stack, heap-allocate the args closure, hand the kernel the trampoline + closure.
addThreadedUserBinary opts a binary into single_threaded=false; thread-test is
the first, and proves a worker runs in the shared address space via a shared
global the main thread polls.
Gate thread-spawn PASS; 16 guardrail cases green (incl. args/init/process on the
new jump_to_user_arg path) + aspace-refcount; build + host tests clean.
The compositor now boots on the GOP framebuffer and upgrades to the virtio-gpu driver
the moment it announces itself — the pluggable-scanout payoff.
The shared surface. The scanout resource is an shm region the driver creates
(shm_physical, a new syscall, hands it the guest-physical for attach_backing) and passes
to the compositor as a capability. The compositor maps it and composites straight into
it: on x86 DMA is cache-coherent, so the cacheable shared pages the CPU paints are exactly
what the device transfers-and-flushes — no copy, no explicit flush.
The handshake. After bring-up the driver looks up .display and sends attach_scanout with
the geometry + the surface capability. The compositor maps the surface, looks up the
driver's .scanout endpoint itself (the driver registered it — no need to pass it), switches
to backend.VirtioGpu, and re-composites the current frame. present() over the native
backend is a present request on .scanout -> transfer-to-host + resource flush. The first
native present is deferred to a one-shot timer: presenting inline from the announce handler
would deadlock, since the driver is still blocked on our reply and not yet serving .scanout.
After it lands, the compositor reads a pixel back from the shared surface to confirm the
frame reached the device's backing.
- shm_physical (syscall 36) + runtime.shm.physical.
- scanout-protocol (the compositor->driver present channel), separate from the
client-facing display protocol; the display protocol gains attach_scanout.
- backend.VirtioGpu joins backend.Gop in the tagged union; select() still boots GOP.
- the virtio-gpu driver's scanout backing is now shm (was DMA); it announces + serves
.scanout present requests (transfer-to-host + flush of the shared surface).
Also fixes a latent framebuffer-geometry corruption the display service hit only when it
enumerated the device tree alongside a busy device-manager: Gop.init now captures the
geometry into a small value the instant device_enumerate returns (rather than re-reading
the 328-byte descriptor across the later claim/mmio_map syscalls) and retries on a zero
geometry. The underlying device-table clobber is a separate kernel bug, tracked apart.
Gate: python3 test/qemu_test.py display-native (QEMU -device virtio-gpu-pci) — "display:
scanout upgraded to virtio-gpu" + "display: native present verified" + "display-demo: ok",
passing 3/3. host tests, display-service, display-demo, shm, and virtio-gpu still pass.
The first native scanout backend's driver half. The device manager matches the
virtio-gpu PCI function by its Display/Other class triple and spawns the driver,
which claims the device, confirms vendor 0x1AF4/device 0x1050 from config space,
enables memory-space + bus mastering, and walks the virtio vendor capabilities to
find the common-config and notify structures in its BAR.
From there it is the standard modern-virtio bring-up: reset, negotiate VERSION_1,
stand up the control virtqueue in coherent DMA, then drive the GPU end to end —
RESOURCE_CREATE_2D, ATTACH_BACKING (a coherent DMA buffer; V4 swaps in the shm-
shared surface), SET_SCANOUT, paint a known pattern, TRANSFER_TO_HOST_2D,
RESOURCE_FLUSH, and wait for the device's used-ring ack. Reading the backing back
proves it is CPU-visible RAM; the ack proves the device consumed the frame —
together the automated stand-in for "it's on screen", no screenshot.
- system/drivers/virtio-gpu/: the driver, plus virtio-gpu-protocol.zig (control
commands) and virtio-pci.zig (the 1.0 PCI transport + split-virtqueue), both with
host-tested struct sizes.
- device-manager matches the display/other class triple to "virtio-gpu"; the driver
self-confirms the vendor/device id, since the class alone cannot distinguish it.
- ServiceId.scanout (11): the driver registers it so the compositor finds it in V4.
Gate: python3 test/qemu_test.py virtio-gpu (QEMU -device virtio-gpu-pci) — the
device-manager stack discovers the function, the driver brings up a 640x480 scanout
and flushes a test pattern: "virtio-gpu: scanout 640x480 online" + "flush acked,
pixel check ok". host tests, display-service, shm, and device-list still pass.
Note: the pre-existing pci-scan case triple-faults on main (verified at 88ad432,
before this change); it is unrelated and tracked separately.
Generalize capability passing from endpoints to memory objects. The per-task
handle table now holds kind-tagged entries (scheduler.HandleObject{kind, ptr});
closeHandles and shareCapability dispatch by kind, so a shared-memory object
rides an ipc_call send_cap exactly like an endpoint and is refcount-freed only
when its last capability drops.
- shm_create(len) -> vaddr, handle: contiguous, zeroed, cacheable frames wrapped
in a refcounted ShmObject, mapped into the caller's shm arena (PML4[230]).
- shm_map(cap) -> vaddr: map the same physical pages into a receiver that got the
capability. mapUserSharedInto maps WB-cacheable + device_grant, so a sharer's
teardown never frees the shared frames — the object owns them.
- runtime.shm: create(len) -> Region{ptr, handle, len}, map(handle) -> ptr.
Gate: qemu_test.py shm — shm-client creates a region, writes a pattern, passes
its capability to shm-server, which maps it and reads the same bytes back
(shm: shared 4096 bytes ok). ipc/ipc-call/ipc-cap/supervision/dma/usermem/
display-service and host tests all still pass — the handle change broke no IPC.
Drive the compositor from a separate process, proving the pipeline end to end.
- runtime.display: a Layer handle (fill / blitTile / configure / damage /
destroy), createLayer, and a color(r,g,b) helper that caches the mode and
packs via protocol.pack. Coordinates are signed over the u32 wire fields
(@bitCast both ways), so a layer may sit or move partly off-screen.
- system/services/display-demo: the input-source analog for the compositor — a
full-screen wallpaper, a rectangle it slides back and forth (moved by
configure each frame, so the damage-driven present repaints old + new), and a
cursor. Presents in a loop paced by runtime.time. Wired into build + initrd.
Fix this surfaced: protocol.message_maximum was 4096, but the kernel caps every
IPC message at MESSAGE_MAXIMUM = 256, so replyWait rejected the oversized receive
buffer with -E2BIG and the serve loop had been spinning since D2 (invisibly, as
those gates matched init-time heartbeats). Set it to 256; blit_tile is now
explicitly a small-tile path (larger bitmaps are the deferred shm surface).
Gate: `python3 test/qemu_test.py display-demo` — the demo drives frames of
motion through the layer client API and logs `display-demo: ok`. Regression:
zig build test, display (D1), display-service (D2/D3), and default zig build.
Turn the service into a real compositor. A layer is a server-owned surface (its
own mmap'd cacheable buffer) with a screen position, z-order, and visibility.
Clients create layers, draw into them by command, mark damage, and present; the
compositor repaints only the damaged region — clear to the wallpaper, paint the
visible layers bottom-to-top (z-sorted), flush that rectangle back -> front (WC).
- compositor.zig: the pure, host-tested core — Rect (intersect/unite), Surface,
fillRect, composite (opaque, clipped to a damage rect), blitTile (unaligned-
safe read of a client tile). No syscall/runtime dependency.
- protocol.zig: pack(format, r, g, b) — native pixel encoding for rgbx/bgrx, the
shared colour vocabulary of client and server. Host-tested.
- display.zig: the layer table + create/configure/destroy/fill_rect/blit_tile/
damage/present ops wired onto the compositor, plus a damage-accumulating present.
- Startup self-check: two overlapping layers composited on the real framebuffer,
read back to confirm the overlap shows the top layer and outside shows the
bottom — logs `display: compositor self-check ok`.
Gate: `zig build test` green (compositor + pack), and the display-service case's
self-check passes on hardware. Both new pure modules added to the test loop.
Stand up /system/services/display: a ring-3 process that claims the framebuffer
D1 seeded, maps it write-combining as the front buffer, allocates a cacheable
back buffer, and presents composed frames. The GUI track's compositor, reached
by name over ServiceId.display (= 9).
- protocol.zig: the display wire protocol (info/create_layer/configure_layer/
destroy_layer/fill_rect/blit_tile/damage/present). `info` and a whole-screen
`present` are live; the layer ops fail-stub until D3.
- display.zig: enumerate -> claim -> mmio_map(WC) the LFB, mmap a cacheable back
buffer, clear it and present it (proving the double-buffer path), then serve.
- runtime.display + barrel exports (display, display_protocol): a cached
`.display` client with info()/present(), the runtime.block shape.
- init spawns "display" in boot_services; build.zig wires the protocol onto the
runtime, builds the exe, packs it into the initial-ramdisk, installs it.
mmap fix the back buffer forced: systemMmap was capped at 256 pages (1 MiB) by a
fixed kernel-stack scratch array. Rewrote it to map page-by-page with rollback
(no array) and raised the cap to 8192 pages (32 MiB) — enough for a 4K back
buffer. A real limitation met.
Gate: `python3 test/qemu_test.py display-service` matches the service's own
serial heartbeats (display: online WxH / presented frame 0), printed only after
the full claim -> WC-map -> back-buffer -> present chain. Regression-checked
usermem, heap, init, and D1's display.
Serial is now a QEMU/dev aid, not a real-hardware necessity: a legacy-free board often has no live COM1, and the boot log is kept in RAM (klog) and flushed to disk. So the serial sink is compiled in only under -Dserial (default false).
- kernel.zig gates serialInit + the log sink on build_options.serial
- boot/efi.zig gates its EFI: progress breadcrumbs (con_out) via progress(); fatal-error messages stay always-on so a failed boot still explains itself
- run-x86-64 boots a serial-enabled image variant (factored addKernel/addBootImage helpers) so a dev boot always captures serial0, without baking serial into the flashable image
- test/qemu_test.py builds -Dserial=true (it asserts on serial markers)
- the loopback probe stays as a real-HW safety net for -Dserial images
The first step of the Zig self-hosting roadmap (docs/zig-self-hosting.md): give
danos programs a danos-native file API and remove the premature POSIX compatibility
shim. This also resolves the earlier misplacement of a full-write helper into the
compat layer — that behaviour now lives natively in runtime.fs.File.writeAll.
- library/runtime/fs.zig: the danos-native file client over the VFS (open/read/
write/writeAll/seekTo/attributes/close, directory listing, mount). Handles are
*values* — a File/Directory owns its VFS node id and byte offset — so there is no
per-process fd table or descriptor limit, unlike the POSIX fd model the shim
emulated. This is where the operations that later become std.os.danos are staged.
- Retire library/posix/ (unistd, stdio): only five call sites used it, all file
operations, all migrated to runtime.fs — fat (mount), the vfs-test and fat-test
clients, and init/log-flush (the boot-log flush). stdio was already dead.
- build.zig: drop the posix module, its addUserBinary parameter, the per-binary
import, and the ~26 call-site arguments.
- Docs: the VFS protocol's client is now runtime.fs; the docs index and
coding-standards note posix is retired and the foreign-ABI naming exception now
applies to the future std.os.danos seam; the process-lifecycle note points the
future musl layer at that same seam rather than the deleted directory.
Deferred by design (see the roadmap): the C-ABI runtime.os errno seam is built at
fork time (its shape must match std/os/danos.zig); truncate/mkdir/rename are
Phase 2; stdio-byte fds and cwd are later slices.
Verified: zig build, zig build test, zig build check-fat-image, and a sequential
QEMU sweep — vfs, vfs-client-death (the park/hold-handle path), fat-mount, log-flush,
orderly-shutdown, initial-ramdisk (log-flush silent in the bare sweep), smoke, init,
usb-storage, device-manager — all green.
On a headless or real board nothing captures serial, so the boot log — the whole
diagnostic stream — is lost at power-off. This retains it in the kernel and copies
it to the boot USB volume as /mnt/usb/DANOS.LOG, the on-disk equivalent of QEMU's
`-serial file:`. Pull the stick, read DANOS.LOG on another machine.
How it fits together:
- Kernel RAM sink (log.zig): a fixed 256 KiB in-image buffer registered as a log
sink in kmain, right after serial. Because userspace debug_write funnels through
log.write, it captures the entire stream — kernel lines and every service's
output — from the first line. Fills linearly and stops when full (earliest boot
output, the most valuable, is kept); no allocation, so it is panic-safe.
- klog_read syscall (32): copies that buffer out to a user buffer, the mirror of
debug_write — same overflow-safe user-half bounds check, kernel -> user copy,
under the kernel lock so the snapshot can't grow mid-copy. Wrapped by
runtime.system.klogRead.
- log-flush (new one-shot, in the initial-ramdisk): waits for the fat server to
mount /mnt/usb, then copies the whole log to /mnt/usb/DANOS.LOG. init spawns it
once the boot services are up (fire-and-forget; it polls the mount itself). If
no volume is mounted — no stick, or the no-VFS ramdisk sweep — it exits silently.
- init shutdown flush: init repeats the copy inline at the top of shutDown(),
BEFORE it tears down the storage services (the fat server is stopped first), so
a clean poweroff captures the fullest log while /mnt/usb is still writable.
- unistd.writeAll: loops write() past the 224-byte VFS payload cap; both flush
paths use it.
The filename is 8.3 (DANOS.LOG) at the mount root — the FAT short-name rule, and
there is no mkdir on the FAT path yet. Extend-only writes mean the two same-session
flushes never leave stale bytes (the shutdown log is a superset of the boot log);
a shorter log on a later boot of the same stick can leave a stale tail — a noted,
cosmetic limitation, not worth pulling O_TRUNC into the FAT write path for now.
Verified end to end under QEMU: a new `log-flush` case (reusing the orderly-shutdown
build) asserts both markers then S5, and DANOS.LOG is read back out of the image
afterwards (11804 bytes, containing the kernel init line, the FAT mount line, and
the boot flush marker). Regression stays green: zig build, zig build test, zig
build check-fat-image, and a sequential QEMU sweep — smoke, init, initial-ramdisk
(log-flush silent in the bare sweep), orderly-shutdown, fat-mount, usb-storage,
usb-hid, vfs, input, device-manager, process, signals, dma, fault-pf.
The system now boots off a real FAT32 filesystem on a USB mass-storage device
instead of QEMU's synthesized VVFAT drive. A new in-repo image builder formats
that filesystem from the FHS boot tree, and both QEMU call sites (the run step
and the test harness) attach it as a usb-storage device on the xHCI bus, so
every boot exercises the full USB path OVMF -> BOOTX64.efi -> kernel.
- tools/make-fat-image.py: a Python 3 stdlib-only FAT32 formatter (mirrors
tools/make-initial-ramdisk.py — no external host dependencies). It lays down
the boot sector + BPB/EBPB32, FSInfo, backup boot sector, two FATs, and the
root/subdir/file cluster chains, emitting long-name entries where a name is
not 8.3. Packs the four boot inputs (EFI/BOOT/BOOTX64.efi, system/kernel,
system/services/init, boot/initial-ramdisk.img) into their boot paths. A
--verify subcommand re-checks the 0xAA55 signature, recomputes the cluster
count -> FAT32, and resolves EFI/BOOT/BOOTX64.efi, all with no dependencies.
- build.zig: a mk_fat step builds zig-out/danos-usb.img from the four boot
artifacts (so changing -Dtest-case rebuilds the image with that kernel), a
check-fat-image step runs --verify, and run-x86-64 boots the image on a
usb-storage device (if=none,id=bootusb + usb-storage,bus=xhci.0,bootindex=0),
keeping usb-kbd/usb-mouse on the same controller.
- test/qemu_test.py: the default boot config now boots off danos-usb.img on a
usb-storage device (xHCI + usb-kbd + usb-mouse + the boot stick). The seven
per-case qemu_extra blocks that added their own qemu-xhci/usb-kbd/usb-mouse
(or a VVFAT stick) collided on id=xhci and are removed — the default provides
the bus and the boot device. usb-storage and fat-mount now exercise the real
FAT32 boot image (usb-storage reads its 0x55AA boot sector; fat mounts it at
/mnt/usb). A build_case override lets a case reuse another's kernel, used by a
new usb-boot case: an explicit, named boot-from-USB regression guard.
Verified: zig build, zig build test, and zig build check-fat-image are green
(FAT32, 128992 clusters, BOOTX64.efi present); a broad sequential QEMU sweep
passes — smoke, init, vfs, input, device-manager, usb-report, usb-hid,
usb-storage, fat-mount, device-list, driver-restart, acpi-report, iommu,
orderly-shutdown, usb-boot, dma, msi, initial-ramdisk, args, process — proving
the boot switch holds across kernel tests, the full init tree, the USB stack,
the FAT mount, and orderly shutdown.
Add a FAT12/16/32 filesystem the VFS mounts at /mnt/usb, reading and writing a
USB stick through the block device. Verified end to end under QEMU: the fat
server mounts the volume, the VFS routes /mnt/usb to it, and a client lists the
root and reads a file (the ELF magic of /mnt/usb/system/kernel).
- engine.zig: the FAT engine over a BlockDevice interface — mount (a bare FAT or,
as QEMU's VVFAT and most real sticks present it, an MBR-partitioned disk), FAT
chain walk (12/16/32), cluster allocation, directory traversal with long-name
read, and file read / write / create. Host-tested against a RAM-backed FAT16
image (create, cluster-spanning write, mid-file overwrite, read-back, list).
- on-disk.zig: the align(1) boot-sector / directory / long-name / FSInfo structs
and the cluster-count FAT-type detection.
- fat.zig: the server — wraps the .block device (a DMA bounce buffer) in a
BlockDevice, mounts the FAT, serves the vfs-protocol as a backend, and mounts
itself into the VFS at /mnt/usb. Spawned by init as a boot service.
- runtime.block: the block-device client (geometry / read / write by physical
address, so whole sectors never cross IPC).
- Raise the kernel service-name registry (maximum_services) 8 -> 16: it is
indexed directly by ServiceId, and fat = 8 was being rejected, so the fat
server exited before registering.
- VFS: an absolute path with no matching mount is now not-found rather than
silently created in the flat ramfs — so /mnt/usb fails cleanly until mounted.
Tests: fat-mount (the full stack: block -> FAT -> VFS mount -> list + file read)
passes; host units cover the engine and on-disk structs; the vfs, shutdown, and
USB regression suite stays green (10/10).
Turn the flat-ramfs VFS into a router: a mount table maps an absolute path prefix
(e.g. /mnt/usb) to a backend server's endpoint, and open/read/write/status/
readdir/close on a path under a mount are forwarded to that backend, which speaks
the same vfs-protocol. This is what a FAT filesystem mounts into.
- protocol: append readdir / mount / unmount operations, a NodeKind enum (the FSH
file types) that now fills FileStatus.kind, a DirectoryEntry record, and a
directory open flag. Appended values keep existing clients and tests unchanged.
- vfs.zig: a mount table, longest-prefix routing, forwarding of every op on a
backend handle, mount/unmount handlers (the backend arrives as the call's
capability), and release-on-death that also closes the backend's handles.
- path.zig: pure, host-tested mount-prefix matching that never captures a
non-boundary like /mnt/usbextra.
- unistd: mount(), opendir / readdir / closedir clients.
Bare names still resolve in the flat ramfs — the backward-compat contract; the
vfs and vfs-client-death tests pass unchanged. End-to-end mount+read is exercised
by the FAT server (M6). Host units cover path matching and protocol sizes.
Flesh out the xHCI host-controller driver into a full transfer engine and build
the three USB class drivers on top, all verified end to end under QEMU.
- xHCI engine (usb-xhci-library.zig): controller reset, command/event rings with
cycle-bit bookkeeping (gated on a No-Op-command proof), device slots, Address
Device, control transfers, full chapter-9 enumeration, Configure Endpoint, and
interrupt/bulk transfers. Each interface is device_registered with its
(class,subclass,protocol) identity, unique per (port,interface).
- Bus<->class transfer protocol (usb-transfer-protocol.zig + runtime.usb): open /
control / interrupt-subscribe (async report pump on a poll timer) / bulk-by-
physical-address, so sector data never crosses the 256-byte IPC limit.
- USB HID keyboard + mouse (usb-hid/): decode boot-protocol reports and publish
to the input service. A USB usage is already the input protocol's keycode.
- USB mass storage (usb-storage/): Bulk-Only Transport + transparent SCSI,
serving a block device under the new .block service id (block-protocol).
- device-manager matches USB interfaces to class drivers (usbDriverForIdentity).
- usb-abi / usb-ids made importable modules; add HID and mass-storage class
requests, packTriple, and a usb_device DeviceClass.
- Fix test/qemu_test.py on macOS: the QMP unix-socket path was built from the
deep worktree path and exceeded the 104-byte sun_path limit, so QEMU exited
before booting. It now lives under a short temp path.
Tests: usb-report, usb-hid, usb-storage pass under python3 test/qemu_test.py;
host units (usb-abi, usb-ids, hid-report, bulk-only-transport, scsi) green.
Time is a kernel concern in danos: the kernel owns the scheduling timer and
already exposes monotonic time via the clock/sleep/timer_bind syscalls, so a
userspace time service would be a redundant, slower path. This adds the generic
runtime.time module over those syscalls, retires the two demonstration drivers,
reorganizes the milestone docs, and makes the monotonic clock correct on Intel,
AMD, and inside any VM.
runtime.time (library/runtime/time.zig)
- Instant/Duration interface: now, sleep, spin, after, monotonicNanos, available
- a thin layer over system.clock/sleep/timerOnce; unit-tested arithmetic
Remove the demo drivers hpet and bus (a teaching example belongs in the docs,
not shipped in the tree)
- system/drivers/ now holds only real drivers: pci-bus, ps2-bus, usb-xhci-bus
- device-manager end-to-end test repointed to pci-bus (asserts on kernel state:
the process table and the device tree, not a racy serial marker)
- device_register containment moved to a new in-kernel `containment` test
- the driver-model worked example moved inline into docs/drivers.md
Reorganize milestone docs into topic docs
- m17-m18 / m19-m20 / m21 plans dissolved into process-lifecycle, device-manager,
discovery, and acpi docs; new docs/power.md and docs/timers.md; ~20 citations
repointed; plan docs deleted
TSC reliability (apic.zig, smp.zig, cpu.zig, kernel.zig)
- check the invariant-TSC bit (CPUID 0x80000007 EDX[8]) on Intel and AMD
- cross-core "warp" check at SMP bring-up, pairwise BSP<->AP as each core comes up
- fall back to the HPET clocksource when the TSC is not invariant (a bare VM) or
not synchronized (a warp), switched continuously so time never jumps
- boot log reports the outcome; new tsc-sync test exercises the TSC + warp path
Verified: zig build; zig build test; 60/60 QEMU cases (incl. new containment and
tsc-sync).
The AML interpreter now handles the Notify opcode (0x86, previously
unhandled): it resolves the target device, evaluates the code, and
records the pair in a bounded per-evaluate queue the caller drains with
takeNotifications. A host unit test with hand-encoded AML — a method that
issues Notify(DEV_, 0x80) — proves the device and code come back; aml.zig
joins the zig build test loop so the interpreter is covered on the host.
The acpi service's SCI handler now services general-purpose events too:
for each set-and-enabled GPE bit it evaluates the \_GPE._Lxx (level) or
_Exx (edge) handler method, drains the Notify queue that produced, and
publishes a domain event per notified device — PNP0C0A battery, ACPI0003
AC, PNP0C0D lid, else generic notify — then clears the status bit and
acks. The embedded controller's _Qxx queries are out of scope (hardware
track). QEMU raises no GPEs on this config, so the QEMU suite is the
regression net (the power button still works with GPE servicing in the
path); correctness is the unit test. Suite 59/59.
The kernel publishes the FADT as one more acpi-tables memory resource
(tagged by its intact FACP header — the AML blobs are header-stripped);
the acpi service reads the PM1 event/control and GPE register ports from
that copy, so the kernel's own FADT parse is untouched. A power-protocol
module (ServiceId.power = 5, domain-named so an ARM PSCI service can serve
the same id) carries subscribe / shutdown / events.
The acpi service converts to runtime.service.run — device discovery, the
.power protocol, and the SCI notification all fold into one loop. At
startup it enables ACPI mode if SCI_EN is clear (the SMI dance), binds the
SCI (found as the node's len-1 irq resource, distinct from the broad
window), and sets PWRBTN_EN. On the SCI it reads PM1_STS, clears
PWRBTN_STS write-1, logs the press, publishes power_button to
subscribers, and always acks. The power-button scenario proves it with a
real QMP system_powerdown injected mid-run through the M21.0 channel.
Two related changes to make device identities legible in the boot log and in
the code that matches on them.
Logging: the pci-bus driver decodes each function's class/subclass/prog-IF
triple to human names (via the existing pci-class module), and the acpi
service appends each _HID's human name (via acpi-ids) to its report line. So
"class 0x01 (Mass Storage Controller) subclass 0x06 (Serial ATA Controller)
progif 0x01 (AHCI 1.0)" reads straight off the log when writing a driver.
Naming: a new coding standard ("Named values, not magic numbers") says a value
with meaning gets a name, prefer an enum for value sets. Applied:
- pci-class is refactored from u8-switch tables into a BaseClass enum plus
per-class SubClass/ProgIf enums with name() methods (the usb-ids shape). The
public className/subclassName/progIfName(u8...) API is unchanged, so the
hardware-byte decoders (pci-bus, the kernel dump) are untouched; output is
byte-identical.
- the device-manager builds the xHCI class triple from named parts instead of
a bare 0x0C0330.
- the acpi service's _CRS walk names its resource-descriptor tags as
SmallResourceType/LargeResourceType enums, and the _HID integer decode uses
the AML module's existing *_opcode constants (now re-exported from aml.zig)
rather than bare 0x0A/0xFF/... literals.
The AML module becomes a build module compiled into both the kernel (for
the \_S5 sleep state it still needs) and the new acpi service — one
source, two builds, no fork. The kernel publishes a single acpi-tables
node: the DSDT/SSDT blobs as memory resources, a broad io_port grant (the
honest trust boundary — firmware AML names whatever ports it chose, known
only after parsing), and the SCI for the M21 event track. The acpi
service claims the node, maps each blob through the ordinary mmio grant
(which preserves the sub-page offset onto the bytecode), and runs the
same parser the kernel does. It self-verifies its namespace Device count
against the kernel's — 34 = 34 — deterministically via an argv the
acpi-parse test passes, so no racing the shared serial buffer. Parse-only
touches no hardware; OperationRegion evaluation waits for _CRS/_STA in
M20.2. The manager spawns 'discovery' (the neutral ramdisk name) at
startup. Suite 56/56.
The manager matches the pci_host_bridge node and spawns pci-bus with the
bridge id as its assignment — hello, supervision, restart, all the M18
contract for free. The driver claims the bridge, maps the ECAM window
(resource 0) through the ordinary mmio grant, and repeats the kernel's
brute-force bus/device/function walk from user space. The pci-scan
scenario builds its expected marker from the kernel's own function count,
so the two enumerations must agree exactly — the equivalence that
licenses retiring the kernel walk in M19.3.
system/services/acpi and system/services/fdt exist as documented
placeholders (silent clean-exit mains; the headers say exactly what each
becomes and why). The build's -Ddiscovery=acpi|fdt option fills the
ramdisk's neutral 'discovery' slot — the device manager will spawn
"discovery" by that name in M20.3 and never learn which firmware it is
on (m19-m20-plan.md decision 7). x86 defaults to acpi; the aarch64
target flips the default when it lands.
Applications ask the device manager for the tree (enumerate: a header
plus ChildEntry records) and subscribe to published add/remove events by
handing their endpoint over as the call's capability — the input-service
pattern; events are the same ChildAdded/ChildRemoved structs the bus
drivers send, one encoding in both directions. device-list is the first
client: it prints the tree, subscribes, and narrates the events through
a driver restart. The protocol's message maximum is capped at the
kernel's IPC MESSAGE_MAXIMUM (256 bytes, ten entries per reply; paging
joins the protocol when a tree outgrows one message). The startUserTask
debug print is gone: it wrote to serial unserialized against user-space
lines and sheared concurrent log markers in half — the root cause of the
scenario flakes.
The manager is now a harness service on the well-known .device_manager
endpoint. Every driver spawns supervised; drivers with an assignment must
hello (device-manager-protocol, versioned) within a deadline enforced by
a timer sweep. Exit reasons drive the restart decision: clean exits stay
down, faults restart with 300/600/1200ms backoff, and three fast deaths
mark a driver failed instead of respawning forever. usb-xhci-bus is the
first conforming driver; the crash-test fixture claims a device, hellos,
and faults on purpose — each respawn re-proving claim release on death
through the manager's own path. maximum_tasks grows 16 -> 32: the
initial-ramdisk sweep (15 binaries at once) was intermittently
overflowing the static pool.
The ps2-bus executable was built with the artifact name "bus" — a
copy-paste from the generic bus driver's line above it. The initial
ramdisk was unaffected (the packer pairs names with binaries
explicitly), so the driver ran at boot; but the FHS install uses the
artifact's own name, so both drivers landed on
zig-out/system/drivers/bus, one overwriting the other, and
zig-out/system/drivers/ps2-bus never existed.
Turn acpi-ids.zig's flat name table into a HardwareId enum modeled on
ps2-library's Port: one entry() switch holds the registry (variant ->
_HID string + human-readable name), with hid(), description(), and
fromHid() methods. The free description(hid) lookup the kernel's
device-tree dump uses survives, implemented over the enum, and a new
test round-trips every variant through fromHid.
Callers now name the device instead of quoting its id:
- ps2-library's DeviceType.hid() and ps2-bus's descriptor lookups use
HardwareId.ps2_keyboard / .ps2_mouse.
- device-manager's driverFor parses the HID once with fromHid and
switches on named values.
- acpi.zig's isPciRootNode carried the same ids twice, as strings and
as packed-EISA integers (0x030AD041/0x080AD041); both branches now
decode to the string form and answer through one isPciRootHid helper
using .pci_bus / .pci_express_root_bridge.
- build.zig threads the acpi-ids module (previously kernel-only) into
every user binary, like xkeyboard-config.
Replace the mouse driver's synthetic stream with the real path, the
way the keyboard was wired:
- The auxiliary port's IRQ12 is enumerated on the mouse's own ACPI
node (PNP0F13), and the kernel only lets a device's claimer bind or
ack its IRQs — so ps2-bus now claims that node alongside the
controller whenever port 2 carries a device, binds IRQ12 to its one
endpoint, and re-arms whichever line the notification's badge names.
The forwarding loop already routed auxiliary bytes by status bit 5.
- mouse-packet.zig (new, pure, host-tested): three-byte stream-mode
packet assembly — bit-3 sync with resynchronization, ACK/BAT bytes
dropped at packet start, nine-bit two's-complement movement,
overflow packets discarded, and PS/2 positive-Y-up converted to the
screen convention (positive down).
- mouse.zig mirrors the keyboard driver: no hardware claim, attaches
to the bus as its mouse, and publishes button_down/button_up per
changed button plus motion events with the pressed-button mask.
Verified end to end in QEMU via monitor mouse_move/mouse_button:
motion round-trips in screen coordinates, buttons transition with the
right mask, and keyboard events keep flowing alongside. The follow-up
is the IntelliMouse magic-knock for a scroll wheel (four-byte packets)
and scroll events.
Replace the keyboard driver's synthetic stream with the real path:
- ps2-bus binds IRQ1 (interrupt bits set only after the bind), drains
port 0x60 on each interrupt, and forwards every byte to the attached
child driver over async ipc_send, routed by the status register's
auxiliary-output bit. Children attach via the new well-known ps2_bus
service, handing over their endpoint as a capability.
- scancode.zig (new, pure, host-tested): scancode set 2 -> USB HID
usage decoding (F0/E0/E1 prefix state machine) plus keyboard state —
pressed-key bitmap, typematic-repeat classification, modifier and
caps-lock tracking.
- keyboard.zig decodes the forwarded stream and publishes real
key_down/key_press/key_up events, filling key_press characters via
xkeyboard-config (layout from argv[2], default us) and synthesizing
ASCII control characters for Enter/Tab/Backspace/Escape.
- protocol.zig names the full HID usage set in Keycode; build.zig
threads the xkeyboard-config module into user binaries.
Verified end to end in QEMU via monitor sendkey: shift, caps lock,
and control-character synthesis all decode correctly. The mouse
driver still publishes its synthetic stream; attaching it to the
bus the same way is the follow-up.
Turn a keycode + modifiers into a character. The input module delivers HID
usage keycodes but nothing mapped them to characters; rather than hand-maintain
layout tables, vendor the X11 xkeyboard-config database and compile it to native
Zig at build time (no X11 runtime), the way make-initial-ramdisk.py packs the
ramdisk.
- tools/make-xkeyboard-config.py: `fetch` downloads the pinned xkeyboard-config
release (2.44, sha256-verified), resolves the include graph for the configured
layouts, and vendors only the reached symbols files + keysymdef.h + COPYING +
PROVENANCE into library/xkeyboard-config/vendor/. `generate` parses that
(keycodes via a HID->xkb-name table, symbols with include/augment/override and
per-key type, keysymdef for keysym->Unicode) and emits generated/layouts.zig
deterministically.
- library/xkeyboard-config/xkeyboard-config.zig: the API over the generated data
— map(layout, hid_usage, mods) -> { keysym, character }, byName, and the
level-selection semantics (the generated tables stay pure data). Host tests
assert US letters/digits with Shift/Caps, GB £ vs US # on Shift+3, and French
AZERTY q-where-US-has-a — the end-to-end proof of the parse->emit->lookup path.
- build.zig: `xkeyboard-config` + `layouts` modules, the test wired into
`zig build test`, and a `zig build gen-xkeyboard-config` convenience step.
- Layouts: us, gb, de, fr, es, dvorak. Scope (documented): group 1, no dead-key
composition, curated key types. Standalone library; wiring it into the input
path to fill KeyEvent.character is a documented follow-up.
zig build test green (incl. the new keymap tests); regeneration is byte-identical;
full QEMU suite 48/48 (unaffected — no kernel/runtime/service change).
Programs can now subscribe to keyboard events (key_down/key_up/key_press)
and drivers can broadcast them, through a new user-space input service.
The delivery model is forced by danos IPC: a synchronous rendezvous holds
one pending reply, so a server cannot park N subscribers blocked in a
"wait for next event" call — delivery must be push. But a synchronous push
has no timeout and the kernel never wakes a sender parked on a dead peer's
endpoint, so one dying subscriber would hang all input. So this lands the
roadmap's planned asynchronous buffered send and builds the service on it:
- ipc_send (syscall 26): non-blocking post to an endpoint's bounded payload
ring, delivered through reply_wait as a buffered message (notify_message_bit).
A full ring drops the oldest. It can never hang on a dead/slow peer.
- input-protocol + runtime.input helpers (subscribe/next, connectSource/
publish) — the first real consumer of M13 capability passing: a subscriber
hands the service its own endpoint as a capability.
- input service (fan-out via ipc_send, dead-subscriber pruning), a synthetic
input-source, and input-test; the ps2-bus keyboard driver publishes to it.
Real IRQ1 scancode decoding (which must live in the bus, the PNP0303 owner)
is a documented follow-up; the source is synthetic for now.
- build/init wiring, an `input` QEMU case, and docs/input.md.
Full QEMU suite 48/48, including the new input case and every IPC/endpoint
regression (ipc, ipc-call, ipc-cap, vfs, hpet, bus, irqfree).
process_enumerate snapshots the task table (the device_enumerate shape, so
ps is a user program); system_spawn returns the child id, records the caller
as supervisor, and takes an exit endpoint; process_kill is allowed only for
the supervisor. Every death — exit, fault, or kill — posts a child-exit badge
to that endpoint (the IRQ-as-IPC pattern as SIGCHLD). A target caught off-CPU
is reaped in place; a running one is condemned and finished at its next
system call or tick, guarded so teardown never lands mid-kernel-operation.
Tested by process-list, process-kill, and supervision (a ring-3 supervisor
exercising the whole surface); design notes in docs/process-management.md.
Processes now start with C-compatible arguments: the kernel builds the
System V AMD64 entry block (argc, argv, empty envp, auxiliary vector)
at the top of the stack, argv[0] is the path or initial-ramdisk name
the process was spawned as, and system_spawn carries an optional
NUL-separated blob that becomes argv[1..]. The runtime parses the block
(runtime.argumentCount/argument) and its spawn wrappers pass arguments
through. The name is also recorded on the task, so a fault report says
which binary died, not just its id.
The user stack grows from one page to eight (32 KiB,
parameters.user_stack_pages), with the page below left unmapped as a
guard so an overflow faults into a clean process kill rather than
corrupting the image. Task.name_buffer is zero-initialised, not
undefined: an undefined default is materialised as a 0xAA fill that
moved the static task pool out of .bss and made the whole kernel ~7x
slower under QEMU TCG (caught by the affinity test).
Proven end to end by the new args test: args-echo respawns itself with
arguments via the syscall blob, burns more stack than one page could
hold, and echoes its argv intact. Full suite: 44/44.
The flat analog of pci-class for acpi_device nodes. ACPI has no class/subclass/prog-IF
taxonomy — a device's identity is its _HID string itself (PNP0303 *is* "PS/2
keyboard") — so this is a plain id -> name registry, not a hierarchical decoder.
New system/devices/acpi-ids.zig (module `acpi-ids`): the common standard PnP/ACPI
hardware IDs; vendor-specific ids (QEMU0002, etc.) have no registry name and print the
raw HID. Shared reference data like pci-class. The dump now names each _HID:
KBD_ [acpi_device] hid=PNP0303 (PS/2 Keyboard)
COM1 [acpi_device] hid=PNP0501 (16550A-compatible Serial Port)
RTC_ [acpi_device] hid=PNP0B00 (Real-Time Clock (RTC))
LNKA [acpi_device] hid=PNP0C0F (PCI Interrupt Link Device)
FWCF [acpi_device] hid=QEMU0002 (vendor-specific: raw HID)
Host test covers known ids and the unknown/empty fallthrough. Suite 41/41 plus host
tests.
Daniel Samson
Daniel Samson