danos/docs/file-system-development/storage-architecture.md

The storage architecture: layers, boundaries, responsibilities

Status: the layered model below is the settled design (storage-design-rationale.md records how it was reached, and volume-manager-plan.md how it was built). Built (V0–V4 + the storage-stack S1/S2): the data path, the driver range confinement (per-sender clamp + the confinement gate), the medium_changed presence event, the volume manager itself — it probes the partition table, confines each filesystem to its partition, spawns one filesystem per volume, and supervises it — the removal half of the lifecycle (a pulled stick unmounts), the identity ladder (GPT GUID + name, FAT serial + label, MBR), and the mount map: filesystems.csv (signature → binary) + volumes.csv (identity → optional override), a volume's mount path IS its content id (/volumes/<id>), with the label as display metadata a volumes query returns. Multi-volume is built: the manager adopts every storage device, probes each device's whole partition table, and spawns one range-confined FAT per volume — several volumes across several devices, or several partitions sharing one device's channel — each at its own /volumes/<id> path with its own supervision. The boot volume is identified by content (a volume backs /system/configuration + /system/logs only when it resolves /system/configuration on its own media), so it works as any partition of any device. exFAT is built as a second engine (system/services/exfat): full read + write, directories, rename, and on-disk up-case folding, reusing library/kernel/file-system-harness wholesale — the reuse claim, proven — and a volume routes to fat or exfat by its VBR, at an exfat-<serial> id-path. Removal is robust to all three triggers now: a pulled device (presence polling), a medium that leaves while its device stays (the volume manager CONSUMES medium_changed), and a storage driver that crashes while its device stays present (a channel-liveness geometry() probe reaps the volume and rebuilds it on the restarted driver's fresh channel). The re-adopt-and-remount path is QEMU-proven by the driver-crash rebuild; a physical unplug/replug exercises the same path but is bench-pending (QEMU cannot re-present a usb-storage device_add). Still pending: the filesystem UUID rung (ext-family superblocks, which need such an engine); and arbitration when two volumes both resolve the boot markers (S3 mounts both and logs each claim; picking one is deferred). A few markers below are left where a duty is still pending.

The model

One pattern, applied twice: an application is a client of a service over a protocol; a service is a client of a driver over a protocol.

application
    │  vfs protocol            (routed by the kernel mount table)
    ▼
filesystem service  ──  one process per volume
    │                   inside: [file-protocol shell] ── Zig API ── [fs engine]
    │  block protocol          (channel received at spawn)
    ▼
storage driver      ──  one process per device (usb-storage per stick)
    │  usb-transfer protocol   (channel received via hello, by lineage)
    ▼
bus driver          ──  one process per controller (usb-xhci-bus)
    │  hardware

Three kinds of boundary, deliberately different:

• Protocol boundaries sit where processes meet (vfs, block, usb-transfer). Each side is independently restartable; channels are established by capability handoff, never by registry names (communication.md "Establishment: two planes"). Steady-state file I/O crosses exactly two — vfs and block.
• Library boundaries sit where concerns meet inside one process. The filesystem service's engine (on-disk format logic, host-testable, behind the four-function BlockDevice vtable) and shell (IPC serving, open-node table, mount registration) talk through a Zig API. No protocol between them: they share fate regardless — a corrupt engine corrupts the answers either way — so a channel there would add a hop per file operation and buy nothing. The seam exists for compile-time testability and reuse; the process is the restart unit.
• Control-plane relationships sit beside the data path, never on it. Two supervisors, one per layer: the device manager wires and revives the device layers (bus and storage drivers — devices only); the volume manager (built) wires and revives the volume layer (filesystem services). Neither touches steady-state I/O.

Who does what

Bus driver (usb-xhci-bus, per controller): hardware only. Enumerates, reports children to the device manager, serves the transfer contract to its own children's class drivers, routed by lineage.

Storage driver (usb-storage, per device; later nvme, ahci): hardware → blocks. Speaks its transport upward from its device; serves the block protocol (geometry, read, write, flush, attach, detach — and the pushed medium_changed presence event, published today from a slow TEST UNIT READY poll; planned: translating it from the transport's native signal instead of polling). Content-blind, permanently: it reads block 0 only as a bring-up self-check and parses nothing — no MBR, no GPT, no filesystem magic, ever. It has one content-blind mechanism (built): named sub-ranges — "serve blocks [a, b) as a channel of the same block contract". It clamps and offsets; it never knows the numbers came from a partition table. The clamp lives here and nowhere else because a channel must carry exactly the authority it grants: handing a filesystem the whole disk plus a polite base offset would let a buggy or compromised filesystem scribble the neighboring partition — the same authority-overshoot the device-authority track eliminated for MMIO and DMA.

Volume manager (built; system/services/volume-manager): the policy home of the volume layer, one service, supervised by init. It watches the device manager's tree for a storage provider; when one appears it consumer-hellos for the block channel, reads the partition table and the first blocks itself (it is the prober), defines the volume's sub-range on the driver, spawns the matching filesystem service confined to that range, and supervises it (backoff, crash-loop cap). (Built): it picks the filesystem binary from filesystems.csv by the volume's content signature, and mounts the volume at its content id (/volumes/<id>) — or a volumes.csv override. The label is display metadata the volumes query returns, never the path. Those tables are CSV configuration, read by it (the policy), enforced by nobody else:

• filesystems.csv (built) — content signature → filesystem binary. Adding a filesystem adds a row.
• volumes.csv (built) — the mount map, danos's fstab: an OPTIONAL volume identity → mount prefix override (a volume with no row mounts at its default /volumes/<id>), keyed on content identity and never on port, path, or arrival order (the lesson of Linux's /dev/sda1-era fstab, which broke on every port move until UUID= replaced it). Identity is read off the medium by the prober, strongest first: GPT partition GUID → filesystem UUID → FAT serial + label → MBR signature + partition index → anonymous (generated name, no persistence). Same identity, same mount point: a drive moved to another port — USB to another hub, SATA to another bay, even a stick returning in a different dock — lands exactly where it was. Duplicate identity (cloned sticks, together) is policy: first keeps the name, the second mounts suffixed and is logged loudly. The boot volume is the recorded identity of the volume carrying /system/configuration and /system/logs, findable on any port. Every volume's default mount is /volumes/<id> — its rendered content identity.

Filesystem service (the FAT service today; one process per volume): the proven unit — block-client + engine + file-protocol provider in one binary. It receives its block channel at spawn; it never discovers devices. It registers its own mounts with the kernel; its write cache lives inside the process, so a write error is observed by the code that owns the volume and surfaces on the owning channel (the anti-fsyncgate rule — never a system-wide dirty pool). (Built:) fat receives its mount path as argv[2] from the volume manager (the volume's id-path, e.g. /volumes/fat-12345678) and mounts its root there. It installs the two /system hierarchy rewrites (/system/configuration, /system/logs) only when it is the boot volume — decided by content: it resolves /system/configuration on its own media at mount, so a data volume mounts at its id-path alone and never shadows the running system. It no longer self-acquires a volume — the V3b flip made it receive its volume id and block channel from the volume manager, consistent with "it never discovers devices" above. Because several volumes now serve at once, no filesystem binds a shared service name; clients reach each through the kernel mount table (fs_resolve routes by prefix to the backing endpoint).

Kernel (mechanism only): the mount table routes paths to backend endpoints — resolve and redirect, never data. Remount-replace is the restart story; a dead backend's slot is swept lazily on the next resolution. fs_unmount is ownership-gated (built, V0) — only the mounting task may unmount its own prefix; anyone else is refused (EPERM). No dead-owner exception: a dead owner's slot is swept lazily by resolution, and restart goes through remount-replace, never through a stranger's unmount.

Adding a filesystem

1. Write the engine: pure Zig, no IPC, behind the BlockDevice vtable (four functions). On-disk layout in align(1) extern structs. Host-test it with zig build test fixtures — the FAT engine (engine.zig, on-disk.zig) is the template, and its ~500 lines of host tests the standard.
2. Reuse the shell: the filesystem harness — establishment, the badge-scoped open-node table, the nine vfs-protocol handlers, mount registration, the removal path — is shared code, not per-filesystem code. (Done: extracted from fat's original 434-line shell into library/kernel/file-system-harness.zig; fat imports it and instantiates harness.Server(engine.FileSystem).) An engine plus a main wiring it into the harness is a complete filesystem service.
3. Add the configuration row: one line in filesystems.csv mapping the on-disk signature to the binary. No other component changes: the volume manager probes, matches, spawns; the vfs protocol is already backend-neutral (a second provider serves it in production today — init's synthetic registry backend).
4. Prove removal: extend the removable-media suite (below) for the new filesystem — surprise-yank during writes must lose only what was unflushed, said honestly, with the volume consistent enough to remount.

The engine never sees: partitions (it receives a volume-shaped block channel; base offsets are the driver's clamp), device discovery (the channel arrives at spawn), mount policy (prefixes are handed to it), or other volumes (one process, one volume).

Removable media: responsibilities on removal, per layer

The design rule, learned from what Linux cannot do: there is exactly one surprise-removal path — kill the filesystem process, retire its mounts, respawn on return. No half-alive states, no remount-ro, no mounts that error forever (Plan 9's dead-server wart).

The path folds three triggers into one lifecycle: the device leaving (a pulled stick — presence polling); the medium leaving while the device stays (an SD card pulled from its reader, an ATAPI tray opened, a USB card reader); and a storage driver crashing while its device stays in the tree. The second trigger is the pushed medium_changed event on the block protocol, published from a TEST UNIT READY poll — the volume manager now consumes it (subscribed per device), running the same kill-retire path and re-probing on medium return, so a swapped card is never served with the previous card's filesystem state. The third is caught by a channel-liveness geometry() probe: presence polling alone sees the device still present, but the channel is dead, so the manager reaps the volume and rebuilds it on the restarted driver's fresh channel. Still planned is translating the transport's native signal (SCSI UNIT ATTENTION, AHCI PxSSTS, NVMe namespace-change AER) in place of the presence poll.

Layer	Observes	Must do	Guarantees
Bus driver	port/hub status change	tear down the device's slots (children first, recursively — built, hot-plug matrix), report child_removed per interface	the device tree is honest within one reconcile tick
Device manager	child_removed / reporter death	prune the child; reap the bound driver (built) — the storage driver for that stick dies now, not never	no zombie storage processes; re-report rebinds
Storage driver	its own death (it IS the removed device's driver)	nothing — dying is its removal handling; DMA/IOMMU/claims release mechanically at death	in-flight transfers fail visibly to callers, never hang
Volume manager (built)	a device leaving the tree (poll), a medium_changed event, or a dead channel under a still-present device (a crashed driver — geometry() liveness probe)	kill that volume's filesystem service (its mounts retire), then re-adopt + remount on return or on the restarted driver's fresh channel	one removal path for all three triggers; mounts never dangle; the manager never serves from behind a dead channel
Filesystem service	its block channel dies (EPEER) mid-operation, or it is killed by the volume manager	if it observes the death first: flush nothing (the medium is gone), answer in-flight requests with errors, exit; dirty write-back data is lost and said to be lost	the unflushed write-back window is dropped on a surprise yank — danos writes no on-disk dirty/clean-shutdown marker today; the process never serves from behind a dead channel
Kernel	backend endpoint death	lazy mount-slot sweep on next resolve (built); ownership-gated fs_unmount (built, V0)	resolution under a dead mount is not_found, not a hang
Application	not_found / error on paths under the vanished mount	its own error handling — the contract is honest absence, identical to the path never existing	no operation blocks forever on removed media

On return the same table runs upward in reverse: the bus re-enumerates and re-reports; the device manager respawns the storage driver (built); the volume manager re-probes — same content, same volume identity — respawns the filesystem service, and remounts at the same prefix; applications see the subtree reappear. A different stick in the same port is a different volume (identity is content, not port) and mounts wherever its identity says — possibly nowhere but /volumes/….

The boot volume: identity is what makes yanking it survivable

The boot volume is the sharpest instance of the return story, because the system's own log persistence rides it (/system/logs), and because nothing else about the running system depends on it at all — every binary and the boot-time configuration live in the initial ramdisk. Its lifecycle, end to end:

• At first mount the volume manager records the identity of the volume carrying /system/configuration and /system/logs — THE boot volume, from then on a fact about content, not about a port.
• While it is absent, the system runs on. The kernel log ring keeps accumulating — it is the buffer that makes the absence survivable — and the logger keeps draining it; only persistence pauses. File operations under the retired mounts fail honestly (not_found). The ring is bounded, so a long absence overwrites its oldest entries: that window is the data loss, and it must be said — the logger marks the gap in the file when persistence resumes, never splicing the stream silently.
• On return — any port, any hub, even a different transport — the prober reads the same identity, the mount map answers with the same prefixes, the filesystem service is respawned, and /system/logs is the same tree it was: the logger resumes appending into the SAME <boot-stamp> directory, per-binary files continuing where they left off (plus the gap marker if the ring wrapped).
• What never comes back is the write-back window lost at the yank — the dirty-honesty rule, unchanged; the boot volume gets no exemption.

This is also the requirement that shapes the logger: it treats the log tree as a volume that comes and goes — failed flushes are retried on the same patient cadence the fat service already uses for storage that arrives late, never abandoned after the first not_found.

What is lost on a surprise yank is exactly the write-back window of the filesystem service, no more: the engine owns its cache, so the blast radius of a yank is one volume's unflushed writes, which are simply lost — danos records no on-disk dirty/clean-shutdown bit yet. A filesystem format with better crash honesty (journaling, copy-on-write — the lesson of QNX's Power-Safe) narrows that window further and slots in as implementation N+1 through the table above, changing nothing else.

How the lifecycle is enforced

Responsibility tables are convention; convention is worthless (the bounds audit's lesson). The volume lifecycle is enforced with the same three levers the device lifecycle already uses, mapped one-to-one:

1. A filesystem cannot acquire — it can only be given. Filesystem binaries hold NO establishment grants: no open device-manager, no registry name to look up. The only block channel a filesystem process ever has is the one handed to it at spawn by the volume manager. Serving the wrong volume, a second volume, or a self-discovered volume is not forbidden but impossible — the same way a driver cannot claim hardware it was not delegated (device-authority.md).
2. The volume manager supervises with teeth. Mount-within-deadline or be stopped (a filesystem wedged on a corrupt volume is killed and crash-loop capped; the volume is marked bad, not retried forever). On removal the manager KILLS the filesystem process and retires its mounts — the polite observe-EPEER-and-exit path is an optimization; the kill is the guarantee, because a process cannot be trusted to observe its own obsolescence (the reap argument, proven on the device tree).
3. The shared harness is how every filesystem inherits the lifecycle by construction. The harness — not the engine — owns the state machine: establishment at spawn, mount registration, the flush-on-close hook (an in-memory device-dirty check that commits the device write cache on close), error-out-and-exit on channel death. The engine sits behind the four-function vtable and never sees a channel; it cannot opt out of the lifecycle for the same reason it cannot find a device. This is why the harness is extracted BEFORE the second engine is written.

Beneath all three, two kernel mechanisms: ownership-gated fs_unmount (only the mounting endpoint's holder may unmount), and the lazy dead-endpoint mount sweep as the backstop nothing can disable. And above them, the check: a lifecycle conformance drill — mount, serve, yank mid-write, verify honest loss, replug, remount — parameterized over filesystem implementations, so "danos supports filesystem X" MEANS "X passes the drill through the harness", exactly as provider conformance means passing the reserved-verb suite.