Storage removal is now robust to all three ways a volume can leave, and the docs
say so. storage-architecture.md and storage-design-rationale.md move medium_changed
from "planned to be consumed" to consumed, and record the third trigger:
- the DEVICE leaving the tree (a pulled stick) — presence polling
- the MEDIUM leaving while its device stays (a reader) — the volume manager
now consumes the pushed medium_changed event
- the storage DRIVER crashing while its device stays — a channel-liveness
geometry() probe reaps the volume and rebuilds it on the restarted driver's
fresh channel; presence polling alone cannot see this (the V4 open edge)
The re-adopt-and-remount path is QEMU-proven by the driver-crash rebuild; a
physical unplug/replug is bench-verified (QEMU cannot re-present a usb-storage
device_add). The transport-native eject signal (SCSI UNIT ATTENTION, AHCI
PxSSTS, NVMe namespace-change AER) in place of the TEST UNIT READY poll stays
the documented future refinement.
Flip the storage docs to record exFAT as a built second engine reusing
the shared filesystem harness wholesale (the reuse the architecture
promised): full read + write, directories, rename, on-disk up-case
folding, routed by VBR to fat or exfat at an exfat-<serial> id-path.
Note the two surface limits shared by BOTH engines as vfs-layer concerns,
not exFAT shortcuts: u32 file offsets (a 4 GiB cap) and ASCII-only names.
Flip the storage docs from "one FAT volume today" to the built state:
the manager adopts every device, probes each device's whole partition
table, and spawns one range-confined FAT per volume — several volumes
across several devices, or several partitions on one device's channel.
The boot volume is identified by content; a filesystem installs the
/system rewrites only when it resolves /system/configuration on its own
media. No filesystem binds a shared service name any more — clients route
through the kernel mount table.
Correct two now-stale claims in the rationale, in the honest direction:
"multi-volume providers are reserved, not implemented" becomes built
(multi-namespace-per-provider is the untried NVMe case); and the cloned-
duplicate "second mounts suffixed" was never built — S3 mounts both,
they collide on the shared content id-path (last wins), each boot claim
logged, and distinguishing them is S4 arbitration.
Flip the storage docs to match S2: filesystems.csv (signature -> binary) and
volumes.csv (identity -> optional override) are built; a volume's mount path IS
its content id (/volumes/<id>), never a port name; the label is display metadata
a `volumes` query returns. fat receives its mount path via argv[2] rather than
hardcoding it. Still pending: rung 2 (filesystem UUID, needs a non-FAT engine),
multi-volume (fat's boot rewrites stay unconditional until S3), medium_changed
consumption, remount bench-verification.
Add volume-manager to build.zig's root package-test loop, so `zig build test`
runs the partition parser's nine host tests and a fixture added there can never
be silently skipped. Flip the identity-ladder build status in
storage-design-rationale.md: rungs 1 (GPT partition GUID) and 3 (FAT serial +
label) are built, joining rung 4; rung 2 (filesystem UUID) waits on a non-FAT
engine; the volumes.csv map and the id-derived mount path land with S2.
An all-tracks docs-vs-code audit (the same method that caught the storage
drift) found 23 confirmed inaccuracies where a doc's build-status claim no
longer matches the source — status markers that were never flipped after a
track landed, and a few paths left over from completed flag-days. All verified
against the code before editing; docs only, no behavior change.
The systemic ones:
- IOMMU enforcement (driver-model.md, drivers.md): docs said enforcement was
not built and "device_claim = ring 0" / "memory-safe is not true yet". It is
built (per-device VT-d/AMD-Vi domains programmed at device_claim, -ECONFINE
rollback, dma_alloc buffers bound and torn down at death; fail-open only with
no IOMMU). Restated; M16 marker flipped to done.
- The FHS flag-day paths: /etc/devices.csv -> /system/configuration/devices.csv
(devices-csv.md, new-driver-checklist.md, device-manager.md), /var/log ->
/system/logs (logging.md, new-driver-checklist.md), /mnt/usb -> /volumes/usb
(process-management.md). Following the old paths silently breaks driver match.
- protocol-namespace P4 "remaining" -> landed (only P5 remains); shared-fate
fan-out "not yet enforced" -> enforced; wall_clock "not built" -> built;
SMP affinity + fault-recovery "left" -> built; process_enumerate raw-pointer
trust model -> checked copyToUser/EFAULT; bounds.md maximum_devices static
hole -> dynamic per-registrar quota; init spawns fat -> volume-manager;
config "hardcoded, move to /etc" -> already CSV data files; vdso.md three-
value call; zig-self-hosting library/ layout; python argv "new" -> built.
Found and fixed by a multi-agent audit across 12 doc clusters, each finding
adversarially verified against the source.
A V5 close-out audit (docs against the actual code) found the storage docs
overclaiming in both directions: the status header was flipped to "built" but
several body markers were not, and two passages describe mechanisms the code
never implemented. Nine confirmed, adversarially verified against the source:
Underclaims (marked Planned, actually built):
- storage-architecture: driver named sub-ranges / range confinement (V2, tested
by the block-range case); the pushed medium_changed event (published today
from a TEST UNIT READY poll); ownership-gated fs_unmount (V0, process.zig
gates it with EPERM); the filesystem-harness extraction (V1). Scoped the
remaining *planned* to the genuinely-pending parts (native-signal translation,
the volume manager consuming medium_changed).
Overclaims (described, never built):
- storage-architecture: the "FAT dirty flag on disk" guarantee — no on-disk
dirty/clean-shutdown bit exists; only an in-memory device-dirty bool gating a
device write-cache flush on close. Fixed in all three places.
- storage-architecture: fat "acquires its own volume (first mass-storage child
by enumeration order)" — the V3b flip removed self-acquisition; fat is handed
its volume id and channel by the volume manager.
- rationale + plan: the FAT engine's base_lba "deleted rather than moved" — it
and the engine's MBR walk still exist as now-inert legacy; the authoritative
walk lives in partition.zig.
- rationale: NVMe namespaces "decision 4 settles as endpoint-per-volume" —
decision 4 settles the opposite (per-sender confinement, one endpoint);
endpoint-per-volume is named only as an unbuilt future refactor.
- rationale: the five-rung identity ladder and volumes.csv map stated in flat
present tense — only rung 4 (MBR signature + index) is built; added the
build-status hedge and marked each rung.
Docs only; no code or behavior change. Suite unaffected (127/127).
The volume manager, per-sender range confinement + gate, medium_changed
event, per-volume spawning + supervision, ownership-gated fs_unmount, and
the removal half of the lifecycle are built. Left honestly pending: the
volumes.csv/filesystems.csv maps, the fuller identity ladder, multi-volume,
the VM consuming medium_changed (removal uses device-presence polling), and
remount-on-replug end-to-end (bench-pending — QEMU can't re-present the
boot-controller device). Full suite 127/127.
Per-sender range confinement at the provider, one serving endpoint: the
badge-scoped provider pattern the xHCI bus already uses (the per-client
device-token table), applied to blocks. Endpoint-per-volume would buy the
same enforced property only by inventing a multi-endpoint harness; it stays
available as a future refactor, same wire contract. The plan's flag-for-veto
is gone: nothing in the track waits on a choice.
The sharpest instance of the return story, folded into the architecture: the
boot volume is a recorded content identity (the volume carrying
/system/configuration and /system/logs), the system runs on without it (the
ramdisk is the OS; only log persistence pauses), the kernel ring is the
buffer during absence — bounded, so a wrapped ring is a data loss window
that gets MARKED in the file on resume, never spliced silently — and on
return at any port the same identity remounts the same prefixes and the
logger appends into the same boot-stamp tree. The logger requirement is
named: failed flushes retry on the patient cadence, never abandoned after
the first not_found. The dirty-honesty rule stands; the boot volume gets no
exemption.
Decision 8 in the rationale, mirrored into the architecture's volume-manager
section: the mount map keys on CONTENT identity, never port or arrival order
— Linux's /dev/sda1-era fstab broke on every port move until UUID= replaced
it, and danos skips that era. The identity ladder the prober reads off the
medium: GPT partition GUID, filesystem UUID, FAT serial+label, MBR
signature+index, anonymous. Consequences are mechanical: port moves change
nothing (USB, hub, SATA bay, or transport swaps), replug remounts at the
same path, the boot volume is a recorded identity findable anywhere, and
cloned duplicates are a loud policy case instead of silent shadowing.
/volumes/<name> stands as the hierarchy's home for attached media; minting
identifiers (formatting, entropy) stays deliberately out of scope.
storage-stack-discussion.md was misplaced at the docs root — that level is
for track plans; this is the file-system domain's design record. Moved to
file-system-development/storage-design-rationale.md, renamed to say what it
is, cross-references updated.
The NVMe/SATA assessment folded into the storage discussion: what transfers
untouched (the block contract and everything above it — one driver binary
plus one devices.csv row per transport), NVMe as the shorter stack whose
namespaces are the reserved multi-volume case, AHCI as an open shape choice
(leaning per-port processes, the matrix-proven granularity), and the three
pressure points named honestly: multi-volume is reserved-not-implemented,
synchronous call/reply bottlenecks NVMe until the shm-ring data plane, and
media lifecycle is not device lifecycle.
That last one becomes decision 7 and enters the architecture doc: the
removal path has TWO TRIGGERS, ONE LIFECYCLE — channel death (device leaves)
and a planned pushed medium_changed event (medium leaves, device stays: card
readers and trays, USB ones today), translated by the storage driver from
its transport's native signal, presence never content, consumed by the
volume manager into the same kill-retire-remount path. Without it a swapped
card would be served with the previous card's filesystem state.
The enforcement section of the storage architecture: the three levers the
device lifecycle already proved, mapped onto volumes — a filesystem can only
be GIVEN its volume (no establishment grants, channel at spawn), the volume
manager supervises with teeth (deadline, kill-on-removal, crash-loop cap),
and the shared harness makes every engine inherit the state machine by
construction (engines never see channels). Kernel backstops: ownership-gated
fs_unmount + the lazy dead-endpoint sweep. Checkable via a lifecycle
conformance drill parameterized over filesystems — supporting a filesystem
MEANS passing it.
The settled shape from the storage-stack discussion, written as the
reference: the data path (vfs -> filesystem service -> block -> driver) as
the application/service/protocol/driver model applied twice; the three kinds
of boundary (protocol between processes, library inside them, control-plane
beside them); the volume manager as the policy home (planned — the FAT
service squats on its duties today, marked as such); adding a filesystem as
engine + shared harness + one configuration row; and the per-layer
responsibility table for removable media — one removal path, kill/retire/
respawn, dirty data lost and SAID to be lost. Indexed from docs/README.md.
Three services had each written the same thing and got it three different
ways: input polled the process list to notice a dead subscriber, and only
when someone else subscribed; the power service never noticed at all; the
device manager noticed drivers but not subscribers. The harness owns the
table now, driven by the events a protocol declares — it registers on the
reserved verb, frames each event once, posts to everyone interested without
waiting on any of them, and reclaims a slot when the kernel says its owner
died. Interest masks moved to the envelope, so a subscriber that wants only
mice asks the same way everywhere.
Two consequences the plan had not foreseen. The device manager now hears a
supervised child's death twice, once as its supervisor and once as a
subscriber, so restart backoff counted every crash twice and gave up after
half as many; it retires the id before counting. And the kernel's published
exit table had eight slots for what is now six subscriptions in a plain
boot, so it holds sixteen.
The other half is a hole the design named early and left standing: a
backend handed out a small integer and then honoured it from anyone. A
process that guessed a file's node id read another client's file; a display
layer had no owner at all, so any client could reconfigure or destroy any
layer; a USB device token was never checked against the client that opened
it. Each is now bound to the task that opened it, and a wrong owner gets
exactly what an unknown id gets — the refusal must not become the oracle
the identical answers elsewhere were designed to remove. Closing a file
changed with it: it used to succeed unconditionally, which would have told
a caller which ids existed.
Suite 111/111, with a new case in which one process holds a file and a
layer, hands both ids to a second process, and finds them untouched after
that process has tried everything with them.
The folded header stops being a rule in a document and becomes the layout
on the wire. Verbs number from sixteen, leaving describe, enumerate,
subscribe and unsubscribe reserved and answered the same way by every
provider — none of them writes a line to do it. What each protocol used to
carry in a field of its own now travels in the header: a vfs node and a
display layer are the packet's target, and a reply opens with a status the
envelope stamps rather than one each protocol spelled for itself.
Display gains the most. One forty-byte request had served eleven verbs, so
attach_scanout smuggled stride through x, refresh through y and format
through colour, and every coordinate crossed as a bitcast. Per-operation
structs end all three: the fields have their own names and their own signs,
and the tile payload grows to 224 bytes because the prefix shrank. Scanout
loses a message maximum of 64 it had no business declaring — it answers
calls, and the floor for a call is 256 — and virtio-gpu stops hard-coding
that number at its harness.
Two changes are semantic rather than notational. A directory now ends at an
entry with no name, because the fixed part of a reply always travels and a
zero-length reply no longer exists to mean anything. And input joins the
service harness, the last loop in the tree that answered no ping and heard
no terminate; its subscriber table, its pruning and its fan-out are the
same code, and a shutdown now asks it to stop instead of killing it.
A new conformance case reads the registry's own listing and asks every
protocol it finds for its name, its version and its verb count, then offers
a verb nobody defines and requires -ENOSYS — the envelope's promise,
checked against providers rather than against itself. What it cannot reach
in that boot it names on the serial line instead of passing quietly.
Suite 110/110.
A protocol is now defined through envelope.Define rather than beside it.
Every packet begins with the same 16-byte prefix: a Header of {operation,
target} for a request or event, a Status for a reply. The header is folded
into each protocol's own layout, never stacked on top of it — Define walks
a spec's types and refuses one that carries a Header or Status field, so
the rule cannot be missed by reading. Operations number from 16, leaving
describe, enumerate, subscribe and unsubscribe reserved and answerable the
same way everywhere; describe the envelope answers itself, and an unknown
verb is -ENOSYS without a provider writing a line.
The size floor becomes something protocols can compile against: 256 bytes
for a call, 64 for a push, matching the kernel's own limits, checked at the
Define call with the prefix counted in. A protocol that outgrows either
fails the build with the reason and the remedy — shrink, or move the bulk
to shared memory, because packets never fragment. Events carry the header
too and number in their own space, so appending an operation can never
renumber a shipped event.
Beside it, the client half of the model: a Channel is what opening a
/protocol name yields, wrapping the endpoint capability so a program holds
a conversation rather than a handle. Nothing speaks through either yet —
the vfs protocol gains NodeKind.protocol and the convention that an open
reply may carry a capability, and that is the whole runtime change.
Suite 107/107, unchanged: P1 converts nothing.
The security-track design set. communication.md is the model: four layers
(namespace / protocol / channel / transport), packets and signals, parties
addressed by the channel and objects by target, transports as replaceable
buffer+doorbell mechanisms. protocol-namespace.md is L3+L2: /protocol names
contracts, resolution establishes a channel, init is the registrar,
restriction is per-process namespace delegation (with the microphone-prompt
worked example), the envelope is the universal packet header, migration
P1-P5 retires ServiceId. file-system-hierarchy.md replaces the unix-FHS
spec with the danos-native tree (/applications, /protocol, /system,
/volumes) and its migration table. ipc.md is re-cut as the kernel-ipc
transport document. smep-smap.md designs the kernel hardening: copy
discipline for the eight raw user-pointer syscalls, then SMEP, then SMAP
as a permanent tripwire.
Daniel Samson