Files
danos/docs/os-development/communication.md
T
Daniel Samson 0d5a7394ef docs: establishment planes — the design and the two-seam plan
The namespace holds protocols, never instances; what multiplies is provider
processes, and their establishment routes through the device manager, which
owns the topology. communication.md gets the model; the plan converts the
usb-transfer and block seams in one flag-day, closes the restart-zombie hole
the scoping found, and pins the three-controller mouse bug as a QEMU case.
2026-08-09 11:26:53 +01:00

8.8 KiB

Communication: the four layers

Design, agreed 2026-07-31. The model document — the vocabulary and layering every other communication document speaks.

danos separates what is said from how the bytes move, so that the mechanism is replaceable. The shape is a network stack's, cut into four layers; a program only ever touches the top two.

L3  namespace   /protocol/... names establishment points          protocol-namespace.md
L2  protocol    the language: packet schemas, verbs, targets      the envelope, library/protocol/*
L1  channel     two ends exchanging packets and signals           the client library's Channel
L0  transport   a buffer + a doorbell: moves the bytes            ipc.md (kernel-ipc), later shm-ring, …

Vocabulary

Term Meaning
protocol The language: which packets exist, what their fields mean, which verbs a provider answers. Defined transport-independently in a library/protocol/* module.
channel An open conversation between two processes, speaking one protocol. Established by opening a /protocol/... name; both ends can send and receive.
packet The unit a protocol transmits: a bounded, atomic header+payload. Never fragmented — if it doesn't fit, it isn't a packet; bulk data rides shared memory with a packet as the doorbell.
signal A payload-less poke below the packet layer: "something happened, come look." Coalescing — the count may collapse, the fact may not.
transport What moves the bytes of one channel: a buffer plus a doorbell. Chosen (and upgradable) at establishment, invisible above L1.
endpoint A termination point where a transport delivers. The kernel-ipc transport's endpoint is its kernel mailbox object.

Addressing: parties by channel, objects by target

There are no network-style addresses in a packet. The two questions addresses answer are answered at different layers:

  • Who am I talking to? The channel, decided once at establishment. Opening /protocol/input yields a channel; every packet sent on it goes to the peer. Nothing to route per-packet — like TCP, where no HTTP request carries the server's IP.
  • Who sent this? Attached to every received packet by the channel layer, from identity the transport can verify — under kernel-ipc, the kernel-stamped badge. The sender never writes a source field, which is what makes source unforgeable (the property a network's spoofable source header lacks).
  • Which of your things? The packet's target field: object addressing within the already-chosen peer — the vfs protocol's node id, the display protocol's layer id, a block volume. target = 0 addresses the provider itself; a protocol without objects never uses it.

target is how instance multiplicity stays out of the namespace. Ten USB sticks and the namespace still holds exactly one name, /protocol/block: a channel to the provider, enumerate lists the current volumes as targets, a targets_changed signal announces hotplug, and a read names its volume in target. The unix /dev/sda,/dev/sdb problem is dissolved, not renamed.

If a future transport genuinely routes between machines, it carries real source/destination addressing internally at L0 — the way IP runs under TCP — and none of it surfaces into the packet header. Protocols stay ignorant of distance.

Establishment: two planes, one namespace

target routes objects within a provider. It cannot route between provider processes, because "who am I talking to" was decided at establishment — so when a protocol's objects live in several processes, something must resolve establishment to the right one. A real machine forced the question: three xHCI controllers, three usb-xhci-bus processes, one usb-transfer name, and whichever instance bound first owned every class driver's discovery — a mouse on the second controller was unreachable (could not open device 50). The registry's exclusive bind was built for parties that are genuinely singular; per-device driver processes are deliberately not singular — one process per device found is what makes a single driver recompilable and restartable while the machine runs.

So establishment has two planes, and the namespace stays instance-free on both:

  • Applications establish by name. /protocol/input, /protocol/display, /protocol/ufs — served by services, the layer that aggregates (driver-model.md: services face applications). These are genuinely singular, the registry's exclusive bind is correct for them, and multiplicity behind them is target's job. Ten mice still merge into one /protocol/input.
  • Drivers establish by topology. A driver instance creates its channel — its own endpoint — and hands it up in the hello it already sends, as a capability. The device manager is the establishment router for the device tree because it already owns the routing fact: it knows which instance reported which child. A consumer's hello for device N returns a capability to N's provider. The name usb-transfer names the contract; no process binds it.

Establishment is always a synchronous call, and always consumer-initiated: the kernel carries one capability in each direction of a call (request and reply), and deliberately none on a push — a provider cannot foist a channel on anyone. Re-establishment after a provider restart is therefore also consumer-initiated: the consumer's channel dies with the provider, and a fresh hello fetches the successor's.

Strip any transport to its skeleton and the same two parts remain:

Transport Buffer Doorbell Status
kernel-ipc kernel-owned mailbox (the Endpoint) the scheduler (rendezvous wake) the first transport — ipc.md
shm-ring user-owned shared-memory ring a signal exists ad hoc (display bulk); to be formalized — the unlock for the 256-byte ceiling
network NIC queue an interrupt someday, when danos networks

Transports differ in their properties, which the channel layer exposes and the protocol layer may depend on:

  • packet ceiling — kernel-ipc: 256 bytes request/reply, 64 pushed. An shm-ring's ceiling is its slot size. Kernel-ipc's 256 is the floor every protocol may assume everywhere.
  • synchrony — kernel-ipc's call is a rendezvous: natural backpressure, no queue to size. An asynchronous transport buffers, so a channel over one needs explicit flow control. Backpressure is a transport property, not a channel guarantee — protocols that rely on it say so.
  • droppability — pushed event packets may drop when a ring fills; request/reply may not.
  • capability carriage — only kernel-ipc can move a capability. Handles are kernel objects; a user-space ring cannot transfer one. So kernel-ipc is always the establishment and control transport — channels are born on it, capabilities ride it — even when a channel's data is negotiated onto something fatter.

That negotiation is the upgrade path: a channel starts on kernel-ipc; the protocol's handshake may then delegate a shared-memory region (as a capability, over kernel-ipc) and move its bulk traffic there. The display path already does exactly this by hand; formalizing it in the channel layer makes it every protocol's option.

The channel (L1)

A channel has two ends, and the ends are peers: each may send packets, each may receive, each may signal. Request/reply is a pattern over the channel — a send with a correlated receive, which the kernel-ipc transport happens to accelerate as a single rendezvous — not the definition of it. The event stream (subscribe, then pushes) and the change signal (poke, then re-read) are the other two patterns; all three are catalogued in protocol-namespace.md's wiring section.

The channel layer's obligations: deliver packets whole, attach the verified source to every receive, expose the transport's properties, and hide the transport's mechanics. The client library's Channel type is this layer made concrete — a program holds channels that speak protocols and never touches a raw handle.

The protocol (L2) and the namespace (L3)

A protocol defines its packets through the envelope — every packet begins {operation, target}, reserved verbs (describe, enumerate, subscribe, unsubscribe) mean the same thing in every protocol, and Define checks every packet against the transport floor at compile time. The full treatment, including how names are granted, resolved, and restricted per process, is protocol-namespace.md.

Establishment points are named by contract — /protocol/display, never /protocol/ipc-1 — because the name must outlive the mechanism: a transport named in the namespace could never be swapped, which would defeat this document's premise.