Daniel Samson
8754d4e46a
The source layout now mirrors the runtime filesystem hierarchy
(docs/danos-file-system-hierarchy-FSH.md): what lives under system/ in the
source is what a running danos represents under /system. Each service and
driver is a sub-project directory that is its own Zig module — cross-project
references go by module name, never by a path into another project's files.
Moves (all git mv, history preserved):
- src/ -> system/ (danos internals; the self-representation)
root.zig -> danos.zig (the kernel<->user contract module)
kernel/arch/ -> kernel/architecture/ (arch -> architecture)
device/ -> devices/ (what /system/devices reflects)
boot/ -> /boot (the loaders, top level)
- sbin/ -> split by role:
init, vfs -> system/services/<name>/<name>.zig
hpetd, busd -> system/drivers/<name>/<name>.zig
vfs-test -> system/services/vfs/vfs-test.zig (inside the vfs project)
- lib/ -> library/runtime/ (room for other libraries beside runtime)
The VFS wire protocol becomes its own module, system/services/vfs/protocol.zig
("vfs-protocol"): the vfs sub-project exposes its interface, and the runtime's
file layer imports it by name. First instance of the "protocol module" pattern
(docs/driver-model.md); usb/block will expose theirs the same way.
Also: fix a naming-standard violation in the protocol — Op -> Operation (and
req -> request, _pad -> _padding). Docs updated: /system/services added to the
FHS doc, a repository-layout section added to the docs index, and stale source
paths swept across comments and docs.
Runtime boot paths are unchanged (the bootloader still loads /sbin/init);
aligning the runtime filesystem to the FHS is a separate follow-up. Suite 35/35
plus host tests green.
101 lines
5.2 KiB
Markdown
101 lines
5.2 KiB
Markdown
# IPC: message-passing channels
|
|
|
|
Inter-process communication is the **backbone of a microkernel**. Once drivers and
|
|
services run isolated in their own address spaces ([vision](vision.md)), they can't
|
|
just call each other — a request becomes a **message**. In a microkernel, whatever
|
|
was a function call across a monolithic kernel is IPC, so it's a first-class
|
|
concern, not an afterthought.
|
|
|
|
There are two layers, built a milestone apart:
|
|
|
|
- **`system/kernel/ipc.zig`** — a bounded blocking channel between *kernel threads*,
|
|
described below. The primitive, and where the blocking discipline was worked out.
|
|
- **`system/kernel/ipc-synchronous.zig`** — synchronous call/reply between *processes*, across
|
|
address spaces. What user-space servers and drivers actually talk over. It's the
|
|
second half of this document.
|
|
|
|
## The channel
|
|
|
|
The first form is a **bounded blocking channel** (`system/kernel/ipc.zig`): a fixed-size
|
|
ring buffer of messages with a producer/consumer rendezvous, built on the
|
|
scheduler's [wait queues](scheduling.md).
|
|
|
|
`Channel(T, capacity)` is generic over the message type and buffer size. It holds a
|
|
ring buffer, a count, and two wait queues:
|
|
|
|
- **`send(msg)`** — if the channel is full, block on the *not-full* queue; otherwise
|
|
write the message, bump the count, and wake a waiting receiver.
|
|
- **`recv()`** — if the channel is empty, block on the *not-empty* queue; otherwise
|
|
take a message, drop the count, and wake a waiting sender.
|
|
|
|
Neither side busy-waits: a full channel parks the sender, an empty one parks the
|
|
receiver, and each operation wakes the other side when it makes progress possible.
|
|
|
|
Two details make it correct:
|
|
|
|
- **Recheck in a loop.** A woken task re-tests the condition (`while (full) wait`)
|
|
rather than assuming the slot is still available — another waiter may have taken
|
|
it first. This is the standard guard against spurious or racing wakeups.
|
|
- **One critical section.** `send`/`recv` run under `saveInterrupts` /
|
|
`restoreInterrupts` (the composable form, see [scheduling.md](scheduling.md)), so
|
|
checking the condition and committing the block/enqueue happen atomically with
|
|
respect to the timer preempting mid-operation. `waitLocked` / `wakeLocked` are the
|
|
variants that assume the caller already holds that critical section.
|
|
|
|
## Verifying it
|
|
|
|
The `ipc` test (see [testing.md](testing.md)) runs a producer and a consumer passing
|
|
**100 messages through a 4-slot channel**. The small buffer means the channel goes
|
|
full and empty over and over, so both the blocking-send and blocking-recv paths are
|
|
exercised heavily. The messages arrive intact and in order (their sum is the
|
|
expected `5050`), and neither task busy-waits — they block and wake each other.
|
|
|
|
## Endpoints: call/reply across address spaces
|
|
|
|
A channel connects two kernel threads sharing one address space. Real servers are
|
|
*processes*, so the payload has to cross an address-space boundary. That's
|
|
`system/kernel/ipc-synchronous.zig`, and its shape is L4's: a synchronous **rendezvous** at an
|
|
`Endpoint`, with the message copied directly from the sender's pages to the receiver's
|
|
(`copyAcross` walks both sets of page tables through the physmap — no CR3 switch, no
|
|
bounce buffer).
|
|
|
|
Two syscalls carry it:
|
|
|
|
- **`ipc_call(h, msg, reply)`** — copy `msg` to the server, block until it replies.
|
|
- **`ipc_reply_wait(h, reply, recv)`** — reply to the client you're still holding (if
|
|
any), then block for the next request. One syscall, because a server's steady state
|
|
is *always* "finish the last one, wait for the next".
|
|
|
|
An endpoint is reached by **handle** — a small integer index into the process's handle
|
|
table (`Task.handles`), exactly like a file descriptor, and just as unforgeable. The
|
|
bootstrap problem (how do you get the first handle?) is solved by a tiny name registry:
|
|
a server calls `ipc_register(service_id, h)` under a well-known small integer, and a
|
|
client calls `ipc_lookup(service_id)`.
|
|
|
|
The server never learns the client's identity beyond a **badge**, delivered alongside
|
|
the message: the caller's task id.
|
|
|
|
### Interrupts are messages too
|
|
|
|
`notifyFromIsr` posts an *asynchronous* notification to an endpoint — no payload, no
|
|
reply owed — and wakes whoever is blocked in `reply_wait`. Its badge has the top bit
|
|
set (`notify_badge_bit`), which is how a driver's single event loop distinguishes "a
|
|
client wants something" from "the hardware wants something". Notifications sit in a
|
|
small coalescing ring on the endpoint, so an interrupt taken while the driver was busy
|
|
elsewhere is not lost.
|
|
|
|
This is what makes a user-space driver possible at all, and it's the subject of
|
|
[drivers.md](drivers.md).
|
|
|
|
## What's next (not done here)
|
|
|
|
- **Priority inheritance** through IPC, so a high-priority client blocked on a
|
|
low-priority server doesn't suffer unbounded priority inversion.
|
|
- **Handle transfer.** A server can't hand a client a handle to a third endpoint, so
|
|
every capability is either well-known (the registry) or inherited — there's no way
|
|
to delegate one.
|
|
- **Asynchronous / buffered send** for the cases where a rendezvous is the wrong
|
|
shape (logging, notifications between servers).
|
|
- **A bounded reply.** `MSG_MAX` is 256 bytes and the copy runs under the big kernel
|
|
lock; a bulk transfer wants shared pages, not a copy.
|