reorg: move mmio into library/device and spell out its API

mmio is device-driver code, so it joins the other domains under
library/device/mmio/ (module name "mmio" unchanged — a pure relocation, only
the build paths move). And its abbreviated function names are spelled out per
docs/coding-standards.md:

  read  -> readRegister          mb  -> memoryBarrier
  write -> writeRegister         rmb -> readMemoryBarrier
                                 wmb -> writeMemoryBarrier

All call sites updated (virtio-gpu, usb-xhci-library, pci.Function); the two
display-driver placeholders import mmio but use nothing, so they're untouched.
Docs (driver-model graph, README layout, drivers.md, the FHS note) follow the
new path and names.

zig build + test green; virtio-gpu, display-native, display-reattach, usb-hid,
usb-hub, usb-storage, pci-scan pass.
This commit is contained in:
Daniel Samson
2026-07-22 21:28:32 +01:00
parent 7d540c4b2f
commit 5b874fc756
9 changed files with 82 additions and 83 deletions
+3 -3
View File
@@ -428,9 +428,9 @@ pub fn build(b: *std.Build) void {
// Typed volatile MMIO register access + memory-ordering barriers, for drivers on // Typed volatile MMIO register access + memory-ordering barriers, for drivers on
// top of an mmio_map grant. Depends only on `builtin` (arch-conditional barriers); // top of an mmio_map grant. Depends only on `builtin` (arch-conditional barriers);
// no target set, so it inherits each driver's. See library/mmio/mmio.zig. // no target set, so it inherits each driver's. See library/device/mmio/mmio.zig.
const mmio_module = b.addModule("mmio", .{ const mmio_module = b.addModule("mmio", .{
.root_source_file = b.path("library/mmio/mmio.zig"), .root_source_file = b.path("library/device/mmio/mmio.zig"),
}); });
// A device driver's view of its claimed PCI function: config-space header fields, BAR // A device driver's view of its claimed PCI function: config-space header fields, BAR
@@ -969,7 +969,7 @@ pub fn build(b: *std.Build) void {
"library/device/acpi/aml/aml.zig", // AML parse + interpret, incl. Notify dispatch (M21) "library/device/acpi/aml/aml.zig", // AML parse + interpret, incl. Notify dispatch (M21)
"library/device/usb/usb-abi.zig", // wire sizes + bit packings + set-up packet encodings "library/device/usb/usb-abi.zig", // wire sizes + bit packings + set-up packet encodings
"library/device/usb/usb-ids.zig", // class/subclass/protocol code assignments "library/device/usb/usb-ids.zig", // class/subclass/protocol code assignments
"library/mmio/mmio.zig", // barriers assemble + registers round-trip "library/device/mmio/mmio.zig", // barriers assemble + registers round-trip
"system/drivers/ps2-bus/scancode.zig", // set-2 decode + keyboard state machine "system/drivers/ps2-bus/scancode.zig", // set-2 decode + keyboard state machine
"system/drivers/ps2-bus/mouse-packet.zig", // 3-byte mouse packet assembly "system/drivers/ps2-bus/mouse-packet.zig", // 3-byte mouse packet assembly
"system/drivers/usb-hid/hid-report.zig", // HID boot-report keyboard/mouse decode "system/drivers/usb-hid/hid-report.zig", // HID boot-report keyboard/mouse decode
+2 -3
View File
@@ -245,10 +245,9 @@ system/ → /system danos's own internals (the self-representation)
fat.zig, engine.zig, on-disk.zig) fat.zig, engine.zig, on-disk.zig)
library/ → /lib libraries, one sub-directory each library/ → /lib libraries, one sub-directory each
runtime/ the danos-native runtime + file API (fs) — the stable application ABI runtime/ the danos-native runtime + file API (fs) — the stable application ABI
mmio/ volatile register access + memory barriers device/ device code by domain — mmio/ model/ pci/ usb/ acpi/ — each a
device/ device code by domain — model/ pci/ usb/ acpi/ — each a
shareable data module (device-abi, pci-class, usb-abi/ids, shareable data module (device-abi, pci-class, usb-abi/ids,
acpi-ids) plus a logic module (pci, usb, aml) acpi-ids) plus a logic module (mmio, pci, usb, aml)
protocol/ driver↔service wire contracts (vfs block display scanout input protocol/ driver↔service wire contracts (vfs block display scanout input
power device-manager usb-transfer), one module per directory power device-manager usb-transfer), one module per directory
boot/ → /boot the loaders boot/ → /boot the loaders
+1 -1
View File
@@ -87,7 +87,7 @@ A block driver is now **writable, but not yet memory-safe.** Every storage contr
worth naming is a bus master: it is programmed by handing it the physical address of a worth naming is a bus master: it is programmed by handing it the physical address of a
descriptor ring and left to read and write memory on its own. That ring is exactly what descriptor ring and left to read and write memory on its own. That ring is exactly what
**`dma_alloc`** now provides — physically contiguous, pinned, uncacheable, with its **`dma_alloc`** now provides — physically contiguous, pinned, uncacheable, with its
physical address disclosed — and **`/lib/mmio`**'s barriers order the descriptor writes physical address disclosed — and **`/lib/device/mmio`**'s barriers order the descriptor writes
against the doorbell, and **`msi_bind`** delivers completions. So an AHCI or NVMe driver against the doorbell, and **`msi_bind`** delivers completions. So an AHCI or NVMe driver
can be written today (the M14/M15 work in [driver-model.md](driver-model.md); the earlier can be written today (the M14/M15 work in [driver-model.md](driver-model.md); the earlier
"cannot host a block driver at all" is no longer true). "cannot host a block driver at all" is no longer true).
+13 -13
View File
@@ -106,9 +106,9 @@ module outlived it, which is rather the point.) The pattern generalises directly
``` ```
library/ library/
runtime/ module "runtime" — syscalls, ipc, lifecycle, memory, threads, log, fs runtime/ module "runtime" — syscalls, ipc, lifecycle, memory, threads, log, fs
mmio/ module "mmio" — volatile register access + barriers [M14]
device/ device code grouped by domain; each domain splits into a shareable device/ device code grouped by domain; each domain splits into a shareable
data module (enums/wire types, std-only) and a logic module (mmio/IPC) data module (enums/wire types, std-only) and a logic module (mmio/IPC)
mmio/ module "mmio" — typed volatile register access + barriers [M14]
model/ module "device-abi" — DeviceDescriptor, DeviceClass, ResourceKind model/ module "device-abi" — DeviceDescriptor, DeviceClass, ResourceKind
pci/ "pci-class" (data) + "pci" — config/BAR/capability walk (Function) pci/ "pci-class" (data) + "pci" — config/BAR/capability walk (Function)
usb/ "usb-abi" + "usb-ids" (data) + "usb" — descriptors, control/interrupt/bulk client usb/ "usb-abi" + "usb-ids" (data) + "usb" — descriptors, control/interrupt/bulk client
@@ -160,12 +160,12 @@ class driver, the device manager, or the kernel may share them freely.
`callCap` and `replyWait(..., send_cap)`, and class drivers consume them now: the `callCap` and `replyWait(..., send_cap)`, and class drivers consume them now: the
PS/2 keyboard and mouse drivers attach to ps2-bus this way, and `runtime.usb` / PS/2 keyboard and mouse drivers attach to ps2-bus this way, and `runtime.usb` /
`runtime.input` open their per-device and subscription channels with `callCap`. `runtime.input` open their per-device and subscription channels with `callCap`.
- **M14** — DMA memory + the memory-ordering layer. `/lib/mmio` gives drivers typed - **M14** — DMA memory + the memory-ordering layer. `/lib/device/mmio` gives drivers typed
volatile access and `mb`/`rmb`/`wmb` (per-arch); `dma_alloc`/`dma_free` grant volatile access and `memoryBarrier`/`readMemoryBarrier`/`writeMemoryBarrier` (per-arch); `dma_alloc`/`dma_free` grant
physically-contiguous, pinned, uncacheable, reclaim-on-teardown buffers with the physically-contiguous, pinned, uncacheable, reclaim-on-teardown buffers with the
physical address exposed (`pmm.allocContiguous`, a DMA arena, `mapUserDmaInto`). physical address exposed (`pmm.allocContiguous`, a DMA arena, `mapUserDmaInto`).
`dma_below_4g` caps the address for legacy engines; `dma_write_combining` is accepted `dma_below_4g` caps the address for legacy engines; `dma_write_combining` is accepted
but falls back to coherent until PAT is programmed. The bus drivers use `/lib/mmio`, but falls back to coherent until PAT is programmed. The bus drivers use `/lib/device/mmio`,
and `dma_alloc` has real consumers now: the xHCI driver's rings and contexts, and `dma_alloc` has real consumers now: the xHCI driver's rings and contexts,
usb-storage's command/status wrappers, virtio-gpu's virtqueue, and the fat usb-storage's command/status wrappers, virtio-gpu's virtqueue, and the fat
service's bounce buffer. service's bounce buffer.
@@ -252,7 +252,7 @@ const dev_ep = ipc.callCap(h, // ... mint a per-device endpoint,
## M14 — DMA memory and the memory-ordering contract, for HCDs ✅ done ## M14 — DMA memory and the memory-ordering contract, for HCDs ✅ done
*Implemented: `/lib/mmio` (typed volatile access + `mb`/`rmb`/`wmb`, per-arch) and *Implemented: `/lib/device/mmio` (typed volatile access + `memoryBarrier`/`readMemoryBarrier`/`writeMemoryBarrier`, per-arch) and
`dma_alloc`/`dma_free` (contiguous, pinned, uncacheable, reclaim-on-teardown, physical `dma_alloc`/`dma_free` (contiguous, pinned, uncacheable, reclaim-on-teardown, physical
address exposed). `dma_write_combining` still falls back to coherent — real WC needs address exposed). `dma_write_combining` still falls back to coherent — real WC needs
PAT, a small follow-up. The rest of this section is the original design note.* PAT, a small follow-up. The rest of this section is the original design note.*
@@ -294,23 +294,23 @@ doorbell.* = i; // volatile store to UC MMIO
// nothing stops the compiler reordering these; the device reads a stale descriptor // nothing stops the compiler reordering these; the device reads a stale descriptor
``` ```
So the rules, which belong in `library/mmio.zig` and behind `arch`: So the rules, which belong in `library/device/mmio/mmio.zig` and behind `arch`:
| Situation | Required | | Situation | Required |
|---|---| |---|---|
| MMIO register read/write | `mmio.read` / `mmio.write` (volatile) | | MMIO register read/write | `mmio.read` / `mmio.write` (volatile) |
| Fill DMA descriptor, then ring doorbell | `wmb()` between them | | Fill DMA descriptor, then ring doorbell | `writeMemoryBarrier()` between them |
| Woken by IRQ, then read what the device wrote | `rmb()` before the read | | Woken by IRQ, then read what the device wrote | `readMemoryBarrier()` before the read |
| MMIO write that must complete before the next read | `mb()` | | MMIO write that must complete before the next read | `memoryBarrier()` |
And the per-arch lowering — the reason this must be an `arch` primitive and not a And the per-arch lowering — the reason this must be an `arch` primitive and not a
sprinkling of `asm volatile`: sprinkling of `asm volatile`:
| | x86_64 | aarch64 | | | x86_64 | aarch64 |
|---|---|---| |---|---|---|
| `mb()` | `mfence` | `dsb sy` | | `memoryBarrier()` | `mfence` | `dsb sy` |
| `rmb()` | `lfence` | `dsb ld` | | `readMemoryBarrier()` | `lfence` | `dsb ld` |
| `wmb()` | `sfence` | `dsb st` | | `writeMemoryBarrier()` | `sfence` | `dsb st` |
| DMA cache coherency | coherent; nothing to do | **not guaranteed**; needs non-cacheable buffers or cache maintenance | | DMA cache coherency | coherent; nothing to do | **not guaranteed**; needs non-cacheable buffers or cache maintenance |
x86 is forgiving here — TSO plus strong-uncacheable MMIO means you usually get away x86 is forgiving here — TSO plus strong-uncacheable MMIO means you usually get away
@@ -318,7 +318,7 @@ with a compiler barrier alone. ARM is not, and [vision.md](vision.md) makes ARM
condition. Build the abstraction while there is one caller to fix. condition. Build the abstraction while there is one caller to fix.
(Zig note: `@fence` was **removed in 0.16**. Use `@atomicRmw(..., .seq_cst)` for a full (Zig note: `@fence` was **removed in 0.16**. Use `@atomicRmw(..., .seq_cst)` for a full
barrier, or per-arch inline asm — which is what `library/mmio.zig` should hide.) barrier, or per-arch inline asm — which is what `library/device/mmio/mmio.zig` should hide.)
## M15 — interrupts for PCI devices ✅ done (MSI) ## M15 — interrupts for PCI devices ✅ done (MSI)
+2 -2
View File
@@ -290,8 +290,8 @@ Several things this list used to warn about are now available (see
[driver-model.md](driver-model.md)): **port I/O** (`io_read`/`io_write`, claim-gated by [driver-model.md](driver-model.md)): **port I/O** (`io_read`/`io_write`, claim-gated by
the device's `io_port` resource — direct ring-3 `in`/`out` is still a #GP, so a PS/2 or the device's `io_port` resource — direct ring-3 `in`/`out` is still a #GP, so a PS/2 or
16550 driver goes through these), **DMA memory** (`dma_alloc`: contiguous, pinned, 16550 driver goes through these), **DMA memory** (`dma_alloc`: contiguous, pinned,
uncacheable, physical address exposed), **memory barriers** (`library/mmio`'s uncacheable, physical address exposed), **memory barriers** (`library/device/mmio`'s
`mb`/`rmb`/`wmb`, imported as the `mmio` module), **fault isolation** (a ring-3 fault kills only the faulting `memoryBarrier`/`readMemoryBarrier`/`writeMemoryBarrier`, imported as the `mmio` module), **fault isolation** (a ring-3 fault kills only the faulting
process — `killCurrentProcess` — and the machine keeps running, process — `killCurrentProcess` — and the machine keeps running,
[resilience](resilience.md)), and **reclaim + restart on death** (every path out of a [resilience](resilience.md)), and **reclaim + restart on death** (every path out of a
process releases its claims and IRQ/MSI bindings — `releaseAllOwnedBy`, process releases its claims and IRQ/MSI bindings — `releaseAllOwnedBy`,
@@ -1,4 +1,4 @@
//! /lib/mmio — typed volatile MMIO register access, plus the memory-ordering //! /lib/device/mmio — typed volatile MMIO register access, plus the memory-ordering
//! barriers a device driver needs. Used by drivers on top of an `mmio_map` grant. //! barriers a device driver needs. Used by drivers on top of an `mmio_map` grant.
//! //!
//! **`volatile` is not a barrier.** In Zig it means only: don't elide this access, and //! **`volatile` is not a barrier.** In Zig it means only: don't elide this access, and
@@ -11,13 +11,13 @@
//! doorbell.* = i; // volatile store to UC MMIO //! doorbell.* = i; // volatile store to UC MMIO
//! // nothing orders these; the device can read a stale descriptor //! // nothing orders these; the device can read a stale descriptor
//! //!
//! Put a `wmb()` between them. The barriers lower per-architecture — which is the whole //! Put a `writeMemoryBarrier()` between them. The barriers lower per-architecture — which
//! reason they are a named primitive and not scattered `asm volatile`: //! is the whole reason they are a named primitive and not scattered `asm volatile`:
//! //!
//! x86_64 aarch64 //! x86_64 aarch64
//! mb() mfence dsb sy //! memoryBarrier() mfence dsb sy
//! rmb() lfence dsb ld //! readMemoryBarrier() lfence dsb ld
//! wmb() sfence dsb st //! writeMemoryBarrier() sfence dsb st
//! //!
//! x86 is forgiving (TSO + strong-uncacheable MMIO), so a compiler barrier usually //! x86 is forgiving (TSO + strong-uncacheable MMIO), so a compiler barrier usually
//! suffices; ARM is not, and ARM is the win condition (docs/vision.md) — so the //! suffices; ARM is not, and ARM is the win condition (docs/vision.md) — so the
@@ -29,52 +29,52 @@ const builtin = @import("builtin");
/// Read a register of type `T` at absolute virtual address `addr` — a location inside /// Read a register of type `T` at absolute virtual address `addr` — a location inside
/// a device's `mmio_map` grant. `volatile`: never elided, never reordered against /// a device's `mmio_map` grant. `volatile`: never elided, never reordered against
/// another volatile access. /// another volatile access.
pub inline fn read(comptime T: type, addr: usize) T { pub inline fn readRegister(comptime T: type, addr: usize) T {
return @as(*const volatile T, @ptrFromInt(addr)).*; return @as(*const volatile T, @ptrFromInt(addr)).*;
} }
/// Write `value` of type `T` to the register at absolute virtual address `addr`. /// Write `value` of type `T` to the register at absolute virtual address `addr`.
pub inline fn write(comptime T: type, addr: usize, value: T) void { pub inline fn writeRegister(comptime T: type, addr: usize, value: T) void {
@as(*volatile T, @ptrFromInt(addr)).* = value; @as(*volatile T, @ptrFromInt(addr)).* = value;
} }
/// Full barrier: all loads and stores before it are globally visible before any after /// Full memory barrier: all loads and stores before it are globally visible before any
/// it. Use when an MMIO write must complete before a following read. /// after it. Use when an MMIO write must complete before a following read.
pub inline fn mb() void { pub inline fn memoryBarrier() void {
switch (builtin.target.cpu.arch) { switch (builtin.target.cpu.arch) {
.x86_64 => asm volatile ("mfence" ::: .{ .memory = true }), .x86_64 => asm volatile ("mfence" ::: .{ .memory = true }),
.aarch64 => asm volatile ("dsb sy" ::: .{ .memory = true }), .aarch64 => asm volatile ("dsb sy" ::: .{ .memory = true }),
else => @compileError("mmio.mb: unsupported architecture"), else => @compileError("mmio.memoryBarrier: unsupported architecture"),
} }
} }
/// Read barrier: loads before it complete before loads after it. Use after an IRQ /// Read memory barrier: loads before it complete before loads after it. Use after an IRQ
/// wake, before reading what the device wrote to shared memory. /// wake, before reading what the device wrote to shared memory.
pub inline fn rmb() void { pub inline fn readMemoryBarrier() void {
switch (builtin.target.cpu.arch) { switch (builtin.target.cpu.arch) {
.x86_64 => asm volatile ("lfence" ::: .{ .memory = true }), .x86_64 => asm volatile ("lfence" ::: .{ .memory = true }),
.aarch64 => asm volatile ("dsb ld" ::: .{ .memory = true }), .aarch64 => asm volatile ("dsb ld" ::: .{ .memory = true }),
else => @compileError("mmio.rmb: unsupported architecture"), else => @compileError("mmio.readMemoryBarrier: unsupported architecture"),
} }
} }
/// Write barrier: stores before it become visible before stores after it. Use between /// Write memory barrier: stores before it become visible before stores after it. Use
/// filling a DMA descriptor in RAM and ringing the device's doorbell. /// between filling a DMA descriptor in RAM and ringing the device's doorbell.
pub inline fn wmb() void { pub inline fn writeMemoryBarrier() void {
switch (builtin.target.cpu.arch) { switch (builtin.target.cpu.arch) {
.x86_64 => asm volatile ("sfence" ::: .{ .memory = true }), .x86_64 => asm volatile ("sfence" ::: .{ .memory = true }),
.aarch64 => asm volatile ("dsb st" ::: .{ .memory = true }), .aarch64 => asm volatile ("dsb st" ::: .{ .memory = true }),
else => @compileError("mmio.wmb: unsupported architecture"), else => @compileError("mmio.writeMemoryBarrier: unsupported architecture"),
} }
} }
test "barriers emit and registers round-trip through a RAM cell" { test "barriers emit and registers round-trip through a RAM cell" {
// The barriers must at least assemble for the host arch; ordering can't be unit // The barriers must at least assemble for the host arch; ordering can't be unit
// tested, but a missing/mistyped mnemonic is caught here. // tested, but a missing/mistyped mnemonic is caught here.
wmb(); writeMemoryBarrier();
rmb(); readMemoryBarrier();
mb(); memoryBarrier();
var cell: u64 = 0; var cell: u64 = 0;
write(u64, @intFromPtr(&cell), 0xDEAD_BEEF); writeRegister(u64, @intFromPtr(&cell), 0xDEAD_BEEF);
try @import("std").testing.expectEqual(@as(u64, 0xDEAD_BEEF), read(u64, @intFromPtr(&cell))); try @import("std").testing.expectEqual(@as(u64, 0xDEAD_BEEF), readRegister(u64, @intFromPtr(&cell)));
} }
+11 -11
View File
@@ -32,23 +32,23 @@ pub const Function = struct {
} }
pub fn vendorId(self: *const Function) u16 { pub fn vendorId(self: *const Function) u16 {
return mmio.read(u16, self.config + pci_class.config_vendor_id); return mmio.readRegister(u16, self.config + pci_class.config_vendor_id);
} }
pub fn deviceId(self: *const Function) u16 { pub fn deviceId(self: *const Function) u16 {
return mmio.read(u16, self.config + pci_class.config_device_id); return mmio.readRegister(u16, self.config + pci_class.config_device_id);
} }
pub fn command(self: *const Function) u16 { pub fn command(self: *const Function) u16 {
return mmio.read(u16, self.config + pci_class.config_command); return mmio.readRegister(u16, self.config + pci_class.config_command);
} }
pub fn status(self: *const Function) u16 { pub fn status(self: *const Function) u16 {
return mmio.read(u16, self.config + pci_class.config_status); return mmio.readRegister(u16, self.config + pci_class.config_status);
} }
/// Set Memory-Space + Bus-Master enable in the command register. Firmware often leaves /// Set Memory-Space + Bus-Master enable in the command register. Firmware often leaves
/// a secondary display's decode off; a bus-mastering device must enable both. /// a secondary display's decode off; a bus-mastering device must enable both.
pub fn enableMemoryAndBusMaster(self: *const Function) void { pub fn enableMemoryAndBusMaster(self: *const Function) void {
const at = self.config + pci_class.config_command; const at = self.config + pci_class.config_command;
mmio.write(u16, at, mmio.read(u16, at) | pci_class.command_memory_and_bus_master); mmio.writeRegister(u16, at, mmio.readRegister(u16, at) | pci_class.command_memory_and_bus_master);
} }
/// Decode BAR `bar` (0..5) and map it: read the BAR register, reject I/O-space BARs, /// Decode BAR `bar` (0..5) and map it: read the BAR register, reject I/O-space BARs,
@@ -60,11 +60,11 @@ pub const Function = struct {
if (bar >= 6) return null; if (bar >= 6) return null;
if (self.bar_virtual[bar] != 0) return self.bar_virtual[bar]; if (self.bar_virtual[bar] != 0) return self.bar_virtual[bar];
const low = mmio.read(u32, self.config + pci_class.config_bar0 + @as(usize, bar) * 4); const low = mmio.readRegister(u32, self.config + pci_class.config_bar0 + @as(usize, bar) * 4);
if (low & pci_class.bar_io_space != 0) return null; // an I/O-space BAR if (low & pci_class.bar_io_space != 0) return null; // an I/O-space BAR
var base: u64 = low & pci_class.bar_memory_base_mask; var base: u64 = low & pci_class.bar_memory_base_mask;
if ((low & pci_class.bar_type_mask) == pci_class.bar_type_64bit) { // 64-bit: high half is the next dword if ((low & pci_class.bar_type_mask) == pci_class.bar_type_64bit) { // 64-bit: high half is the next dword
const high = mmio.read(u32, self.config + pci_class.config_bar0 + (@as(usize, bar) + 1) * 4); const high = mmio.readRegister(u32, self.config + pci_class.config_bar0 + (@as(usize, bar) + 1) * 4);
base |= @as(u64, high) << 32; base |= @as(u64, high) << 32;
} }
@@ -82,7 +82,7 @@ pub const Function = struct {
pub fn capabilities(self: *const Function) CapabilityIterator { pub fn capabilities(self: *const Function) CapabilityIterator {
const present = self.status() & pci_class.status_capabilities_list != 0; const present = self.status() & pci_class.status_capabilities_list != 0;
const first = if (present) const first = if (present)
mmio.read(u8, self.config + pci_class.config_capabilities_pointer) & pci_class.capability_pointer_mask mmio.readRegister(u8, self.config + pci_class.config_capabilities_pointer) & pci_class.capability_pointer_mask
else else
0; 0;
return .{ .config = self.config, .cursor = first }; return .{ .config = self.config, .cursor = first };
@@ -90,7 +90,7 @@ pub const Function = struct {
}; };
/// One capability header. `offset` is the ABSOLUTE virtual address of the header, so the /// One capability header. `offset` is the ABSOLUTE virtual address of the header, so the
/// caller reads its body with `mmio.read(T, cap.offset + n)`. /// caller reads its body with `mmio.readRegister(T, cap.offset + n)`.
pub const Capability = struct { id: u8, offset: usize }; pub const Capability = struct { id: u8, offset: usize };
pub const CapabilityIterator = struct { pub const CapabilityIterator = struct {
@@ -102,8 +102,8 @@ pub const CapabilityIterator = struct {
if (self.cursor == 0 or self.guard >= 48) return null; if (self.cursor == 0 or self.guard >= 48) return null;
self.guard += 1; self.guard += 1;
const at = self.config + self.cursor; const at = self.config + self.cursor;
const id = mmio.read(u8, at + 0); const id = mmio.readRegister(u8, at + 0);
self.cursor = mmio.read(u8, at + 1) & pci_class.capability_pointer_mask; self.cursor = mmio.readRegister(u8, at + 1) & pci_class.capability_pointer_mask;
return .{ .id = id, .offset = at }; return .{ .id = id, .offset = at };
} }
}; };
@@ -162,7 +162,7 @@ const ProducerRing = struct {
// holds it) is written after `parameter`/`status`, with a barrier between. // holds it) is written after `parameter`/`status`, with a barrier between.
slot.parameter = trb.parameter; slot.parameter = trb.parameter;
slot.status = trb.status; slot.status = trb.status;
mmio.wmb(); mmio.writeMemoryBarrier();
slot.control = control; slot.control = control;
const physical = self.region.physical + index * @sizeOf(Trb); const physical = self.region.physical + index * @sizeOf(Trb);
self.enqueue_index += 1; self.enqueue_index += 1;
@@ -628,10 +628,10 @@ pub const Controller = struct {
// interrupter is enabled, so a hot-plug port-change event is silently // interrupter is enabled, so a hot-plug port-change event is silently
// dropped otherwise. Enabling it is harmless to a polling driver. // dropped otherwise. Enabling it is harmless to a polling driver.
write32(self.interrupter(interrupter_management), 1 << 1); // IE write32(self.interrupter(interrupter_management), 1 << 1); // IE
mmio.wmb(); mmio.writeMemoryBarrier();
// Run. // Run.
mmio.wmb(); mmio.writeMemoryBarrier();
write32(self.operational(op_usbcmd), read32(self.operational(op_usbcmd)) | usbcmd_run | usbcmd_interrupter_enable); write32(self.operational(op_usbcmd), read32(self.operational(op_usbcmd)) | usbcmd_run | usbcmd_interrupter_enable);
if (!waitClear(self.operational(op_usbsts), usbsts_halted)) return null; if (!waitClear(self.operational(op_usbsts), usbsts_halted)) return null;
@@ -699,7 +699,7 @@ pub const Controller = struct {
/// address of the enqueued TRB (which the Command Completion Event echoes). /// address of the enqueued TRB (which the Command Completion Event echoes).
fn submitCommand(self: *Controller, trb: Trb) u64 { fn submitCommand(self: *Controller, trb: Trb) u64 {
const physical = self.command_ring.push(trb); const physical = self.command_ring.push(trb);
mmio.wmb(); mmio.writeMemoryBarrier();
self.ringDoorbell(0, 0); // doorbell 0, target 0 = command ring self.ringDoorbell(0, 0); // doorbell 0, target 0 = command ring
return physical; return physical;
} }
@@ -711,7 +711,7 @@ pub const Controller = struct {
const slot = &ring[self.event_ring.dequeue_index]; const slot = &ring[self.event_ring.dequeue_index];
const control = slot.control; const control = slot.control;
if ((control & cycle_bit != 0) == self.event_ring.cycle) { if ((control & cycle_bit != 0) == self.event_ring.cycle) {
mmio.rmb(); mmio.readMemoryBarrier();
const event = Trb{ .parameter = slot.parameter, .status = slot.status, .control = control }; const event = Trb{ .parameter = slot.parameter, .status = slot.status, .control = control };
self.event_ring.dequeue_index += 1; self.event_ring.dequeue_index += 1;
if (self.event_ring.dequeue_index >= trbs_per_ring) { if (self.event_ring.dequeue_index >= trbs_per_ring) {
@@ -1216,7 +1216,7 @@ pub const Controller = struct {
.control = trbControl(.status_stage, status_direction | (1 << 5)), // DIR | IOC .control = trbControl(.status_stage, status_direction | (1 << 5)), // DIR | IOC
}); });
mmio.wmb(); mmio.writeMemoryBarrier();
self.ringDoorbell(device.slot_id, 1); // DCI 1 = EP0 self.ringDoorbell(device.slot_id, 1); // DCI 1 = EP0
const code = self.awaitTransfer(device.slot_id, 1, @intCast(data.len)) orelse return false; const code = self.awaitTransfer(device.slot_id, 1, @intCast(data.len)) orelse return false;
if (code != @intFromEnum(CompletionCode.success) and code != @intFromEnum(CompletionCode.short_packet)) return false; if (code != @intFromEnum(CompletionCode.success) and code != @intFromEnum(CompletionCode.short_packet)) return false;
@@ -1457,7 +1457,7 @@ pub const Controller = struct {
.status = length, .status = length,
.control = trbControl(.normal, (1 << 5)), // IOC .control = trbControl(.normal, (1 << 5)), // IOC
}); });
mmio.wmb(); mmio.writeMemoryBarrier();
const number: u8 = endpoint.address & 0x0F; const number: u8 = endpoint.address & 0x0F;
const direction_in = endpoint.address & 0x80 != 0; const direction_in = endpoint.address & 0x80 != 0;
const dci = doorbellContextIndex(number, direction_in); const dci = doorbellContextIndex(number, direction_in);
@@ -1507,7 +1507,7 @@ pub const Controller = struct {
.status = subscription.max_length, .status = subscription.max_length,
.control = trbControl(.normal, (1 << 5)), // IOC .control = trbControl(.normal, (1 << 5)), // IOC
}); });
mmio.wmb(); mmio.writeMemoryBarrier();
self.ringDoorbell(subscription.slot_id, subscription.dci); self.ringDoorbell(subscription.slot_id, subscription.dci);
} }
+17 -17
View File
@@ -102,17 +102,17 @@ var used_shadow: u16 = 0;
// --- common-config register access (little-endian MMIO at `common_base`) --------------- // --- common-config register access (little-endian MMIO at `common_base`) ---------------
fn cfgRead(comptime T: type, comptime field: []const u8) T { fn cfgRead(comptime T: type, comptime field: []const u8) T {
return mmio.read(T, common_base + @offsetOf(vp.CommonCfg, field)); return mmio.readRegister(T, common_base + @offsetOf(vp.CommonCfg, field));
} }
fn cfgWrite(comptime T: type, comptime field: []const u8, value: T) void { fn cfgWrite(comptime T: type, comptime field: []const u8, value: T) void {
mmio.write(T, common_base + @offsetOf(vp.CommonCfg, field), value); mmio.writeRegister(T, common_base + @offsetOf(vp.CommonCfg, field), value);
} }
/// Write a 64-bit common-config register as two 32-bit halves (low then high) — the widest /// Write a 64-bit common-config register as two 32-bit halves (low then high) — the widest
/// access every virtio-pci host is required to accept for the queue-address registers. /// access every virtio-pci host is required to accept for the queue-address registers.
fn cfgWrite64(comptime field: []const u8, value: u64) void { fn cfgWrite64(comptime field: []const u8, value: u64) void {
const at = common_base + @offsetOf(vp.CommonCfg, field); const at = common_base + @offsetOf(vp.CommonCfg, field);
mmio.write(u32, at, @truncate(value)); mmio.writeRegister(u32, at, @truncate(value));
mmio.write(u32, at + 4, @truncate(value >> 32)); mmio.writeRegister(u32, at + 4, @truncate(value >> 32));
} }
fn orStatus(bit: u8) void { fn orStatus(bit: u8) void {
cfgWrite(u8, "device_status", cfgRead(u8, "device_status") | bit); cfgWrite(u8, "device_status", cfgRead(u8, "device_status") | bit);
@@ -140,12 +140,12 @@ fn submit(request_len: usize, response_len: usize) bool {
const avail_ring: [*]u16 = @ptrFromInt(ring.virtual + avail_offset + 4); const avail_ring: [*]u16 = @ptrFromInt(ring.virtual + avail_offset + 4);
avail_ring[avail_shadow % queue_size] = 0; // head of the chain is descriptor 0 avail_ring[avail_shadow % queue_size] = 0; // head of the chain is descriptor 0
mmio.wmb(); mmio.writeMemoryBarrier();
avail_shadow +%= 1; avail_shadow +%= 1;
mmio.write(u16, ring.virtual + avail_offset + 2, avail_shadow); // avail.idx mmio.writeRegister(u16, ring.virtual + avail_offset + 2, avail_shadow); // avail.idx
mmio.wmb(); mmio.writeMemoryBarrier();
mmio.write(u16, notify_addr, 0); // ring the control queue's doorbell mmio.writeRegister(u16, notify_addr, 0); // ring the control queue's doorbell
return waitUsed(); return waitUsed();
} }
@@ -155,8 +155,8 @@ fn submit(request_len: usize, response_len: usize) bool {
fn waitUsed() bool { fn waitUsed() bool {
var tries: u32 = 0; var tries: u32 = 0;
while (tries < 2000) : (tries += 1) { while (tries < 2000) : (tries += 1) {
mmio.rmb(); mmio.readMemoryBarrier();
const idx = mmio.read(u16, ring.virtual + used_offset + 2); // used.idx const idx = mmio.readRegister(u16, ring.virtual + used_offset + 2); // used.idx
if (idx != used_shadow) { if (idx != used_shadow) {
used_shadow = idx; used_shadow = idx;
return true; return true;
@@ -231,16 +231,16 @@ fn initialise(endpoint: ipc.Handle) bool {
var caps = function.capabilities(); var caps = function.capabilities();
while (caps.next()) |cap| { while (caps.next()) |cap| {
if (cap.id != vp.pci_cap_vendor) continue; if (cap.id != vp.pci_cap_vendor) continue;
const cfg_type = mmio.read(u8, cap.offset + 3); const cfg_type = mmio.readRegister(u8, cap.offset + 3);
if (cfg_type != vp.cfg_common and cfg_type != vp.cfg_notify) continue; if (cfg_type != vp.cfg_common and cfg_type != vp.cfg_notify) continue;
const bar = mmio.read(u8, cap.offset + 4); const bar = mmio.readRegister(u8, cap.offset + 4);
const offset = mmio.read(u32, cap.offset + 8); const offset = mmio.readRegister(u32, cap.offset + 8);
if (function.mapBar(bar)) |bar_base| { if (function.mapBar(bar)) |bar_base| {
if (cfg_type == vp.cfg_common) { if (cfg_type == vp.cfg_common) {
common_base = bar_base + offset; common_base = bar_base + offset;
} else { } else {
notify_base = bar_base + offset; notify_base = bar_base + offset;
notify_multiplier = mmio.read(u32, cap.offset + 16); // virtio_pci_notify_cap tail notify_multiplier = mmio.readRegister(u32, cap.offset + 16); // virtio_pci_notify_cap tail
} }
} }
} }
@@ -290,7 +290,7 @@ fn initialise(endpoint: ipc.Handle) bool {
std.log.info("command-buffer allocation failed", .{}); std.log.info("command-buffer allocation failed", .{});
return false; return false;
}; };
mmio.write(u16, ring.virtual + avail_offset, 1); // VIRTQ_AVAIL_F_NO_INTERRUPT: we poll mmio.writeRegister(u16, ring.virtual + avail_offset, 1); // VIRTQ_AVAIL_F_NO_INTERRUPT: we poll
cfgWrite(u16, "queue_size", queue_size); cfgWrite(u16, "queue_size", queue_size);
cfgWrite64("queue_desc", ring.physical + desc_offset); cfgWrite64("queue_desc", ring.physical + desc_offset);
cfgWrite64("queue_driver", ring.physical + avail_offset); cfgWrite64("queue_driver", ring.physical + avail_offset);
@@ -372,7 +372,7 @@ fn initialise(endpoint: ipc.Handle) bool {
} }
// The scanout surface is CPU-visible RAM: read the pattern back to prove the mapping, // The scanout surface is CPU-visible RAM: read the pattern back to prove the mapping,
// which together with the flush ack above is the automated stand-in for "it's on screen". // which together with the flush ack above is the automated stand-in for "it's on screen".
mmio.rmb(); mmio.readMemoryBarrier();
if (pixels[0] != testPixel(0) or pixels[pixel_count / 2] != testPixel(@intCast(pixel_count / 2))) { if (pixels[0] != testPixel(0) or pixels[pixel_count / 2] != testPixel(@intCast(pixel_count / 2))) {
std.log.info("pixel read-back mismatch", .{}); std.log.info("pixel read-back mismatch", .{});
return false; return false;
@@ -435,7 +435,7 @@ fn readEdid() void {
/// the panel. Reused by the V3 self-test and by every compositor present over `.scanout`. V4 /// the panel. Reused by the V3 self-test and by every compositor present over `.scanout`. V4
/// presents the full surface; the damage-rect fast path is a later refinement. /// presents the full surface; the damage-rect fast path is a later refinement.
fn presentFull() bool { fn presentFull() bool {
mmio.wmb(); // the surface writes must be visible before the device transfers them mmio.writeMemoryBarrier(); // the surface writes must be visible before the device transfers them
{ {
// Transfer the current-mode rectangle from the guest backing to the host resource. The // Transfer the current-mode rectangle from the guest backing to the host resource. The
// device uses the resource's (max) width as the row stride, so the top-left rect at // device uses the resource's (max) width as the row stride, so the top-left rect at