reorg: move mmio into library/device and spell out its API
mmio is device-driver code, so it joins the other domains under
library/device/mmio/ (module name "mmio" unchanged — a pure relocation, only
the build paths move). And its abbreviated function names are spelled out per
docs/coding-standards.md:
read -> readRegister mb -> memoryBarrier
write -> writeRegister rmb -> readMemoryBarrier
wmb -> writeMemoryBarrier
All call sites updated (virtio-gpu, usb-xhci-library, pci.Function); the two
display-driver placeholders import mmio but use nothing, so they're untouched.
Docs (driver-model graph, README layout, drivers.md, the FHS note) follow the
new path and names.
zig build + test green; virtio-gpu, display-native, display-reattach, usb-hid,
usb-hub, usb-storage, pci-scan pass.
This commit is contained in:
@@ -428,9 +428,9 @@ pub fn build(b: *std.Build) void {
|
|||||||
|
|
||||||
// Typed volatile MMIO register access + memory-ordering barriers, for drivers on
|
// Typed volatile MMIO register access + memory-ordering barriers, for drivers on
|
||||||
// top of an mmio_map grant. Depends only on `builtin` (arch-conditional barriers);
|
// top of an mmio_map grant. Depends only on `builtin` (arch-conditional barriers);
|
||||||
// no target set, so it inherits each driver's. See library/mmio/mmio.zig.
|
// no target set, so it inherits each driver's. See library/device/mmio/mmio.zig.
|
||||||
const mmio_module = b.addModule("mmio", .{
|
const mmio_module = b.addModule("mmio", .{
|
||||||
.root_source_file = b.path("library/mmio/mmio.zig"),
|
.root_source_file = b.path("library/device/mmio/mmio.zig"),
|
||||||
});
|
});
|
||||||
|
|
||||||
// A device driver's view of its claimed PCI function: config-space header fields, BAR
|
// A device driver's view of its claimed PCI function: config-space header fields, BAR
|
||||||
@@ -969,7 +969,7 @@ pub fn build(b: *std.Build) void {
|
|||||||
"library/device/acpi/aml/aml.zig", // AML parse + interpret, incl. Notify dispatch (M21)
|
"library/device/acpi/aml/aml.zig", // AML parse + interpret, incl. Notify dispatch (M21)
|
||||||
"library/device/usb/usb-abi.zig", // wire sizes + bit packings + set-up packet encodings
|
"library/device/usb/usb-abi.zig", // wire sizes + bit packings + set-up packet encodings
|
||||||
"library/device/usb/usb-ids.zig", // class/subclass/protocol code assignments
|
"library/device/usb/usb-ids.zig", // class/subclass/protocol code assignments
|
||||||
"library/mmio/mmio.zig", // barriers assemble + registers round-trip
|
"library/device/mmio/mmio.zig", // barriers assemble + registers round-trip
|
||||||
"system/drivers/ps2-bus/scancode.zig", // set-2 decode + keyboard state machine
|
"system/drivers/ps2-bus/scancode.zig", // set-2 decode + keyboard state machine
|
||||||
"system/drivers/ps2-bus/mouse-packet.zig", // 3-byte mouse packet assembly
|
"system/drivers/ps2-bus/mouse-packet.zig", // 3-byte mouse packet assembly
|
||||||
"system/drivers/usb-hid/hid-report.zig", // HID boot-report keyboard/mouse decode
|
"system/drivers/usb-hid/hid-report.zig", // HID boot-report keyboard/mouse decode
|
||||||
|
|||||||
+2
-3
@@ -245,10 +245,9 @@ system/ → /system danos's own internals (the self-representation)
|
|||||||
fat.zig, engine.zig, on-disk.zig)
|
fat.zig, engine.zig, on-disk.zig)
|
||||||
library/ → /lib libraries, one sub-directory each
|
library/ → /lib libraries, one sub-directory each
|
||||||
runtime/ the danos-native runtime + file API (fs) — the stable application ABI
|
runtime/ the danos-native runtime + file API (fs) — the stable application ABI
|
||||||
mmio/ volatile register access + memory barriers
|
device/ device code by domain — mmio/ model/ pci/ usb/ acpi/ — each a
|
||||||
device/ device code by domain — model/ pci/ usb/ acpi/ — each a
|
|
||||||
shareable data module (device-abi, pci-class, usb-abi/ids,
|
shareable data module (device-abi, pci-class, usb-abi/ids,
|
||||||
acpi-ids) plus a logic module (pci, usb, aml)
|
acpi-ids) plus a logic module (mmio, pci, usb, aml)
|
||||||
protocol/ driver↔service wire contracts (vfs block display scanout input
|
protocol/ driver↔service wire contracts (vfs block display scanout input
|
||||||
power device-manager usb-transfer), one module per directory
|
power device-manager usb-transfer), one module per directory
|
||||||
boot/ → /boot the loaders
|
boot/ → /boot the loaders
|
||||||
|
|||||||
@@ -87,7 +87,7 @@ A block driver is now **writable, but not yet memory-safe.** Every storage contr
|
|||||||
worth naming is a bus master: it is programmed by handing it the physical address of a
|
worth naming is a bus master: it is programmed by handing it the physical address of a
|
||||||
descriptor ring and left to read and write memory on its own. That ring is exactly what
|
descriptor ring and left to read and write memory on its own. That ring is exactly what
|
||||||
**`dma_alloc`** now provides — physically contiguous, pinned, uncacheable, with its
|
**`dma_alloc`** now provides — physically contiguous, pinned, uncacheable, with its
|
||||||
physical address disclosed — and **`/lib/mmio`**'s barriers order the descriptor writes
|
physical address disclosed — and **`/lib/device/mmio`**'s barriers order the descriptor writes
|
||||||
against the doorbell, and **`msi_bind`** delivers completions. So an AHCI or NVMe driver
|
against the doorbell, and **`msi_bind`** delivers completions. So an AHCI or NVMe driver
|
||||||
can be written today (the M14/M15 work in [driver-model.md](driver-model.md); the earlier
|
can be written today (the M14/M15 work in [driver-model.md](driver-model.md); the earlier
|
||||||
"cannot host a block driver at all" is no longer true).
|
"cannot host a block driver at all" is no longer true).
|
||||||
|
|||||||
+13
-13
@@ -106,9 +106,9 @@ module outlived it, which is rather the point.) The pattern generalises directly
|
|||||||
```
|
```
|
||||||
library/
|
library/
|
||||||
runtime/ module "runtime" — syscalls, ipc, lifecycle, memory, threads, log, fs
|
runtime/ module "runtime" — syscalls, ipc, lifecycle, memory, threads, log, fs
|
||||||
mmio/ module "mmio" — volatile register access + barriers [M14]
|
|
||||||
device/ device code grouped by domain; each domain splits into a shareable
|
device/ device code grouped by domain; each domain splits into a shareable
|
||||||
data module (enums/wire types, std-only) and a logic module (mmio/IPC)
|
data module (enums/wire types, std-only) and a logic module (mmio/IPC)
|
||||||
|
mmio/ module "mmio" — typed volatile register access + barriers [M14]
|
||||||
model/ module "device-abi" — DeviceDescriptor, DeviceClass, ResourceKind
|
model/ module "device-abi" — DeviceDescriptor, DeviceClass, ResourceKind
|
||||||
pci/ "pci-class" (data) + "pci" — config/BAR/capability walk (Function)
|
pci/ "pci-class" (data) + "pci" — config/BAR/capability walk (Function)
|
||||||
usb/ "usb-abi" + "usb-ids" (data) + "usb" — descriptors, control/interrupt/bulk client
|
usb/ "usb-abi" + "usb-ids" (data) + "usb" — descriptors, control/interrupt/bulk client
|
||||||
@@ -160,12 +160,12 @@ class driver, the device manager, or the kernel may share them freely.
|
|||||||
`callCap` and `replyWait(..., send_cap)`, and class drivers consume them now: the
|
`callCap` and `replyWait(..., send_cap)`, and class drivers consume them now: the
|
||||||
PS/2 keyboard and mouse drivers attach to ps2-bus this way, and `runtime.usb` /
|
PS/2 keyboard and mouse drivers attach to ps2-bus this way, and `runtime.usb` /
|
||||||
`runtime.input` open their per-device and subscription channels with `callCap`.
|
`runtime.input` open their per-device and subscription channels with `callCap`.
|
||||||
- **M14** — DMA memory + the memory-ordering layer. `/lib/mmio` gives drivers typed
|
- **M14** — DMA memory + the memory-ordering layer. `/lib/device/mmio` gives drivers typed
|
||||||
volatile access and `mb`/`rmb`/`wmb` (per-arch); `dma_alloc`/`dma_free` grant
|
volatile access and `memoryBarrier`/`readMemoryBarrier`/`writeMemoryBarrier` (per-arch); `dma_alloc`/`dma_free` grant
|
||||||
physically-contiguous, pinned, uncacheable, reclaim-on-teardown buffers with the
|
physically-contiguous, pinned, uncacheable, reclaim-on-teardown buffers with the
|
||||||
physical address exposed (`pmm.allocContiguous`, a DMA arena, `mapUserDmaInto`).
|
physical address exposed (`pmm.allocContiguous`, a DMA arena, `mapUserDmaInto`).
|
||||||
`dma_below_4g` caps the address for legacy engines; `dma_write_combining` is accepted
|
`dma_below_4g` caps the address for legacy engines; `dma_write_combining` is accepted
|
||||||
but falls back to coherent until PAT is programmed. The bus drivers use `/lib/mmio`,
|
but falls back to coherent until PAT is programmed. The bus drivers use `/lib/device/mmio`,
|
||||||
and `dma_alloc` has real consumers now: the xHCI driver's rings and contexts,
|
and `dma_alloc` has real consumers now: the xHCI driver's rings and contexts,
|
||||||
usb-storage's command/status wrappers, virtio-gpu's virtqueue, and the fat
|
usb-storage's command/status wrappers, virtio-gpu's virtqueue, and the fat
|
||||||
service's bounce buffer.
|
service's bounce buffer.
|
||||||
@@ -252,7 +252,7 @@ const dev_ep = ipc.callCap(h, // ... mint a per-device endpoint,
|
|||||||
|
|
||||||
## M14 — DMA memory and the memory-ordering contract, for HCDs ✅ done
|
## M14 — DMA memory and the memory-ordering contract, for HCDs ✅ done
|
||||||
|
|
||||||
*Implemented: `/lib/mmio` (typed volatile access + `mb`/`rmb`/`wmb`, per-arch) and
|
*Implemented: `/lib/device/mmio` (typed volatile access + `memoryBarrier`/`readMemoryBarrier`/`writeMemoryBarrier`, per-arch) and
|
||||||
`dma_alloc`/`dma_free` (contiguous, pinned, uncacheable, reclaim-on-teardown, physical
|
`dma_alloc`/`dma_free` (contiguous, pinned, uncacheable, reclaim-on-teardown, physical
|
||||||
address exposed). `dma_write_combining` still falls back to coherent — real WC needs
|
address exposed). `dma_write_combining` still falls back to coherent — real WC needs
|
||||||
PAT, a small follow-up. The rest of this section is the original design note.*
|
PAT, a small follow-up. The rest of this section is the original design note.*
|
||||||
@@ -294,23 +294,23 @@ doorbell.* = i; // volatile store to UC MMIO
|
|||||||
// nothing stops the compiler reordering these; the device reads a stale descriptor
|
// nothing stops the compiler reordering these; the device reads a stale descriptor
|
||||||
```
|
```
|
||||||
|
|
||||||
So the rules, which belong in `library/mmio.zig` and behind `arch`:
|
So the rules, which belong in `library/device/mmio/mmio.zig` and behind `arch`:
|
||||||
|
|
||||||
| Situation | Required |
|
| Situation | Required |
|
||||||
|---|---|
|
|---|---|
|
||||||
| MMIO register read/write | `mmio.read` / `mmio.write` (volatile) |
|
| MMIO register read/write | `mmio.read` / `mmio.write` (volatile) |
|
||||||
| Fill DMA descriptor, then ring doorbell | `wmb()` between them |
|
| Fill DMA descriptor, then ring doorbell | `writeMemoryBarrier()` between them |
|
||||||
| Woken by IRQ, then read what the device wrote | `rmb()` before the read |
|
| Woken by IRQ, then read what the device wrote | `readMemoryBarrier()` before the read |
|
||||||
| MMIO write that must complete before the next read | `mb()` |
|
| MMIO write that must complete before the next read | `memoryBarrier()` |
|
||||||
|
|
||||||
And the per-arch lowering — the reason this must be an `arch` primitive and not a
|
And the per-arch lowering — the reason this must be an `arch` primitive and not a
|
||||||
sprinkling of `asm volatile`:
|
sprinkling of `asm volatile`:
|
||||||
|
|
||||||
| | x86_64 | aarch64 |
|
| | x86_64 | aarch64 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `mb()` | `mfence` | `dsb sy` |
|
| `memoryBarrier()` | `mfence` | `dsb sy` |
|
||||||
| `rmb()` | `lfence` | `dsb ld` |
|
| `readMemoryBarrier()` | `lfence` | `dsb ld` |
|
||||||
| `wmb()` | `sfence` | `dsb st` |
|
| `writeMemoryBarrier()` | `sfence` | `dsb st` |
|
||||||
| DMA cache coherency | coherent; nothing to do | **not guaranteed**; needs non-cacheable buffers or cache maintenance |
|
| DMA cache coherency | coherent; nothing to do | **not guaranteed**; needs non-cacheable buffers or cache maintenance |
|
||||||
|
|
||||||
x86 is forgiving here — TSO plus strong-uncacheable MMIO means you usually get away
|
x86 is forgiving here — TSO plus strong-uncacheable MMIO means you usually get away
|
||||||
@@ -318,7 +318,7 @@ with a compiler barrier alone. ARM is not, and [vision.md](vision.md) makes ARM
|
|||||||
condition. Build the abstraction while there is one caller to fix.
|
condition. Build the abstraction while there is one caller to fix.
|
||||||
|
|
||||||
(Zig note: `@fence` was **removed in 0.16**. Use `@atomicRmw(..., .seq_cst)` for a full
|
(Zig note: `@fence` was **removed in 0.16**. Use `@atomicRmw(..., .seq_cst)` for a full
|
||||||
barrier, or per-arch inline asm — which is what `library/mmio.zig` should hide.)
|
barrier, or per-arch inline asm — which is what `library/device/mmio/mmio.zig` should hide.)
|
||||||
|
|
||||||
## M15 — interrupts for PCI devices ✅ done (MSI)
|
## M15 — interrupts for PCI devices ✅ done (MSI)
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -290,8 +290,8 @@ Several things this list used to warn about are now available (see
|
|||||||
[driver-model.md](driver-model.md)): **port I/O** (`io_read`/`io_write`, claim-gated by
|
[driver-model.md](driver-model.md)): **port I/O** (`io_read`/`io_write`, claim-gated by
|
||||||
the device's `io_port` resource — direct ring-3 `in`/`out` is still a #GP, so a PS/2 or
|
the device's `io_port` resource — direct ring-3 `in`/`out` is still a #GP, so a PS/2 or
|
||||||
16550 driver goes through these), **DMA memory** (`dma_alloc`: contiguous, pinned,
|
16550 driver goes through these), **DMA memory** (`dma_alloc`: contiguous, pinned,
|
||||||
uncacheable, physical address exposed), **memory barriers** (`library/mmio`'s
|
uncacheable, physical address exposed), **memory barriers** (`library/device/mmio`'s
|
||||||
`mb`/`rmb`/`wmb`, imported as the `mmio` module), **fault isolation** (a ring-3 fault kills only the faulting
|
`memoryBarrier`/`readMemoryBarrier`/`writeMemoryBarrier`, imported as the `mmio` module), **fault isolation** (a ring-3 fault kills only the faulting
|
||||||
process — `killCurrentProcess` — and the machine keeps running,
|
process — `killCurrentProcess` — and the machine keeps running,
|
||||||
[resilience](resilience.md)), and **reclaim + restart on death** (every path out of a
|
[resilience](resilience.md)), and **reclaim + restart on death** (every path out of a
|
||||||
process releases its claims and IRQ/MSI bindings — `releaseAllOwnedBy`,
|
process releases its claims and IRQ/MSI bindings — `releaseAllOwnedBy`,
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
//! /lib/mmio — typed volatile MMIO register access, plus the memory-ordering
|
//! /lib/device/mmio — typed volatile MMIO register access, plus the memory-ordering
|
||||||
//! barriers a device driver needs. Used by drivers on top of an `mmio_map` grant.
|
//! barriers a device driver needs. Used by drivers on top of an `mmio_map` grant.
|
||||||
//!
|
//!
|
||||||
//! **`volatile` is not a barrier.** In Zig it means only: don't elide this access, and
|
//! **`volatile` is not a barrier.** In Zig it means only: don't elide this access, and
|
||||||
@@ -11,13 +11,13 @@
|
|||||||
//! doorbell.* = i; // volatile store to UC MMIO
|
//! doorbell.* = i; // volatile store to UC MMIO
|
||||||
//! // nothing orders these; the device can read a stale descriptor
|
//! // nothing orders these; the device can read a stale descriptor
|
||||||
//!
|
//!
|
||||||
//! Put a `wmb()` between them. The barriers lower per-architecture — which is the whole
|
//! Put a `writeMemoryBarrier()` between them. The barriers lower per-architecture — which
|
||||||
//! reason they are a named primitive and not scattered `asm volatile`:
|
//! is the whole reason they are a named primitive and not scattered `asm volatile`:
|
||||||
//!
|
//!
|
||||||
//! x86_64 aarch64
|
//! x86_64 aarch64
|
||||||
//! mb() mfence dsb sy
|
//! memoryBarrier() mfence dsb sy
|
||||||
//! rmb() lfence dsb ld
|
//! readMemoryBarrier() lfence dsb ld
|
||||||
//! wmb() sfence dsb st
|
//! writeMemoryBarrier() sfence dsb st
|
||||||
//!
|
//!
|
||||||
//! x86 is forgiving (TSO + strong-uncacheable MMIO), so a compiler barrier usually
|
//! x86 is forgiving (TSO + strong-uncacheable MMIO), so a compiler barrier usually
|
||||||
//! suffices; ARM is not, and ARM is the win condition (docs/vision.md) — so the
|
//! suffices; ARM is not, and ARM is the win condition (docs/vision.md) — so the
|
||||||
@@ -29,52 +29,52 @@ const builtin = @import("builtin");
|
|||||||
/// Read a register of type `T` at absolute virtual address `addr` — a location inside
|
/// Read a register of type `T` at absolute virtual address `addr` — a location inside
|
||||||
/// a device's `mmio_map` grant. `volatile`: never elided, never reordered against
|
/// a device's `mmio_map` grant. `volatile`: never elided, never reordered against
|
||||||
/// another volatile access.
|
/// another volatile access.
|
||||||
pub inline fn read(comptime T: type, addr: usize) T {
|
pub inline fn readRegister(comptime T: type, addr: usize) T {
|
||||||
return @as(*const volatile T, @ptrFromInt(addr)).*;
|
return @as(*const volatile T, @ptrFromInt(addr)).*;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Write `value` of type `T` to the register at absolute virtual address `addr`.
|
/// Write `value` of type `T` to the register at absolute virtual address `addr`.
|
||||||
pub inline fn write(comptime T: type, addr: usize, value: T) void {
|
pub inline fn writeRegister(comptime T: type, addr: usize, value: T) void {
|
||||||
@as(*volatile T, @ptrFromInt(addr)).* = value;
|
@as(*volatile T, @ptrFromInt(addr)).* = value;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Full barrier: all loads and stores before it are globally visible before any after
|
/// Full memory barrier: all loads and stores before it are globally visible before any
|
||||||
/// it. Use when an MMIO write must complete before a following read.
|
/// after it. Use when an MMIO write must complete before a following read.
|
||||||
pub inline fn mb() void {
|
pub inline fn memoryBarrier() void {
|
||||||
switch (builtin.target.cpu.arch) {
|
switch (builtin.target.cpu.arch) {
|
||||||
.x86_64 => asm volatile ("mfence" ::: .{ .memory = true }),
|
.x86_64 => asm volatile ("mfence" ::: .{ .memory = true }),
|
||||||
.aarch64 => asm volatile ("dsb sy" ::: .{ .memory = true }),
|
.aarch64 => asm volatile ("dsb sy" ::: .{ .memory = true }),
|
||||||
else => @compileError("mmio.mb: unsupported architecture"),
|
else => @compileError("mmio.memoryBarrier: unsupported architecture"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Read barrier: loads before it complete before loads after it. Use after an IRQ
|
/// Read memory barrier: loads before it complete before loads after it. Use after an IRQ
|
||||||
/// wake, before reading what the device wrote to shared memory.
|
/// wake, before reading what the device wrote to shared memory.
|
||||||
pub inline fn rmb() void {
|
pub inline fn readMemoryBarrier() void {
|
||||||
switch (builtin.target.cpu.arch) {
|
switch (builtin.target.cpu.arch) {
|
||||||
.x86_64 => asm volatile ("lfence" ::: .{ .memory = true }),
|
.x86_64 => asm volatile ("lfence" ::: .{ .memory = true }),
|
||||||
.aarch64 => asm volatile ("dsb ld" ::: .{ .memory = true }),
|
.aarch64 => asm volatile ("dsb ld" ::: .{ .memory = true }),
|
||||||
else => @compileError("mmio.rmb: unsupported architecture"),
|
else => @compileError("mmio.readMemoryBarrier: unsupported architecture"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Write barrier: stores before it become visible before stores after it. Use between
|
/// Write memory barrier: stores before it become visible before stores after it. Use
|
||||||
/// filling a DMA descriptor in RAM and ringing the device's doorbell.
|
/// between filling a DMA descriptor in RAM and ringing the device's doorbell.
|
||||||
pub inline fn wmb() void {
|
pub inline fn writeMemoryBarrier() void {
|
||||||
switch (builtin.target.cpu.arch) {
|
switch (builtin.target.cpu.arch) {
|
||||||
.x86_64 => asm volatile ("sfence" ::: .{ .memory = true }),
|
.x86_64 => asm volatile ("sfence" ::: .{ .memory = true }),
|
||||||
.aarch64 => asm volatile ("dsb st" ::: .{ .memory = true }),
|
.aarch64 => asm volatile ("dsb st" ::: .{ .memory = true }),
|
||||||
else => @compileError("mmio.wmb: unsupported architecture"),
|
else => @compileError("mmio.writeMemoryBarrier: unsupported architecture"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
test "barriers emit and registers round-trip through a RAM cell" {
|
test "barriers emit and registers round-trip through a RAM cell" {
|
||||||
// The barriers must at least assemble for the host arch; ordering can't be unit
|
// The barriers must at least assemble for the host arch; ordering can't be unit
|
||||||
// tested, but a missing/mistyped mnemonic is caught here.
|
// tested, but a missing/mistyped mnemonic is caught here.
|
||||||
wmb();
|
writeMemoryBarrier();
|
||||||
rmb();
|
readMemoryBarrier();
|
||||||
mb();
|
memoryBarrier();
|
||||||
var cell: u64 = 0;
|
var cell: u64 = 0;
|
||||||
write(u64, @intFromPtr(&cell), 0xDEAD_BEEF);
|
writeRegister(u64, @intFromPtr(&cell), 0xDEAD_BEEF);
|
||||||
try @import("std").testing.expectEqual(@as(u64, 0xDEAD_BEEF), read(u64, @intFromPtr(&cell)));
|
try @import("std").testing.expectEqual(@as(u64, 0xDEAD_BEEF), readRegister(u64, @intFromPtr(&cell)));
|
||||||
}
|
}
|
||||||
+11
-11
@@ -32,23 +32,23 @@ pub const Function = struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub fn vendorId(self: *const Function) u16 {
|
pub fn vendorId(self: *const Function) u16 {
|
||||||
return mmio.read(u16, self.config + pci_class.config_vendor_id);
|
return mmio.readRegister(u16, self.config + pci_class.config_vendor_id);
|
||||||
}
|
}
|
||||||
pub fn deviceId(self: *const Function) u16 {
|
pub fn deviceId(self: *const Function) u16 {
|
||||||
return mmio.read(u16, self.config + pci_class.config_device_id);
|
return mmio.readRegister(u16, self.config + pci_class.config_device_id);
|
||||||
}
|
}
|
||||||
pub fn command(self: *const Function) u16 {
|
pub fn command(self: *const Function) u16 {
|
||||||
return mmio.read(u16, self.config + pci_class.config_command);
|
return mmio.readRegister(u16, self.config + pci_class.config_command);
|
||||||
}
|
}
|
||||||
pub fn status(self: *const Function) u16 {
|
pub fn status(self: *const Function) u16 {
|
||||||
return mmio.read(u16, self.config + pci_class.config_status);
|
return mmio.readRegister(u16, self.config + pci_class.config_status);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Set Memory-Space + Bus-Master enable in the command register. Firmware often leaves
|
/// Set Memory-Space + Bus-Master enable in the command register. Firmware often leaves
|
||||||
/// a secondary display's decode off; a bus-mastering device must enable both.
|
/// a secondary display's decode off; a bus-mastering device must enable both.
|
||||||
pub fn enableMemoryAndBusMaster(self: *const Function) void {
|
pub fn enableMemoryAndBusMaster(self: *const Function) void {
|
||||||
const at = self.config + pci_class.config_command;
|
const at = self.config + pci_class.config_command;
|
||||||
mmio.write(u16, at, mmio.read(u16, at) | pci_class.command_memory_and_bus_master);
|
mmio.writeRegister(u16, at, mmio.readRegister(u16, at) | pci_class.command_memory_and_bus_master);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Decode BAR `bar` (0..5) and map it: read the BAR register, reject I/O-space BARs,
|
/// Decode BAR `bar` (0..5) and map it: read the BAR register, reject I/O-space BARs,
|
||||||
@@ -60,11 +60,11 @@ pub const Function = struct {
|
|||||||
if (bar >= 6) return null;
|
if (bar >= 6) return null;
|
||||||
if (self.bar_virtual[bar] != 0) return self.bar_virtual[bar];
|
if (self.bar_virtual[bar] != 0) return self.bar_virtual[bar];
|
||||||
|
|
||||||
const low = mmio.read(u32, self.config + pci_class.config_bar0 + @as(usize, bar) * 4);
|
const low = mmio.readRegister(u32, self.config + pci_class.config_bar0 + @as(usize, bar) * 4);
|
||||||
if (low & pci_class.bar_io_space != 0) return null; // an I/O-space BAR
|
if (low & pci_class.bar_io_space != 0) return null; // an I/O-space BAR
|
||||||
var base: u64 = low & pci_class.bar_memory_base_mask;
|
var base: u64 = low & pci_class.bar_memory_base_mask;
|
||||||
if ((low & pci_class.bar_type_mask) == pci_class.bar_type_64bit) { // 64-bit: high half is the next dword
|
if ((low & pci_class.bar_type_mask) == pci_class.bar_type_64bit) { // 64-bit: high half is the next dword
|
||||||
const high = mmio.read(u32, self.config + pci_class.config_bar0 + (@as(usize, bar) + 1) * 4);
|
const high = mmio.readRegister(u32, self.config + pci_class.config_bar0 + (@as(usize, bar) + 1) * 4);
|
||||||
base |= @as(u64, high) << 32;
|
base |= @as(u64, high) << 32;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -82,7 +82,7 @@ pub const Function = struct {
|
|||||||
pub fn capabilities(self: *const Function) CapabilityIterator {
|
pub fn capabilities(self: *const Function) CapabilityIterator {
|
||||||
const present = self.status() & pci_class.status_capabilities_list != 0;
|
const present = self.status() & pci_class.status_capabilities_list != 0;
|
||||||
const first = if (present)
|
const first = if (present)
|
||||||
mmio.read(u8, self.config + pci_class.config_capabilities_pointer) & pci_class.capability_pointer_mask
|
mmio.readRegister(u8, self.config + pci_class.config_capabilities_pointer) & pci_class.capability_pointer_mask
|
||||||
else
|
else
|
||||||
0;
|
0;
|
||||||
return .{ .config = self.config, .cursor = first };
|
return .{ .config = self.config, .cursor = first };
|
||||||
@@ -90,7 +90,7 @@ pub const Function = struct {
|
|||||||
};
|
};
|
||||||
|
|
||||||
/// One capability header. `offset` is the ABSOLUTE virtual address of the header, so the
|
/// One capability header. `offset` is the ABSOLUTE virtual address of the header, so the
|
||||||
/// caller reads its body with `mmio.read(T, cap.offset + n)`.
|
/// caller reads its body with `mmio.readRegister(T, cap.offset + n)`.
|
||||||
pub const Capability = struct { id: u8, offset: usize };
|
pub const Capability = struct { id: u8, offset: usize };
|
||||||
|
|
||||||
pub const CapabilityIterator = struct {
|
pub const CapabilityIterator = struct {
|
||||||
@@ -102,8 +102,8 @@ pub const CapabilityIterator = struct {
|
|||||||
if (self.cursor == 0 or self.guard >= 48) return null;
|
if (self.cursor == 0 or self.guard >= 48) return null;
|
||||||
self.guard += 1;
|
self.guard += 1;
|
||||||
const at = self.config + self.cursor;
|
const at = self.config + self.cursor;
|
||||||
const id = mmio.read(u8, at + 0);
|
const id = mmio.readRegister(u8, at + 0);
|
||||||
self.cursor = mmio.read(u8, at + 1) & pci_class.capability_pointer_mask;
|
self.cursor = mmio.readRegister(u8, at + 1) & pci_class.capability_pointer_mask;
|
||||||
return .{ .id = id, .offset = at };
|
return .{ .id = id, .offset = at };
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -162,7 +162,7 @@ const ProducerRing = struct {
|
|||||||
// holds it) is written after `parameter`/`status`, with a barrier between.
|
// holds it) is written after `parameter`/`status`, with a barrier between.
|
||||||
slot.parameter = trb.parameter;
|
slot.parameter = trb.parameter;
|
||||||
slot.status = trb.status;
|
slot.status = trb.status;
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
slot.control = control;
|
slot.control = control;
|
||||||
const physical = self.region.physical + index * @sizeOf(Trb);
|
const physical = self.region.physical + index * @sizeOf(Trb);
|
||||||
self.enqueue_index += 1;
|
self.enqueue_index += 1;
|
||||||
@@ -628,10 +628,10 @@ pub const Controller = struct {
|
|||||||
// interrupter is enabled, so a hot-plug port-change event is silently
|
// interrupter is enabled, so a hot-plug port-change event is silently
|
||||||
// dropped otherwise. Enabling it is harmless to a polling driver.
|
// dropped otherwise. Enabling it is harmless to a polling driver.
|
||||||
write32(self.interrupter(interrupter_management), 1 << 1); // IE
|
write32(self.interrupter(interrupter_management), 1 << 1); // IE
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
|
|
||||||
// Run.
|
// Run.
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
write32(self.operational(op_usbcmd), read32(self.operational(op_usbcmd)) | usbcmd_run | usbcmd_interrupter_enable);
|
write32(self.operational(op_usbcmd), read32(self.operational(op_usbcmd)) | usbcmd_run | usbcmd_interrupter_enable);
|
||||||
if (!waitClear(self.operational(op_usbsts), usbsts_halted)) return null;
|
if (!waitClear(self.operational(op_usbsts), usbsts_halted)) return null;
|
||||||
|
|
||||||
@@ -699,7 +699,7 @@ pub const Controller = struct {
|
|||||||
/// address of the enqueued TRB (which the Command Completion Event echoes).
|
/// address of the enqueued TRB (which the Command Completion Event echoes).
|
||||||
fn submitCommand(self: *Controller, trb: Trb) u64 {
|
fn submitCommand(self: *Controller, trb: Trb) u64 {
|
||||||
const physical = self.command_ring.push(trb);
|
const physical = self.command_ring.push(trb);
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
self.ringDoorbell(0, 0); // doorbell 0, target 0 = command ring
|
self.ringDoorbell(0, 0); // doorbell 0, target 0 = command ring
|
||||||
return physical;
|
return physical;
|
||||||
}
|
}
|
||||||
@@ -711,7 +711,7 @@ pub const Controller = struct {
|
|||||||
const slot = &ring[self.event_ring.dequeue_index];
|
const slot = &ring[self.event_ring.dequeue_index];
|
||||||
const control = slot.control;
|
const control = slot.control;
|
||||||
if ((control & cycle_bit != 0) == self.event_ring.cycle) {
|
if ((control & cycle_bit != 0) == self.event_ring.cycle) {
|
||||||
mmio.rmb();
|
mmio.readMemoryBarrier();
|
||||||
const event = Trb{ .parameter = slot.parameter, .status = slot.status, .control = control };
|
const event = Trb{ .parameter = slot.parameter, .status = slot.status, .control = control };
|
||||||
self.event_ring.dequeue_index += 1;
|
self.event_ring.dequeue_index += 1;
|
||||||
if (self.event_ring.dequeue_index >= trbs_per_ring) {
|
if (self.event_ring.dequeue_index >= trbs_per_ring) {
|
||||||
@@ -1216,7 +1216,7 @@ pub const Controller = struct {
|
|||||||
.control = trbControl(.status_stage, status_direction | (1 << 5)), // DIR | IOC
|
.control = trbControl(.status_stage, status_direction | (1 << 5)), // DIR | IOC
|
||||||
});
|
});
|
||||||
|
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
self.ringDoorbell(device.slot_id, 1); // DCI 1 = EP0
|
self.ringDoorbell(device.slot_id, 1); // DCI 1 = EP0
|
||||||
const code = self.awaitTransfer(device.slot_id, 1, @intCast(data.len)) orelse return false;
|
const code = self.awaitTransfer(device.slot_id, 1, @intCast(data.len)) orelse return false;
|
||||||
if (code != @intFromEnum(CompletionCode.success) and code != @intFromEnum(CompletionCode.short_packet)) return false;
|
if (code != @intFromEnum(CompletionCode.success) and code != @intFromEnum(CompletionCode.short_packet)) return false;
|
||||||
@@ -1457,7 +1457,7 @@ pub const Controller = struct {
|
|||||||
.status = length,
|
.status = length,
|
||||||
.control = trbControl(.normal, (1 << 5)), // IOC
|
.control = trbControl(.normal, (1 << 5)), // IOC
|
||||||
});
|
});
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
const number: u8 = endpoint.address & 0x0F;
|
const number: u8 = endpoint.address & 0x0F;
|
||||||
const direction_in = endpoint.address & 0x80 != 0;
|
const direction_in = endpoint.address & 0x80 != 0;
|
||||||
const dci = doorbellContextIndex(number, direction_in);
|
const dci = doorbellContextIndex(number, direction_in);
|
||||||
@@ -1507,7 +1507,7 @@ pub const Controller = struct {
|
|||||||
.status = subscription.max_length,
|
.status = subscription.max_length,
|
||||||
.control = trbControl(.normal, (1 << 5)), // IOC
|
.control = trbControl(.normal, (1 << 5)), // IOC
|
||||||
});
|
});
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
self.ringDoorbell(subscription.slot_id, subscription.dci);
|
self.ringDoorbell(subscription.slot_id, subscription.dci);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -102,17 +102,17 @@ var used_shadow: u16 = 0;
|
|||||||
// --- common-config register access (little-endian MMIO at `common_base`) ---------------
|
// --- common-config register access (little-endian MMIO at `common_base`) ---------------
|
||||||
|
|
||||||
fn cfgRead(comptime T: type, comptime field: []const u8) T {
|
fn cfgRead(comptime T: type, comptime field: []const u8) T {
|
||||||
return mmio.read(T, common_base + @offsetOf(vp.CommonCfg, field));
|
return mmio.readRegister(T, common_base + @offsetOf(vp.CommonCfg, field));
|
||||||
}
|
}
|
||||||
fn cfgWrite(comptime T: type, comptime field: []const u8, value: T) void {
|
fn cfgWrite(comptime T: type, comptime field: []const u8, value: T) void {
|
||||||
mmio.write(T, common_base + @offsetOf(vp.CommonCfg, field), value);
|
mmio.writeRegister(T, common_base + @offsetOf(vp.CommonCfg, field), value);
|
||||||
}
|
}
|
||||||
/// Write a 64-bit common-config register as two 32-bit halves (low then high) — the widest
|
/// Write a 64-bit common-config register as two 32-bit halves (low then high) — the widest
|
||||||
/// access every virtio-pci host is required to accept for the queue-address registers.
|
/// access every virtio-pci host is required to accept for the queue-address registers.
|
||||||
fn cfgWrite64(comptime field: []const u8, value: u64) void {
|
fn cfgWrite64(comptime field: []const u8, value: u64) void {
|
||||||
const at = common_base + @offsetOf(vp.CommonCfg, field);
|
const at = common_base + @offsetOf(vp.CommonCfg, field);
|
||||||
mmio.write(u32, at, @truncate(value));
|
mmio.writeRegister(u32, at, @truncate(value));
|
||||||
mmio.write(u32, at + 4, @truncate(value >> 32));
|
mmio.writeRegister(u32, at + 4, @truncate(value >> 32));
|
||||||
}
|
}
|
||||||
fn orStatus(bit: u8) void {
|
fn orStatus(bit: u8) void {
|
||||||
cfgWrite(u8, "device_status", cfgRead(u8, "device_status") | bit);
|
cfgWrite(u8, "device_status", cfgRead(u8, "device_status") | bit);
|
||||||
@@ -140,12 +140,12 @@ fn submit(request_len: usize, response_len: usize) bool {
|
|||||||
|
|
||||||
const avail_ring: [*]u16 = @ptrFromInt(ring.virtual + avail_offset + 4);
|
const avail_ring: [*]u16 = @ptrFromInt(ring.virtual + avail_offset + 4);
|
||||||
avail_ring[avail_shadow % queue_size] = 0; // head of the chain is descriptor 0
|
avail_ring[avail_shadow % queue_size] = 0; // head of the chain is descriptor 0
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
avail_shadow +%= 1;
|
avail_shadow +%= 1;
|
||||||
mmio.write(u16, ring.virtual + avail_offset + 2, avail_shadow); // avail.idx
|
mmio.writeRegister(u16, ring.virtual + avail_offset + 2, avail_shadow); // avail.idx
|
||||||
mmio.wmb();
|
mmio.writeMemoryBarrier();
|
||||||
|
|
||||||
mmio.write(u16, notify_addr, 0); // ring the control queue's doorbell
|
mmio.writeRegister(u16, notify_addr, 0); // ring the control queue's doorbell
|
||||||
return waitUsed();
|
return waitUsed();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -155,8 +155,8 @@ fn submit(request_len: usize, response_len: usize) bool {
|
|||||||
fn waitUsed() bool {
|
fn waitUsed() bool {
|
||||||
var tries: u32 = 0;
|
var tries: u32 = 0;
|
||||||
while (tries < 2000) : (tries += 1) {
|
while (tries < 2000) : (tries += 1) {
|
||||||
mmio.rmb();
|
mmio.readMemoryBarrier();
|
||||||
const idx = mmio.read(u16, ring.virtual + used_offset + 2); // used.idx
|
const idx = mmio.readRegister(u16, ring.virtual + used_offset + 2); // used.idx
|
||||||
if (idx != used_shadow) {
|
if (idx != used_shadow) {
|
||||||
used_shadow = idx;
|
used_shadow = idx;
|
||||||
return true;
|
return true;
|
||||||
@@ -231,16 +231,16 @@ fn initialise(endpoint: ipc.Handle) bool {
|
|||||||
var caps = function.capabilities();
|
var caps = function.capabilities();
|
||||||
while (caps.next()) |cap| {
|
while (caps.next()) |cap| {
|
||||||
if (cap.id != vp.pci_cap_vendor) continue;
|
if (cap.id != vp.pci_cap_vendor) continue;
|
||||||
const cfg_type = mmio.read(u8, cap.offset + 3);
|
const cfg_type = mmio.readRegister(u8, cap.offset + 3);
|
||||||
if (cfg_type != vp.cfg_common and cfg_type != vp.cfg_notify) continue;
|
if (cfg_type != vp.cfg_common and cfg_type != vp.cfg_notify) continue;
|
||||||
const bar = mmio.read(u8, cap.offset + 4);
|
const bar = mmio.readRegister(u8, cap.offset + 4);
|
||||||
const offset = mmio.read(u32, cap.offset + 8);
|
const offset = mmio.readRegister(u32, cap.offset + 8);
|
||||||
if (function.mapBar(bar)) |bar_base| {
|
if (function.mapBar(bar)) |bar_base| {
|
||||||
if (cfg_type == vp.cfg_common) {
|
if (cfg_type == vp.cfg_common) {
|
||||||
common_base = bar_base + offset;
|
common_base = bar_base + offset;
|
||||||
} else {
|
} else {
|
||||||
notify_base = bar_base + offset;
|
notify_base = bar_base + offset;
|
||||||
notify_multiplier = mmio.read(u32, cap.offset + 16); // virtio_pci_notify_cap tail
|
notify_multiplier = mmio.readRegister(u32, cap.offset + 16); // virtio_pci_notify_cap tail
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -290,7 +290,7 @@ fn initialise(endpoint: ipc.Handle) bool {
|
|||||||
std.log.info("command-buffer allocation failed", .{});
|
std.log.info("command-buffer allocation failed", .{});
|
||||||
return false;
|
return false;
|
||||||
};
|
};
|
||||||
mmio.write(u16, ring.virtual + avail_offset, 1); // VIRTQ_AVAIL_F_NO_INTERRUPT: we poll
|
mmio.writeRegister(u16, ring.virtual + avail_offset, 1); // VIRTQ_AVAIL_F_NO_INTERRUPT: we poll
|
||||||
cfgWrite(u16, "queue_size", queue_size);
|
cfgWrite(u16, "queue_size", queue_size);
|
||||||
cfgWrite64("queue_desc", ring.physical + desc_offset);
|
cfgWrite64("queue_desc", ring.physical + desc_offset);
|
||||||
cfgWrite64("queue_driver", ring.physical + avail_offset);
|
cfgWrite64("queue_driver", ring.physical + avail_offset);
|
||||||
@@ -372,7 +372,7 @@ fn initialise(endpoint: ipc.Handle) bool {
|
|||||||
}
|
}
|
||||||
// The scanout surface is CPU-visible RAM: read the pattern back to prove the mapping,
|
// The scanout surface is CPU-visible RAM: read the pattern back to prove the mapping,
|
||||||
// which together with the flush ack above is the automated stand-in for "it's on screen".
|
// which together with the flush ack above is the automated stand-in for "it's on screen".
|
||||||
mmio.rmb();
|
mmio.readMemoryBarrier();
|
||||||
if (pixels[0] != testPixel(0) or pixels[pixel_count / 2] != testPixel(@intCast(pixel_count / 2))) {
|
if (pixels[0] != testPixel(0) or pixels[pixel_count / 2] != testPixel(@intCast(pixel_count / 2))) {
|
||||||
std.log.info("pixel read-back mismatch", .{});
|
std.log.info("pixel read-back mismatch", .{});
|
||||||
return false;
|
return false;
|
||||||
@@ -435,7 +435,7 @@ fn readEdid() void {
|
|||||||
/// the panel. Reused by the V3 self-test and by every compositor present over `.scanout`. V4
|
/// the panel. Reused by the V3 self-test and by every compositor present over `.scanout`. V4
|
||||||
/// presents the full surface; the damage-rect fast path is a later refinement.
|
/// presents the full surface; the damage-rect fast path is a later refinement.
|
||||||
fn presentFull() bool {
|
fn presentFull() bool {
|
||||||
mmio.wmb(); // the surface writes must be visible before the device transfers them
|
mmio.writeMemoryBarrier(); // the surface writes must be visible before the device transfers them
|
||||||
{
|
{
|
||||||
// Transfer the current-mode rectangle from the guest backing to the host resource. The
|
// Transfer the current-mode rectangle from the guest backing to the host resource. The
|
||||||
// device uses the resource's (max) width as the row stride, so the top-left rect at
|
// device uses the resource's (max) width as the row stride, so the top-left rect at
|
||||||
|
|||||||
Reference in New Issue
Block a user