Add runtime.time, drop demo drivers, harden TSC timekeeping

Time is a kernel concern in danos: the kernel owns the scheduling timer and
already exposes monotonic time via the clock/sleep/timer_bind syscalls, so a
userspace time service would be a redundant, slower path. This adds the generic
runtime.time module over those syscalls, retires the two demonstration drivers,
reorganizes the milestone docs, and makes the monotonic clock correct on Intel,
AMD, and inside any VM.

runtime.time (library/runtime/time.zig)
- Instant/Duration interface: now, sleep, spin, after, monotonicNanos, available
- a thin layer over system.clock/sleep/timerOnce; unit-tested arithmetic

Remove the demo drivers hpet and bus (a teaching example belongs in the docs,
not shipped in the tree)
- system/drivers/ now holds only real drivers: pci-bus, ps2-bus, usb-xhci-bus
- device-manager end-to-end test repointed to pci-bus (asserts on kernel state:
  the process table and the device tree, not a racy serial marker)
- device_register containment moved to a new in-kernel `containment` test
- the driver-model worked example moved inline into docs/drivers.md

Reorganize milestone docs into topic docs
- m17-m18 / m19-m20 / m21 plans dissolved into process-lifecycle, device-manager,
  discovery, and acpi docs; new docs/power.md and docs/timers.md; ~20 citations
  repointed; plan docs deleted

TSC reliability (apic.zig, smp.zig, cpu.zig, kernel.zig)
- check the invariant-TSC bit (CPUID 0x80000007 EDX[8]) on Intel and AMD
- cross-core "warp" check at SMP bring-up, pairwise BSP<->AP as each core comes up
- fall back to the HPET clocksource when the TSC is not invariant (a bare VM) or
  not synchronized (a warp), switched continuously so time never jumps
- boot log reports the outcome; new tsc-sync test exercises the TSC + warp path

Verified: zig build; zig build test; 60/60 QEMU cases (incl. new containment and
tsc-sync).
This commit is contained in:
Daniel Samson
2026-07-13 11:56:43 +01:00
parent 5b63a841ba
commit 452080e997
36 changed files with 1150 additions and 1267 deletions
+230 -13
View File
@@ -80,6 +80,35 @@ var timer_hz: u32 = 0;
var tsc_hz: u64 = 0;
var tsc_base: u64 = 0;
/// Whether the TSC is architecturally **invariant** — a constant rate regardless of
/// P/C-state transitions, and thus valid as a clocksource (CPUID leaf 0x80000007,
/// EDX bit 8). AMD and modern Intel set it; the bare qemu64 model does not. Measured
/// frequency alone is not enough: a non-invariant TSC speeds up and slows down with
/// the core clock, so reading it as wall time would drift.
var tsc_invariant: bool = false;
/// Cleared if the cross-core warp check (checkWarpSource) ever sees the TSC read
/// lower on one core than the max another core has already published — i.e. the
/// per-core TSCs are not synchronized, and a task migrating cores could see time go
/// backward. Starts true (assume synchronized until proven otherwise).
var tsc_synced: bool = true;
/// The worst backward skew the warp check observed, in TSC cycles (0 = none).
var tsc_warp_cycles: u64 = 0;
/// The monotonic clock's source. The TSC when it is invariant *and* synchronized —
/// the fast `rdtsc` path taken on real Intel/AMD and modern VMs. Otherwise the HPET
/// main counter: a single fixed-rate counter, immune to both per-core skew and
/// frequency scaling, so it stays accurate on a bare VM or a warped machine.
const ClockSource = enum { tsc, hpet };
var clock_source: ClockSource = .tsc;
/// HPET standby clocksource, set up in calibrate() whenever an HPET exists (whether
/// or not calibration itself measured against it): its frequency, the counter value
/// chosen as the zero point, and its width mask. Only a 64-bit HPET is used as a
/// clocksource — a 32-bit one wraps too fast to be monotonic without accumulation.
var hpet_clock_hz: u64 = 0;
var hpet_clock_base: u64 = 0;
var hpet_clock_mask: u64 = ~@as(u64, 0);
/// Read the 64-bit Time Stamp Counter.
fn rdtsc() u64 {
var low: u32 = undefined;
@@ -220,6 +249,31 @@ pub fn calibrate() void {
}
tsc_base = rdtsc(); // the clock's zero point (boot)
// Decide whether the TSC is trustworthy as a clocksource. Frequency (measured
// above, possibly against the HPET/PIT) is necessary but not sufficient: the TSC
// must also be *invariant* (CPUID 0x80000007 EDX[8]). AMD and modern Intel set
// this; the bare qemu64 model does not.
tsc_invariant = tscIsInvariant();
// Bring up the HPET as a standby clocksource whenever one exists — even on the
// CPUID-0x15 path where calibration never touched it — so a non-invariant TSC
// (here) or an unsynchronized one (checkWarpSource, during SMP bring-up) can fall
// back to a source that is immune to both. hpetHz() maps + enables the counter
// and is idempotent if calibration already used it.
if (configuration_hpet_base != 0) {
if (hpetHz()) |hz| {
hpet_clock_mask = hpetMask();
if (hpet_clock_mask == ~@as(u64, 0)) { // only a 64-bit HPET is monotonic enough
hpet_clock_hz = hz;
hpet_clock_base = readHpet();
}
}
}
// Select the source: the fast TSC when invariant, else the HPET if we have one.
// (checkWarpSource may still demote TSC -> HPET later if the cores' TSCs skew.)
if (!tsc_invariant and hpet_clock_hz != 0) clock_source = .hpet;
}
/// Run the LAPIC timer one-shot from its maximum count while a monotonic reference
@@ -275,7 +329,7 @@ fn calibratePit() void {
// --- reference clocks ------------------------------------------------------
/// TSC frequency from CPUID leaf 0x15 (crystal_hz * numerator / denominator), or
/// null if the CPU doesn't enumerate it (common under QEMU).
/// null if the CPU doesn't enumerate it (common under QEMU, and on AMD).
fn cpuidTscHz() ?u64 {
if (cpuid(0).eax < 0x15) return null;
const r = cpuid(0x15);
@@ -283,6 +337,15 @@ fn cpuidTscHz() ?u64 {
return @as(u64, r.ecx) * r.ebx / r.eax;
}
/// Whether the CPU advertises an **invariant** TSC (CPUID leaf 0x80000007, EDX
/// bit 8) — the architectural guarantee, on both Intel and AMD, that the TSC ticks
/// at a constant rate across P/C-states and never stops. Requires the extended-leaf
/// range to reach 0x80000007 first.
fn tscIsInvariant() bool {
if (cpuid(0x80000000).eax < 0x80000007) return false;
return (cpuid(0x80000007).edx & (1 << 8)) != 0;
}
const CpuidRegs = struct { eax: u32, ebx: u32, ecx: u32, edx: u32 };
fn cpuid(leaf: u32) CpuidRegs {
@@ -310,11 +373,20 @@ fn hpetWrite64(off: usize, value: u64) void {
@as(*volatile u64, @ptrFromInt(configuration_hpet_base + off)).* = value;
}
/// Whether the HPET has been mapped into the physmap yet, so `configuration_hpet_base`
/// already holds the virtual address. `hpetHz` is called more than once (calibration
/// may use the HPET, and the standby-clocksource setup asks for it again), and mapping
/// an already-mapped base a second time would double-offset it into an overflow.
var hpet_mapped: bool = false;
/// Map + enable the HPET and return its tick frequency, or null if unusable.
/// Maps the HPET into the physmap and switches configuration_hpet_base to that virtual
/// address, so the register accessors reach it without the identity map.
/// address, so the register accessors reach it without the identity map. Idempotent.
fn hpetHz() ?u64 {
configuration_hpet_base = paging.mapMmio(configuration_hpet_base, 0x400, true);
if (!hpet_mapped) {
configuration_hpet_base = paging.mapMmio(configuration_hpet_base, 0x400, true);
hpet_mapped = true;
}
const caps = hpetRead64(0x00);
const period_fs = caps >> 32; // femtoseconds per tick
if (period_fs == 0) return null;
@@ -364,24 +436,169 @@ pub fn tscHz() u64 {
return tsc_hz;
}
// Monotonic high-resolution clock, from the TSC. A function per resolution, each
// scaling the cycle delta directly at its unit (the 128-bit intermediate avoids
// overflow across a long uptime). nanos() resolves to a few ns; millis() is what
// the scheduler uses for sleep deadlines.
// Monotonic high-resolution clock. A function per resolution, each scaling the
// counter delta directly at its unit (the 128-bit intermediate avoids overflow
// across a long uptime). nanos() resolves to a few ns on the TSC; millis() is what
// the scheduler uses for sleep deadlines. The source is the TSC when it is invariant
// and synchronized, else the HPET counter (see clock_source) — the branch is one
// global load and the TSC path is unchanged from before.
/// The selected source's counter delta since its zero point.
fn clockCount() u64 {
return switch (clock_source) {
.tsc => rdtsc() -% tsc_base,
// A 64-bit HPET (the only kind we select) never wraps in any realistic
// uptime, so the wrapping subtraction is exact.
.hpet => readHpet() -% hpet_clock_base,
};
}
/// The selected source's frequency (0 if the clock is unavailable/uncalibrated).
fn clockHertz() u64 {
return switch (clock_source) {
.tsc => tsc_hz,
.hpet => hpet_clock_hz,
};
}
pub fn nanos() u64 {
if (tsc_hz == 0) return 0;
return @intCast(@as(u128, rdtsc() -% tsc_base) * 1_000_000_000 / tsc_hz);
const hz = clockHertz();
if (hz == 0) return 0;
return @intCast(@as(u128, clockCount()) * 1_000_000_000 / hz);
}
pub fn micros() u64 {
if (tsc_hz == 0) return 0;
return @intCast(@as(u128, rdtsc() -% tsc_base) * 1_000_000 / tsc_hz);
const hz = clockHertz();
if (hz == 0) return 0;
return @intCast(@as(u128, clockCount()) * 1_000_000 / hz);
}
pub fn millis() u64 {
if (tsc_hz == 0) return 0;
return @intCast(@as(u128, rdtsc() -% tsc_base) * 1_000 / tsc_hz);
const hz = clockHertz();
if (hz == 0) return 0;
return @intCast(@as(u128, clockCount()) * 1_000 / hz);
}
/// Whether the CPU advertises an invariant TSC (CPUID 0x80000007 EDX[8]).
pub fn tscInvariant() bool {
return tsc_invariant;
}
/// Test hook: force the TSC clocksource on, as if the CPU had advertised an invariant
/// TSC. QEMU's TCG accelerator (the only one for an x86 guest on an Apple-Silicon
/// host) does not expose the invariant-TSC bit — its emulated TSC isn't invariant — so
/// the tsc-sync test can't reach the real-Intel/AMD/KVM path through CPUID. This lets
/// that test exercise the TSC clocksource and the cross-core warp check anyway. tsc_base
/// is left as-is so the switch from the HPET is continuous.
pub fn forceTscClocksourceForTest() void {
tsc_invariant = true;
clock_source = .tsc;
}
/// How many per-AP warp checks actually ran (a rendezvous completed) — lets a test
/// confirm the cross-core check executed rather than being skipped.
pub fn warpChecksRun() u32 {
return warp_checks;
}
/// Whether the per-core TSCs are synchronized (no backward warp seen at bring-up).
pub fn tscSynced() bool {
return tsc_synced;
}
/// The active monotonic clocksource, for the boot log and tests.
pub fn clockSourceName() []const u8 {
return switch (clock_source) {
.tsc => "tsc",
.hpet => "hpet",
};
}
// --- cross-core TSC synchronization ("warp") check -------------------------
// Two cores hammer a shared "max seen" TSC value under a lock; if either reads a
// value below that max, its TSC lags the other's, and time would run backward for a
// task migrating between them (Linux calls this a warp). danos brings APs up one at a
// time, so this runs pairwise: the BSP (source) against each AP (target) as it comes
// online. It only matters — and only runs — while the TSC is the clocksource; on a
// machine already on the HPET (a bare VM) the whole rendezvous is skipped.
var warp_lock: u32 = 0;
var warp_last: u64 = 0;
var warp_bsp_ready: u32 = 0;
var warp_ap_ready: u32 = 0;
var warp_stop: u32 = 0;
var warp_checks: u32 = 0; // completed per-AP rendezvous count (for the tsc-sync test)
const warp_rounds: u32 = 1 << 20; // locked reads on the BSP: ~1 ms at GHz rates
const warp_spin_limit: u64 = 1 << 32; // bound every rendezvous wait so a lost core can't hang boot
fn warpTick() void {
while (@cmpxchgWeak(u32, &warp_lock, 0, 1, .acquire, .monotonic) != null) asm volatile ("pause");
const t = rdtsc();
if (t < warp_last) {
const delta = warp_last - t;
if (delta > tsc_warp_cycles) tsc_warp_cycles = delta;
tsc_synced = false;
} else {
warp_last = t;
}
@atomicStore(u32, &warp_lock, 0, .release);
}
/// Spin (bounded) until `flag` is nonzero; false on timeout.
fn warpAwait(flag: *u32) bool {
var spins: u64 = 0;
while (@atomicLoad(u32, flag, .acquire) == 0) : (spins += 1) {
if (spins >= warp_spin_limit) return false;
asm volatile ("pause");
}
return true;
}
/// BSP side of the pairwise TSC warp check, run once per AP as it reports in. No-op
/// unless the TSC is the active clocksource. If the AP's TSC proves to lag, demote
/// the monotonic clock to the HPET without a discontinuity.
pub fn checkWarpSource() void {
if (clock_source != .tsc) return;
warp_last = 0;
@atomicStore(u32, &warp_stop, 0, .release);
@atomicStore(u32, &warp_ap_ready, 0, .release);
@atomicStore(u32, &warp_bsp_ready, 1, .release);
if (!warpAwait(&warp_ap_ready)) { // AP never joined the rendezvous; skip, don't hang
@atomicStore(u32, &warp_bsp_ready, 0, .release);
return;
}
var i: u32 = 0;
while (i < warp_rounds) : (i += 1) warpTick();
@atomicStore(u32, &warp_stop, 1, .release);
@atomicStore(u32, &warp_bsp_ready, 0, .release);
warp_checks += 1;
if (!tsc_synced and hpet_clock_hz != 0) demoteToHpet();
}
/// AP side: join the BSP's warp check, then return so the core can enter the
/// scheduler. Bounded so a missing BSP can't strand the core.
pub fn checkWarpTarget() void {
if (clock_source != .tsc) return;
if (!warpAwait(&warp_bsp_ready)) return;
@atomicStore(u32, &warp_ap_ready, 1, .release);
var spins: u64 = 0;
while (@atomicLoad(u32, &warp_stop, .acquire) == 0) : (spins += 1) {
if (spins >= warp_spin_limit) return;
warpTick();
}
}
/// Switch the clocksource from the TSC to the HPET without a discontinuity: choose
/// the HPET zero point so it reads the same nanosecond value the TSC does right now,
/// so time neither jumps nor runs backward across the switch. Called when the warp
/// check proves the per-core TSCs unsynchronized.
fn demoteToHpet() void {
const now_ns = @as(u128, rdtsc() -% tsc_base) * 1_000_000_000 / tsc_hz;
const equivalent_ticks: u64 = @intCast(now_ns * hpet_clock_hz / 1_000_000_000);
hpet_clock_base = readHpet() -% equivalent_ticks;
clock_source = .hpet;
}
/// Acknowledge the current interrupt so the LAPIC will deliver the next one.
+29
View File
@@ -469,6 +469,35 @@ pub fn clockHz() u64 {
return apic.tscHz();
}
/// Whether the CPU guarantees an **invariant** TSC (CPUID 0x80000007 EDX[8] on
/// x86; the analogous architectural guarantee elsewhere). When false the TSC is not
/// used as the clocksource.
pub fn clockInvariant() bool {
return apic.tscInvariant();
}
/// Whether the per-core clock counters are synchronized (no backward warp observed
/// at SMP bring-up). When false the clock falls back off the TSC.
pub fn clockSynchronized() bool {
return apic.tscSynced();
}
/// The active monotonic clocksource, for the boot log ("tsc" or "hpet" on x86).
pub fn clockSourceName() []const u8 {
return apic.clockSourceName();
}
/// Test hook: force the TSC clocksource on, to exercise the TSC + warp-check path on
/// a hypervisor that won't advertise an invariant TSC (see apic.forceTscClocksourceForTest).
pub fn forceTscClocksourceForTest() void {
apic.forceTscClocksourceForTest();
}
/// How many per-AP TSC warp checks completed (for the tsc-sync test).
pub fn warpChecksRun() u32 {
return apic.warpChecksRun();
}
/// Unmask maskable interrupts (`sti`) so device interrupts get delivered.
pub fn enableInterrupts() void {
asm volatile ("sti");
+13 -1
View File
@@ -148,7 +148,14 @@ pub fn startAp(apic_id: u32, stack_top: usize, percpu: usize, index: usize, cr3:
// Wait up to 100 ms for the AP to reach apEntry and set the flag.
const deadline = apic.millis() + 100;
while (apic.millis() < deadline) {
if (@atomicLoad(u32, &ap_alive, .acquire) != 0) return true;
if (@atomicLoad(u32, &ap_alive, .acquire) != 0) {
// Cross-check this core's TSC against the BSP's before it joins the run
// loop: an unsynchronized TSC must be caught before any task can migrate
// onto this core and observe time going backward. No-op unless the TSC is
// the clocksource (apic.checkWarpSource).
apic.checkWarpSource();
return true;
}
asm volatile ("pause");
}
return false;
@@ -177,6 +184,11 @@ fn apEntry(percpu: usize) callconv(.c) noreturn {
@atomicStore(u32, &ap_alive, 1, .release); // "architecture state up" — BSP is polling this
// Rendezvous with the BSP for the TSC warp check (no-op unless the TSC is the
// clocksource) before joining the run loop, so this core's clock is vetted before
// it can run any task.
apic.checkWarpTarget();
if (secondary_entry) |enterScheduler| enterScheduler(); // joins the run loop
while (true) asm volatile ("hlt"); // (only if no entry was registered)
}