danos/docs/nvidia-gpus.md

Native NVIDIA GPU support — feasibility and roadmap

Status: research snapshot, not implemented. This records what a native display driver for a real discrete NVIDIA GPU — specifically an RTX 3060 (Ampere GA106) — would take, and how it would slot into danos's pluggable scanout architecture. It is a survey of primary sources (NVIDIA's open-gpu-kernel-modules, the Linux nouveau/nvkm driver, NVIDIA's open-gpu-doc, and linux-firmware), not an implementation. The NVIDIA driver landscape moves quickly (GSP defaults, firmware ABIs); treat specifics as a mid-decade snapshot and re-verify against current source before building.

Read display.md and display-v2.md first — this doc assumes the v2 model where scanout is a pluggable backend and a native driver is just another .scanout service (like the virtio-gpu one), announcing to the compositor over attach_scanout.

TL;DR

• A minimal display-only driver (EDID + mode-set + framebuffer scanout, no 3D/compute) for the RTX 3060 can and should avoid the GSP entirely. nouveau has a register-level, CPU-driven display path for Ampere (nvkm/engine/disp/ga102.c) that lights up GA106 with no external firmware; the signed-firmware wall gates the compute/graphics engines (PGRAPH), not the display controller. "GSP is mandatory on Ampere" is true only for NVIDIA's own RM-object route.
• danos's UEFI GOP boot is the single biggest thing in its favour. The VBIOS/GOP has already run devinit and brought up the display PLLs, so a driver attaches to a live, initialized GA106 — no firmware load, no cold-boot POST, no devinit interpreter. You reprogram a running display rather than bring one up from cold.
• It is still a hard, multi-week-to-months expert effort (effort tier ≈ 4/5) dominated by NVDisplay channel-DMA programming, SOR/head routing, DisplayPort AUX + link training, and the display supervisor handshake. The GSP/RM route is tier 5 (near-infeasible solo).
• The licensing tension is counterintuitive: the permissively-licensed reference (NVIDIA open-gpu-kernel-modules, MIT/GPLv2) is the hard GSP path; the register-level display code you actually want lives in GPL nouveau. See Licensing.
• The window is closing: GA10x (Ampere) is the last NVIDIA family with a register-level display path — Ada (RTX 40) deleted its non-GSP display HAL. Targeting Ampere specifically matters.
• Recommendation: for this card, GOP already gives native-resolution scanout with zero GPU code and zero maintenance. A native driver buys only runtime mode changes, hardware vsync/vblank, and multihead. It is justified if that runtime control is a danos goal, or to learn the craft — for which an Intel iGPU or a pre-Turing NVIDIA card reaches "first pixel" far faster.

The GSP wall, and why display sits on the near side of it

On Turing and later, NVIDIA split its driver's Resource Manager into a host CPU-RM and a GSP-RM running on an on-die RISC-V core ("Peregrine"), talking over RPC (LWN 953144). The GSP is a full resource manager, not a display coprocessor — there is no "display-only" GSP image and no small display RPC subset. Its boot chain is entirely signed and mandatory: a VBIOS-resident FWSEC-FRTS app carves a write-protected region (WPR2), a signed Booter on the SEC2 falcon loads the GSP bootloader, and that loads GSP-RM inside WPR. The firmware ships pre-computed signatures and the driver picks one by an on-chip fuse-version register — you cannot self-sign, and there is no stable firmware ABI (it is revised every driver release; nouveau and the Rust nova-core driver each pin exactly one version). A GSP driver is a permanent maintenance liability, not a one-time build (LWN 1037379, nova-core cover letter).

But display doesn't need any of that on Ampere. nvkm/engine/disp/ga102.c dual-dispatches:

if (nvkm_gsp_rm(device->gsp)) return r535_disp_new(&ga102_disp, ...);  // GSP RPC path
return nvkm_disp_new_(&ga102_disp, ...);                               // direct register path

Both branches use the same ga102_disp HAL and the same GA102_DISP_* class IDs; GSP merely swaps register programming for RPC. GA106 (chipset 0x176) is wired to ga102_disp_new in the device table, identical to GA102/103/104/107. Ampere lit up displays via the direct path in Linux 5.11/5.17 — two years before GSP-RM landed (6.7, 2023) (ga102.c, Phoronix GA106).

Caveat — this is now the legacy path. As of Linux 6.18, nouveau defaults to GSP on Turing/Ampere; the direct path is a retained, forceable fallback (nouveau.config=NvGspRm=0, and automatic when GSP firmware is absent). It is stable and proven, but NVIDIA and nova-core are moving to GSP-only, and Ada already deleted its non-GSP display HAL. GA10x is the last family that keeps a register-level display path.

What "direct" actually entails

"Direct" is not "plain register pokes." Only SOR / PLL / DP-link / clock setup is bare MMIO. The mode-set and scanout themselves flow through the NVDisplay channels — a DMA pushbuffer:

• Display classes for Ampere (the C670 family): core GA102_DISP_CORE_CHANNEL_DMA (0xc67d), window 0xc67e, window-immediate 0xc67b, cursor 0xc67a (headers clc67d.h / clc67e.h / clc67a.h in open-gpu-doc classes/display/).
• The core channel needs instance memory, a RAMHT, DMA objects, and a channel user-MMIO region (disp/chan.c). The register-level "plumbing" to allocate/kick a channel is in NVIDIA's GA102 display register manual: NV_PDISP_FE_CHNCTL_CORE/WIN/CURS, NV_PDISP_FE_PBBASE/PBBASEHI (dev_display_withoffset.ref.txt).
• Mode-set is a method stream on the core channel: HEAD_SET_RASTER_*, HEAD_SET_PIXEL_CLOCK_FREQUENCY, HEAD_SET_CONTROL_OUTPUT_RESOURCE, SOR_SET_CONTROL (protocol select), viewport/scaler, then UPDATE. The window channel points at the scanout surface (SET_CONTEXT_DMA_ISO, SET_STORAGE, SET_OFFSET).
• After UPDATE you must complete the display supervisor interrupt handshake (SV1/SV2/SV3).

EDID and DisplayPort are a separate subdev you must port. open-gpu-doc documents none of EDID/DDC/AUX. On the direct path you read EDID in-driver via nouveau's nvkm/subdev/i2c: bit-bang DDC/I²C at address 0x50 (E-DDC 0x30) for TMDS/HDMI, or native DP AUX in i2c/aux.c for DisplayPort. DisplayPort link training (the dp.c train_cr / train_eq state machine over AUX — clock recovery, lane/rate, voltage-swing/pre-emphasis) is the single hardest and most fragile piece; a DVI/HDMI (TMDS) panel avoids it entirely.

The memory floor (smaller than you'd fear)

Neither route hands you a framebuffer allocator — even GSP-RM does not manage the scanout framebuffer; the driver owns VRAM and merely tells GSP where its page directory is. But display-only is a small fraction of a full GEM/TTM stack:

• Pitch-linear (untiled) scanout is allowed on nv50→Ampere — the window's storage method has a PITCH layout mode, so you skip block-linear tiling math (wndwc37e.c).
• The window references its surface through a simple display context-DMA (SET_CONTEXT_DMA_ISO + a 256-byte-granular SET_OFFSET = addr>>8) — a base/limit descriptor, not the GPU's 5-level compute page tables. No full GPU VMM is needed for scanout.
• The surface must live in VRAM in practice (nouveau always pins scanout to VRAM). Open question: whether GA10x can scan out from a system-memory (GART) surface via a sysmem-target ctxdma — which would let danos skip a VRAM allocator. No source forbids it; nouveau never does it (confidence: medium).
• CPU access to the framebuffer for compositing goes through BAR1 (a VRAM aperture); BAR0 is the 16 MB register window. BAR1 can be smaller than 12 GB of VRAM unless Resizable BAR maps it all.

Net: you need (1) a contiguous aligned VRAM allocator (256-byte base, pitch a multiple of 64 bytes — confirm against the Ampere display refs), (2) a little instmem for the channel pushbuffers + iso ctxdma, (3) a BAR1 CPU mapping. You do not need the 5-level VMM, GEM/TTM eviction, or tiling.

Licensing

The tension is the opposite of convenient:

• NVIDIA open-gpu-kernel-modules is dual MIT/GPLv2 — usable under MIT, no copyleft on your other code — but its display logic is the GSP/RM-object route. Its class headers (cl0073.h, cl2080.h, ctrl0073*.h) are useful, permissive references.
• nouveau is GPLv2, and the register-level display sequences you actually want live in nouveau, not in the MIT code. So the easy technical path is the GPL-licensed one. Reading GPL nouveau and reimplementing it in Zig is a derivative-work risk proportional to how closely your code tracks its structure/constants.

Options: (a) accept that the danos NVIDIA display driver is a GPL component. danos's userspace-driver-over-IPC model (a driver is a separate process behind a defined protocol, not linked into the kernel) is about the cleanest possible GPL boundary, so the GPL would be contained to that one binary and the rest of danos could keep its own license — but this is a licensing-boundary judgement that wants real diligence, not a settled fact. (b) clean-room from specification rather than code: envytools + NVIDIA's open-gpu-doc register manuals + the MIT OGKM class headers, treating nouveau as documentation-of-last-resort.

Firmware licensing is moot for the direct path (no firmware is loaded). For completeness: the GSP blobs are marked redistributable under LICENCE.nvidia, which permits use by any OSI-approved open-source OS (not just Linux), on NVIDIA GPUs, unmodified, with no reverse-engineering of the firmware binary. The one gate — is danos released under an OSI license? — is only reached on the GSP route, which this doc recommends against for this card.

Prior art

No one has built a from-scratch native NVIDIA driver outside Linux. FreeBSD ships nvidia-drm-kmod, a port of NVIDIA's own closed nvidia-drm.ko loading the GSP blob (its old nouveau port was removed). Haiku's NVIDIA support is likewise a port of OGKM (GSP, Turing+, very alpha). OpenBSD / DragonFly have neither. Every non-Linux OS that supports modern NVIDIA chose to wrap NVIDIA's GSP stack rather than write a native driver. A danos direct-register driver would have exactly one reference implementation — GPL nouveau — and no non-Linux precedent.

Alternatives

Option	What you get	The tradeoff
Stay on GOP (working today)	Native-res scanout, zero GPU code/firmware/maintenance	Resolution frozen at ExitBootServices; no runtime mode change, no hardware vsync, no multihead
Pre-Turing NVIDIA (Kepler / early Maxwell)	Direct EVO/disp-core + CRTC/PLL modeset, no signed firmware, no coprocessor; mature nouveau reference	Older display class; not this card; only reclocking is firmware-gated
Intel iGPU	Publicly documented register interfaces (Intel PRMs); no coprocessor mediating modeset	i915 is huge + generation-specific; write one generation from the PRM
Native GA106 direct (this doc)	Runtime modeset, vsync, multihead on the actual card	Tier-4 effort; GPL reference; DP link training; legacy/de-emphasized path
GA106 via GSP/OGKM	Also unlocks 3D / reclocking later	Tier-5; ~14k-line ante; unstable version-pinned ABI; unprecedented outside Linux

"First light" milestones (direct path, inheriting GOP state)

Framed as a danos .scanout service (like the virtio-gpu driver), taking the direct register path and inheriting the GOP-initialized display — no signed firmware, no devinit, no GSP:

1. PCI/BAR bring-up — enumerate GA106 (0x176), map BAR0 (registers) and BAR1 (VRAM aperture) via danos MMIO grants; confirm the display engine is GOP-live.
2. VRAM + instmem allocator — contiguous aligned VRAM for the scanout surface (256-byte base)
   • small instmem for pushbuffers / RAMHT / iso ctxdma. No VMM, no TTM.
3. EDID — port nvkm/subdev/i2c DDC (0x50) + DP-AUX (aux.c); read + parse the panel EDID.
4. Core channel up — allocate the 0xc67d core channel as a DMA pushbuffer; stand up the SV1/SV2/SV3 supervisor-interrupt handshake.
5. First pixel = reprogram, don't re-POST — bind a window (0xc67e) at the existing WC framebuffer via SET_CONTEXT_DMA_ISO + SET_OFFSET, pitch-linear, UPDATE; prove you can drive the current GOP mode from your own channel before changing anything.
6. Modeset — push raster timings on a head, route head→SOR→connector, program the pixel-clock PLL, switch to an EDID mode (needs the clc67d/e method opcodes from the OGKM headers + the supervisor timing from nouveau head.c).
7. DisplayPort link training — only if the panel is DP and GOP's link can't be reused; the dp.c train_cr/train_eq state machine. TMDS/HDMI is far simpler.
8. Wire into the compositor .scanout backend (attach_scanout), add vsync via the display interrupt, then multihead.

Keep the GOP backend as the fallback the whole way — a stall at any step still leaves danos with a working display (exactly the resilience v2 already provides via re-attach).

Reading list

Direct path — nouveau (GPLv2):

• nvkm/engine/disp/ga102.c — the GA10x display HAL + the GSP/non-GSP dispatch.
• nvkm/engine/disp/{head.c, ior.c, dp.c, hdmi.c, chan.c} — head/SOR routing, DP AUX + link training, channel-DMA plumbing.
• dispnv50/{corec37d.c, corec57d.c, wndwc37e.c, wndwc57e.c, wndwc67e.c, headc37d.c, cursc37a.c}.
• nvkm/subdev/i2c (DDC + aux.c) for EDID; nvkm/subdev/bios/init.c + devinit/ only if you ever have to re-POST (danos's GOP handoff means you shouldn't).

Object model / GSP path — NVIDIA OGKM (MIT/GPLv2): class headers cl0073.h, cl2080.h, ctrl0073system.h, ctrl0073specific.h; src/nvidia/ for RM control sequences. nvidia-modeset.ko (NVKMS) is a policy layer over RM and can be bypassed entirely. nova-core (Rust) is the forward-looking reference for GSP boot mechanics (falcon signing, queue rings, RPC).

Register / method specs — NVIDIA open-gpu-doc:

• classes/display/README.txt — the channel model + class-to-GPU map (read first).
• classes/display/clc67d.h
  • clc67e.h / clc67a.h — the Ampere core/window/cursor mode-set method vocabulary.
• manuals/ampere/ga102/dev_display_withoffset.ref.txt — NV_PDISP_FE_* channel/pushbuffer registers + SOR.
• DCB — connector→output-resource routing; Devinit + BIOS-Information-Table — VBIOS parsing (bring-up reference; not needed if inheriting GOP).
• The 632 KB Volta dev_display.ref is the best shot at SOR-DP/AUX register detail the smaller Ampere file omits.

Open questions (unresolved by the survey)

Each needs a direct read of the named nouveau file or experimentation on the actual card:

• Exact GA106 register/method offsets and PADLINK→SOR→connector wiring (can vary by board vendor).
• Whether any PLL/devinit re-run is unavoidable vs. fully inherited from GOP.
• Whether DisplayPort needs full retraining on takeover, or the GOP-established link can be reused.
• The precise SV1/SV2/SV3 supervisor sequence.
• Whether a system-memory-target scanout ctxdma could eliminate the VRAM allocator.
• The exact clc67d.h/clc67e.h method opcode numbers (not captured verbatim in the survey).

---

Research snapshot; verify against current nouveau / open-gpu-kernel-modules source before building — NVIDIA's GSP defaults and firmware ABIs change per release.