danos/docs/nvidia-gpus.md

# Native NVIDIA GPU support — feasibility and roadmap

**Status: research snapshot, not implemented.** This records what a *native* display driver for a
real discrete NVIDIA GPU — specifically an **RTX 3060 (Ampere GA106)** — would take, and how it
would slot into danos's pluggable scanout architecture. It is a survey of primary sources
(NVIDIA's [open-gpu-kernel-modules](https://github.com/NVIDIA/open-gpu-kernel-modules), the Linux
[nouveau/nvkm](https://github.com/torvalds/linux/tree/master/drivers/gpu/drm/nouveau) driver,
NVIDIA's [open-gpu-doc](https://nvidia.github.io/open-gpu-doc/), and
[linux-firmware](https://github.com/NVIDIA/linux-firmware)), not an implementation. The NVIDIA
driver landscape moves quickly (GSP defaults, firmware ABIs); treat specifics as a mid-decade
snapshot and re-verify against current source before building.

Read [display.md](display.md) and [display-v2.md](display-v2.md) first — this doc assumes the
v2 model where scanout is a **pluggable backend** and a native driver is just another `.scanout`
service (like the virtio-gpu one), announcing to the compositor over `attach_scanout`.

## TL;DR

- A **minimal display-only driver** (EDID + mode-set + framebuffer scanout, **no** 3D/compute)
  for the RTX 3060 **can and should avoid the GSP entirely**. nouveau has a register-level,
  CPU-driven display path for Ampere (`nvkm/engine/disp/ga102.c`) that lights up GA106 with no
  external firmware; the signed-firmware wall gates the **compute/graphics** engines (PGRAPH),
  **not** the display controller. "GSP is mandatory on Ampere" is true only for NVIDIA's own
  RM-object route.
- **danos's UEFI GOP boot is the single biggest thing in its favour.** The VBIOS/GOP has already
  run devinit and brought up the display PLLs, so a driver attaches to a **live, initialized**
  GA106 — no firmware load, no cold-boot POST, no devinit interpreter. You reprogram a running
  display rather than bring one up from cold.
- It is still a **hard, multi-week-to-months expert effort** (effort tier ≈ 4/5) dominated by
  NVDisplay channel-DMA programming, SOR/head routing, DisplayPort AUX + link training, and the
  display supervisor handshake. The GSP/RM route is tier 5 (near-infeasible solo).
- The **licensing tension is counterintuitive**: the permissively-licensed reference (NVIDIA
  open-gpu-kernel-modules, MIT/GPLv2) is the **hard GSP path**; the register-level display code
  you actually want lives in **GPL nouveau**. See [Licensing](#licensing).
- The **window is closing**: GA10x (Ampere) is the *last* NVIDIA family with a register-level
  display path — Ada (RTX 40) deleted its non-GSP display HAL. Targeting Ampere specifically
  matters.
- **Recommendation:** for *this card*, GOP already gives native-resolution scanout with zero GPU
  code and zero maintenance. A native driver buys only runtime mode changes, hardware
  vsync/vblank, and multihead. It is justified if that runtime control is a danos goal, or to
  *learn the craft* — for which an Intel iGPU or a pre-Turing NVIDIA card reaches "first pixel"
  far faster.

## The GSP wall, and why display sits on the near side of it

On Turing and later, NVIDIA split its driver's Resource Manager into a host **CPU-RM** and a
**GSP-RM** running on an on-die RISC-V core ("Peregrine"), talking over RPC
([LWN 953144](https://lwn.net/Articles/953144/)). The GSP is a *full resource manager*, not a
display coprocessor — there is no "display-only" GSP image and no small display RPC subset. Its
boot chain is entirely signed and mandatory: a VBIOS-resident **FWSEC-FRTS** app carves a
write-protected region (WPR2), a signed **Booter** on the SEC2 falcon loads the GSP bootloader,
and that loads **GSP-RM** inside WPR. The firmware ships pre-computed signatures and the driver
picks one by an on-chip fuse-version register — **you cannot self-sign**, and there is **no stable
firmware ABI** (it is revised every driver release; nouveau and the Rust nova-core driver each pin
exactly one version). A GSP driver is a permanent maintenance liability, not a one-time build
([LWN 1037379](https://lwn.net/Articles/1037379/),
[nova-core cover letter](https://lore.freedesktop.org/nouveau/20250826-nova_firmware-v2-7-93566252fe3a@nvidia.com/T/)).

**But display doesn't need any of that on Ampere.** `nvkm/engine/disp/ga102.c` dual-dispatches:

```
if (nvkm_gsp_rm(device->gsp)) return r535_disp_new(&ga102_disp, ...);  // GSP RPC path
return nvkm_disp_new_(&ga102_disp, ...);                               // direct register path
```

Both branches use the same `ga102_disp` HAL and the same `GA102_DISP_*` class IDs; GSP merely
swaps register programming for RPC. GA106 (chipset `0x176`) is wired to `ga102_disp_new` in the
device table, identical to GA102/103/104/107. Ampere lit up displays via the **direct** path in
Linux 5.11/5.17 — two years before GSP-RM landed (6.7, 2023)
([ga102.c](https://raw.githubusercontent.com/torvalds/linux/master/drivers/gpu/drm/nouveau/nvkm/engine/disp/ga102.c),
[Phoronix GA106](https://www.phoronix.com/news/Nouveau-NVIDIA-GA106)).

**Caveat — this is now the legacy path.** As of Linux 6.18, nouveau defaults to GSP on
Turing/Ampere; the direct path is a retained, forceable fallback (`nouveau.config=NvGspRm=0`, and
automatic when GSP firmware is absent). It is stable and proven, but NVIDIA and nova-core are
moving to GSP-only, and **Ada already deleted its non-GSP display HAL**. GA10x is the last family
that keeps a register-level display path.

## What "direct" actually entails

"Direct" is not "plain register pokes." Only SOR / PLL / DP-link / clock setup is bare MMIO. The
**mode-set and scanout themselves flow through the NVDisplay channels — a DMA pushbuffer**:

- Display classes for Ampere (the C670 family): core `GA102_DISP_CORE_CHANNEL_DMA` (`0xc67d`),
  window `0xc67e`, window-immediate `0xc67b`, cursor `0xc67a` (headers `clc67d.h` / `clc67e.h` /
  `clc67a.h` in [open-gpu-doc `classes/display/`](https://github.com/NVIDIA/open-gpu-doc/tree/master/classes/display)).
- The core channel needs **instance memory, a RAMHT, DMA objects, and a channel user-MMIO
  region** ([disp/chan.c](https://raw.githubusercontent.com/torvalds/linux/master/drivers/gpu/drm/nouveau/nvkm/engine/disp/chan.c)).
  The register-level "plumbing" to allocate/kick a channel is in NVIDIA's GA102 display register
  manual: `NV_PDISP_FE_CHNCTL_CORE/WIN/CURS`, `NV_PDISP_FE_PBBASE/PBBASEHI`
  ([dev_display_withoffset.ref.txt](https://raw.githubusercontent.com/NVIDIA/open-gpu-doc/master/manuals/ampere/ga102/dev_display_withoffset.ref.txt)).
- **Mode-set is a method stream** on the core channel: `HEAD_SET_RASTER_*`,
  `HEAD_SET_PIXEL_CLOCK_FREQUENCY`, `HEAD_SET_CONTROL_OUTPUT_RESOURCE`, `SOR_SET_CONTROL`
  (protocol select), viewport/scaler, then `UPDATE`. The window channel points at the scanout
  surface (`SET_CONTEXT_DMA_ISO`, `SET_STORAGE`, `SET_OFFSET`).
- After `UPDATE` you must complete the display **supervisor** interrupt handshake (SV1/SV2/SV3).

**EDID and DisplayPort are a separate subdev you must port.** open-gpu-doc documents *none* of
EDID/DDC/AUX. On the direct path you read EDID in-driver via nouveau's `nvkm/subdev/i2c`: bit-bang
**DDC/I²C at address `0x50`** (E-DDC `0x30`) for TMDS/HDMI, or native **DP AUX** in `i2c/aux.c`
for DisplayPort. DisplayPort **link training** (the `dp.c` `train_cr` / `train_eq` state machine
over AUX — clock recovery, lane/rate, voltage-swing/pre-emphasis) is the single hardest and most
fragile piece; a DVI/HDMI (TMDS) panel avoids it entirely.

## The memory floor (smaller than you'd fear)

Neither route hands you a framebuffer allocator — even GSP-RM does not manage the scanout
framebuffer; the driver owns VRAM and merely tells GSP where its page directory is. But
display-only is a small fraction of a full GEM/TTM stack:

- **Pitch-linear (untiled) scanout is allowed** on nv50→Ampere — the window's storage method has a
  `PITCH` layout mode, so you skip block-linear tiling math
  ([wndwc37e.c](https://raw.githubusercontent.com/torvalds/linux/master/drivers/gpu/drm/nouveau/dispnv50/wndwc37e.c)).
- The window references its surface through a simple **display context-DMA**
  (`SET_CONTEXT_DMA_ISO` + a 256-byte-granular `SET_OFFSET = addr>>8`) — a base/limit descriptor,
  **not** the GPU's 5-level compute page tables. **No full GPU VMM is needed** for scanout.
- The surface must live in **VRAM** in practice (nouveau always pins scanout to VRAM). *Open
  question:* whether GA10x can scan out from a system-memory (GART) surface via a sysmem-target
  ctxdma — which would let danos skip a VRAM allocator. No source forbids it; nouveau never does
  it (confidence: medium).
- **CPU access** to the framebuffer for compositing goes through **BAR1** (a VRAM aperture); BAR0
  is the 16 MB register window. BAR1 can be smaller than 12 GB of VRAM unless Resizable BAR maps
  it all.

**Net:** you need (1) a contiguous aligned VRAM allocator (256-byte base, pitch a multiple of
64 bytes — confirm against the Ampere display refs), (2) a little instmem for the channel
pushbuffers + iso ctxdma, (3) a BAR1 CPU mapping. You do **not** need the 5-level VMM, GEM/TTM
eviction, or tiling.

## Licensing

The tension is the opposite of convenient:

- **NVIDIA open-gpu-kernel-modules is dual MIT/GPLv2** — usable under MIT, no copyleft on your
  other code — **but its display logic is the GSP/RM-object route.** Its class headers
  (`cl0073.h`, `cl2080.h`, `ctrl0073*.h`) are useful, permissive references.
- **nouveau is GPLv2**, and the **register-level display sequences you actually want live in
  nouveau**, not in the MIT code. So the *easy technical path is the GPL-licensed one.* Reading
  GPL nouveau and reimplementing it in Zig is a derivative-work risk proportional to how closely
  your code tracks its structure/constants.

Options: **(a)** accept that the danos NVIDIA display driver is a **GPL component**. danos's
userspace-driver-over-IPC model (a driver is a separate process behind a defined protocol, not
linked into the kernel) is about the cleanest possible GPL boundary, so the GPL would be contained
to that one binary and the rest of danos could keep its own license — but this is a
licensing-boundary judgement that wants real diligence, not a settled fact. **(b)** clean-room
from *specification* rather than *code*: [envytools](https://envytools.readthedocs.io) + NVIDIA's
open-gpu-doc register manuals + the MIT OGKM class headers, treating nouveau as
documentation-of-last-resort.

**Firmware licensing is moot for the direct path** (no firmware is loaded). For completeness: the
GSP blobs are marked redistributable under `LICENCE.nvidia`, which permits use by **any
OSI-approved open-source OS** (not just Linux), on NVIDIA GPUs, **unmodified**, with **no
reverse-engineering of the firmware binary**. The one gate — is danos released under an OSI
license? — is only reached on the GSP route, which this doc recommends against for this card.

## Prior art

**No one has built a from-scratch native NVIDIA driver outside Linux.** FreeBSD ships
`nvidia-drm-kmod`, a *port of NVIDIA's own closed `nvidia-drm.ko`* loading the GSP blob (its old
nouveau port was removed). Haiku's NVIDIA support is likewise a *port of OGKM* (GSP, Turing+, very
alpha). OpenBSD / DragonFly have neither. Every non-Linux OS that supports modern NVIDIA chose to
**wrap NVIDIA's GSP stack** rather than write a native driver. A danos direct-register driver
would have exactly one reference implementation — GPL nouveau — and no non-Linux precedent.

## Alternatives

| Option | What you get | The tradeoff |
|---|---|---|
| **Stay on GOP** (working today) | Native-res scanout, zero GPU code/firmware/maintenance | Resolution frozen at ExitBootServices; **no runtime mode change, no hardware vsync, no multihead** |
| **Pre-Turing NVIDIA** (Kepler / early Maxwell) | Direct EVO/disp-core + CRTC/PLL modeset, **no signed firmware, no coprocessor**; mature nouveau reference | Older display class; not this card; only reclocking is firmware-gated |
| **Intel iGPU** | **Publicly documented** register interfaces (Intel PRMs); no coprocessor mediating modeset | i915 is huge + generation-specific; write one generation from the PRM |
| **Native GA106 direct** (this doc) | Runtime modeset, vsync, multihead on the actual card | Tier-4 effort; GPL reference; DP link training; legacy/de-emphasized path |
| **GA106 via GSP/OGKM** | Also unlocks 3D / reclocking later | Tier-5; ~14k-line ante; unstable version-pinned ABI; unprecedented outside Linux |

## "First light" milestones (direct path, inheriting GOP state)

Framed as a danos `.scanout` service (like the virtio-gpu driver), taking the direct register path
and inheriting the GOP-initialized display — no signed firmware, no devinit, no GSP:

1. **PCI/BAR bring-up** — enumerate GA106 (`0x176`), map **BAR0** (registers) and **BAR1** (VRAM
   aperture) via danos MMIO grants; confirm the display engine is GOP-live.
2. **VRAM + instmem allocator** — contiguous aligned VRAM for the scanout surface (256-byte base)
   + small instmem for pushbuffers / RAMHT / iso ctxdma. No VMM, no TTM.
3. **EDID** — port `nvkm/subdev/i2c` DDC (`0x50`) + DP-AUX (`aux.c`); read + parse the panel EDID.
4. **Core channel up** — allocate the `0xc67d` core channel as a DMA pushbuffer; stand up the
   SV1/SV2/SV3 supervisor-interrupt handshake.
5. **First pixel = reprogram, don't re-POST** — bind a window (`0xc67e`) at the existing WC
   framebuffer via `SET_CONTEXT_DMA_ISO` + `SET_OFFSET`, pitch-linear, `UPDATE`; prove you can
   drive the *current* GOP mode from your own channel before changing anything.
6. **Modeset** — push raster timings on a head, route head→SOR→connector, program the pixel-clock
   PLL, switch to an EDID mode (needs the `clc67d/e` method opcodes from the OGKM headers + the
   supervisor timing from nouveau `head.c`).
7. **DisplayPort link training** — only if the panel is DP and GOP's link can't be reused; the
   `dp.c` `train_cr`/`train_eq` state machine. TMDS/HDMI is far simpler.
8. **Wire into the compositor `.scanout` backend** (`attach_scanout`), add vsync via the display
   interrupt, then multihead.

Keep the GOP backend as the fallback the whole way — a stall at any step still leaves danos with a
working display (exactly the resilience v2 already provides via re-attach).

## Reading list

**Direct path — nouveau (GPLv2):**
- `nvkm/engine/disp/ga102.c` — the GA10x display HAL + the GSP/non-GSP dispatch.
- `nvkm/engine/disp/{head.c, ior.c, dp.c, hdmi.c, chan.c}` — head/SOR routing, DP AUX + link
  training, channel-DMA plumbing.
- `dispnv50/{corec37d.c, corec57d.c, wndwc37e.c, wndwc57e.c, wndwc67e.c, headc37d.c, cursc37a.c}`.
- `nvkm/subdev/i2c` (DDC + `aux.c`) for EDID; `nvkm/subdev/bios/init.c` + `devinit/` **only** if
  you ever have to re-POST (danos's GOP handoff means you shouldn't).

**Object model / GSP path — NVIDIA OGKM (MIT/GPLv2):** class headers `cl0073.h`, `cl2080.h`,
`ctrl0073system.h`, `ctrl0073specific.h`; `src/nvidia/` for RM control sequences.
`nvidia-modeset.ko` (NVKMS) is a *policy* layer over RM and can be bypassed entirely.
[nova-core](https://lore.freedesktop.org/nouveau/) (Rust) is the forward-looking reference for GSP
boot mechanics (falcon signing, queue rings, RPC).

**Register / method specs — NVIDIA open-gpu-doc:**
- [`classes/display/README.txt`](https://raw.githubusercontent.com/NVIDIA/open-gpu-doc/master/classes/display/README.txt)
  — the channel model + class-to-GPU map (read first).
- [`classes/display/clc67d.h`](https://raw.githubusercontent.com/NVIDIA/open-gpu-doc/master/classes/display/clc67d.h)
  + `clc67e.h` / `clc67a.h` — the Ampere core/window/cursor mode-set method vocabulary.
- [`manuals/ampere/ga102/dev_display_withoffset.ref.txt`](https://raw.githubusercontent.com/NVIDIA/open-gpu-doc/master/manuals/ampere/ga102/dev_display_withoffset.ref.txt)
  — `NV_PDISP_FE_*` channel/pushbuffer registers + SOR.
- [`DCB`](https://github.com/NVIDIA/open-gpu-doc/tree/master/DCB) — connector→output-resource
  routing; [`Devinit`](https://github.com/NVIDIA/open-gpu-doc/tree/master/Devinit) +
  [`BIOS-Information-Table`](https://github.com/NVIDIA/open-gpu-doc/tree/master/BIOS-Information-Table)
  — VBIOS parsing (bring-up reference; not needed if inheriting GOP).
- The 632 KB Volta [`dev_display.ref`](https://download.nvidia.com/open-gpu-doc/Display-Ref-Manuals/1/gv100/dev_display.ref)
  is the best shot at SOR-DP/AUX register detail the smaller Ampere file omits.

## Open questions (unresolved by the survey)

Each needs a direct read of the named nouveau file or experimentation on the actual card:

- Exact GA106 register/method offsets and PADLINK→SOR→connector wiring (can vary by board vendor).
- Whether *any* PLL/devinit re-run is unavoidable vs. fully inherited from GOP.
- Whether DisplayPort needs full retraining on takeover, or the GOP-established link can be reused.
- The precise SV1/SV2/SV3 supervisor sequence.
- Whether a system-memory-target scanout ctxdma could eliminate the VRAM allocator.
- The exact `clc67d.h`/`clc67e.h` method opcode numbers (not captured verbatim in the survey).

---

*Research snapshot; verify against current nouveau / open-gpu-kernel-modules source before
building — NVIDIA's GSP defaults and firmware ABIs change per release.*