diff --git a/docs/c-library-compatibility.md b/docs/c-library-compatibility.md new file mode 100644 index 0000000..7c67fe7 --- /dev/null +++ b/docs/c-library-compatibility.md @@ -0,0 +1,205 @@ +# The C library compatibility layer + +A design note and milestone plan for **libdanos-c** — the mini C library that lets +`zig cc` cross-compile C programs for danos. It is milestone **P0** of +[python-on-danos-milestones.md](python-on-danos-milestones.md), expanded here the +way [character-devices-and-tty.md](character-devices-and-tty.md) expands P1. +CPython is the driving consumer, but the layer is general: any portable C program +within its surface should build. + +## What it is — and the three things it is not + +The deliverable is a **sysroot**: a set of C headers plus a static `libdanos-c.a`, +handed to `zig cc -target x86_64-freestanding-none` via `-isystem` and linked into +every C binary. Three explicit non-goals keep it small: + +- **Not a musl port.** Whole-musl assumes Linux syscall semantics at its bottom + (the door the Zig roadmap deferred, twice now). We *lift* musl's pure-computation + source files and *write* a danos-native bottom — see the layer split below. +- **Not full POSIX — *yet*.** Stage 1's surface is "what CPython's minimal + configuration and ordinary portable C need" — roughly 100–150 functions — and + at that stage absence is a *feature*: configure scripts probe and adapt, and a + linker error is honest. But the end state is a **full C compatibility layer** + (see "The road to full coverage" below); the absence table is a schedule of + arrivals, not a wall. +- **Not a second runtime.** The library is a thin C-ABI re-spelling of the same + danos-native surface `runtime` already provides. It contains no policy of its + own; when the Zig track's `runtime.os` seam is authored, the libc bottom + re-targets it near-mechanically — the fourth appearance of the roadmap's "same + surface" symmetry. + +One scoping rule sits above all three — the **size doctrine**: this layer serves +**applications only**. The kernel and the system services never link libdanos-c; +they stay danos-native Zig over `runtime`, small and static, because leanness is +an operating-system property. Applications have their own budget and may be as +big as they need to be. The libc is how big software *lands on* danos, never how +danos itself is built. + +## The layer split: lift the mathematics, write the plumbing + +The realization that makes 100–150 functions tractable: a libc is two very +different kinds of code, and the hard kind is portable. + +| Layer | Contents | Source | +|-------|----------|--------| +| **Pure computation** | `string.h`/`memcpy` family, all of libm, `strtod`/`dtoa`, `strtol`, `qsort`, `ctype` tables, `gmtime` calendar math, the `printf`/`scanf` engines, `setjmp` (a dozen instructions of x86-64 asm) | **Lift from musl**, vendored under `library/c/third-party/musl/` (MIT; files compile standalone) | +| **OS plumbing** | fds (`open`/`read`/`write`/`close`/`lseek`/`stat`/`getcwd`/`chdir`/`isatty`), `mmap`/`munmap`, clocks, `exit`, `getenv`, `getentropy` | **Write in Zig**, exporting C ABI over the `runtime` syscall + VFS client surface | +| **The middle** | `malloc` over danos `mmap` (simple free-list; CPython's arenas sit above), `FILE*` buffering, `errno` | **Write in Zig** (small, danos-shaped) | +| **Entry** | `crt0`: the existing danos entry shim ([sysv.md](os-development/sysv.md)) bridged to C `main(argc, argv, envp)`, `environ` initialised, `exit` flushing stdio | **Write** | + +Two liftings deserve their own line because getting them wrong is silent +corruption rather than a linker error: + +- **`strtod`/float formatting.** Python's float `repr` guarantees shortest + round-trip; that property lives entirely in these routines. musl's are correct; + an improvised one would be subtly wrong for years. Lift, never write. +- **The stdio engines.** musl's `vfprintf`/`vfscanf` are self-contained around + its `FILE` abstraction (function-pointer read/write slots), so the whole + formatted-I/O engine lifts too — we implement only the fd-backed slots + (`__stdio_write`-shaped) and the buffering glue. + +## Header policy + +Hand-write the headers as danos's own minimal set rather than importing musl's +(musl's are entangled with Linux ABI details), borrowing declarations freely. +Freestanding compiler headers (`stdint.h`, `stddef.h`, `stdarg.h`, `stdbool.h`, +`float.h`, `limits.h`) come from clang via `zig cc` — do not duplicate them. +`errno.h` values are the danos errno enum re-spelled with POSIX names; there is no +Linux numbering to be compatible with, so the enum is the truth. + +Deliberate absences, and their planned arrivals — this table is the +compatibility matrix, and "the road to full coverage" below is the schedule +that empties it: + +| Absent | Arrives with | +|--------|--------------| +| `pthread.h` | the post-P5 pthread subset over `thread_spawn`/futex — but see the risk below | +| real `signal.h` (beyond no-op `signal()`/`raise` stubs) | M17 signals-over-IPC in the libc | +| `dlfcn.h` | [dynamic-libraries.md](dynamic-libraries.md) D1 | +| `fork`/`exec*`/`wait*` | P5 exposes danos spawn as `posix_spawn`; `fork` itself never (see below) | +| `socket.h` | a future networking track | +| locale beyond `"C"` | stage 3 evaluation (CPython is UTF-8-mode happy without it) | +| pipes (`pipe()`) | P5 process-control cluster | + +## The road to full coverage + +The layer grows in three stages; only stage 1 is a current milestone (P0), but +the stages exist so stage-1 decisions never have to be unmade: + +- **Stage 1 — CPython-minimal** (P0, the slicing below): ~100–150 functions, + static-only, absences honest. +- **Stage 2 — the danos-complete layer**: the full hosted C11 standard library, + plus every POSIX facility danos semantics support, landing as its enabling + milestone lands — pipes and `posix_spawn` at P5, real signals at M17, the + pthread subset after P5, `dlfcn.h` at + [dynamic-libraries](dynamic-libraries.md) D1, sockets with networking. Stage 2 + is not one milestone but the standing rule that **every system capability + gets its C spelling when it ships**, so the matrix above drains as the OS + grows. +- **Stage 3 — ecosystem grade**: the point where "portable C program" generally + means "builds on danos" (autotools-style probing included). Reaching it is + mostly stage 2 compounding, plus the long tail (locale, wide-char, + `fnmatch`/`glob`/`regex` — the last three lift from musl like the rest). At + this stage, re-evaluate hand-grown-vs-musl-port once with real data; the + standing recommendation remains danos-native — musl's bottom assumes Linux + syscall semantics, and by stage 3 the danos bottom exists and is tested — + with musl continuing as the quarry for computation code. + +Two boundaries are permanent and worth stating at every stage: **`fork` never +comes** — danos is a spawn-shaped OS, and `fork`'s address-space-duplication +semantics are hostile to everything from capabilities to threads; software that +hard-requires `fork` (not `posix_spawn`) stays off the platform. And the +**public ABI stays the vDSO + IPC protocols** — a full libc is a compatibility +*layer*, not a second stable system ABI. + +## Milestone slicing + +1. **sysroot-skeleton** — layout under `library/c/` (a build package: + `include/`, Zig sources, vendored musl subtree); `crt0`; string/mem + + `ctype` lifted; a `build.zig` step making C binaries first-class targets. + *Test:* a C program using only computation links and runs in QEMU + (`c-hello` printing via a raw `write` extern to `debug_write`). +2. **fd-plumbing** — `errno`; open/read/write/close/lseek/stat/unlink/mkdir/ + rename over the `runtime` VFS client; `getcwd`/`chdir`/`getenv`/ + `getentropy` arriving as P1 lands them (stubbed truthfully until then: + `getenv` empty, `getentropy` `ENOSYS`). *Test:* QEMU `c-file-io` — create, + write, reopen, read back, stat size + mtime through FAT. +3. **malloc** — free-list allocator over danos `mmap`; `calloc`/`realloc`/ + `free`; alignment guarantees documented. *Test:* host + QEMU allocator + torture (interleaved sizes, realloc growth, alignment asserts). +4. **stdio** — `FILE*`, buffering modes, the lifted printf/scanf engines wired + to the fd slots; `snprintf` family; stdin/stdout/stderr over fd 0/1/2. + *Test:* host round-trip suite for format engines (especially `%.17g` + float round-trip); QEMU `c-stdio` cooked-line echo once P1's console exists. +5. **mathematics-and-time** — libm lifted wholesale; `strtod`/`strtol`; + `clock_gettime` (monotonic + realtime over `clock`/`wall_clock`); + `gmtime`/`mktime`/`strftime` (UTC only — no timezone database); + `setjmp`/`longjmp`; `qsort`/`bsearch`; `abort`/`assert`. *Test:* host + `strtod`/`dtoa` vectors against known-hard cases; QEMU `c-time` sanity + against the wall clock. + +Slices 1, 3, 4-host, and 5-host have **no dependency on P1** and can start +immediately; slice 2 and the QEMU halves interleave with P1 as it lands. + +**Exit for the layer as a whole** (= P0's exit): `c-hello` and `c-file-io` green +in the QEMU suite, and the host-side computation tests green — at which point P2 +(CPython configure) becomes the layer's real integration test. + +## Testing strategy: two targets, on purpose + +The computation layer is target-independent, so it is unit-tested **on the host** +(built for the host triple, compared against the host libc's answers — +thousands of cheap oracle checks for `strtod`, `printf`, libm edge cases). The +plumbing layer only means anything **on danos**, so it is tested in the QEMU +suite like every other subsystem. Keeping the split explicit stops the slow-QEMU +suite from absorbing tests that a host `zig test` runs in milliseconds. + +## Risks and gotchas + +- **CPython's configure may insist on pthreads.** WASI-class targets build + threadless, but verify this *first* in P2 bring-up; the fallback is a + truthfully-single-threaded `pthread.h` stub set (create returns `EAGAIN`, + mutexes are no-ops — valid when only one thread can exist). Decide from + evidence, not assumption. +- **`long double` is x87 80-bit on x86-64.** musl's libm handles it, but keep + CPython away from it (`configure` uses `double` throughout by default); + don't hand-write anything touching x87. +- **errno is a contract, not a convention.** The Zig plumbing must map every + `runtime` error to a POSIX name consistently — CPython turns errno into + exception types (`FileNotFoundError` is `ENOENT`). One table, tested. +- **`malloc` alignment**: 16-byte minimum on x86-64 (SSE spills in + compiled C). The free-list must guarantee it from day one; retrofitting + alignment bugs out of an allocator is misery. +- **Vendoring discipline.** The musl subtree is lift-only — never edited in + place (patches live beside it if ever needed), pinned to one musl release, + with the file list documented so a version bump is a re-copy, not an + archaeology dig. +- **stdio buffering vs. crashes.** Buffered stdout + a crashing program eats + output — the classic debugging trap. `stderr` stays unbuffered (per C + standard) and `exit`/`abort` flush; document that `_exit` does not. + +## Decisions needing sign-off + +- **Lift-from-musl for all pure computation** (vendored, pinned, unedited) rather + than writing or porting whole-musl. +- **Hand-written danos-native headers**; danos errno values are the numbering. +- **`library/c/` as a build package** producing both the sysroot and the + first-class C-binary build step. +- The **deliberate-absence table** as the living compatibility matrix, drained + by the three-stage road above — with exactly one permanent "never": `fork`. +- **Full coverage as the end state** (stage 3), reached by the standing rule + that every system capability ships with its C spelling — not by a musl port. + +## Related + +- [python-on-danos-milestones.md](python-on-danos-milestones.md) — this is P0. +- [dynamic-libraries.md](dynamic-libraries.md) — ships in this sysroot + (`dlfcn.h` + the loader) once its D1 lands. +- [python-on-danos.md](python-on-danos.md) — the design note that scoped the + layer. +- [character-devices-and-tty.md](character-devices-and-tty.md) — P1; supplies + the console that makes stdio interactive. +- [zig-self-hosting.md](zig-self-hosting.md) — the `runtime.os` seam the + plumbing layer will re-target when it exists. +- [os-development/sysv.md](os-development/sysv.md) — the entry stack `crt0` + bridges. diff --git a/docs/character-devices-and-tty.md b/docs/character-devices-and-tty.md new file mode 100644 index 0000000..9c2635f --- /dev/null +++ b/docs/character-devices-and-tty.md @@ -0,0 +1,166 @@ +# Character devices, the console, and the tty question + +A design note for the **stream** half of the device world. danos has block devices +(the USB storage service behind the FAT mount) but no character devices — and three +tracks now need them at once: the terminal application, Zig self-hosting Phase 1 +("wire fd 0/1/2 to a console byte stream"), and [Python on danos](python-on-danos.md) +Phase 1. This note settles what a character device *is* on danos before any of those +tracks build one. + +## The Unix picture, briefly + +Unix splits devices in two: **block devices** are seekable arrays of fixed-size +sectors (disks); **character devices** are unseekable byte streams (keyboards, +serial ports, terminals, `/dev/null`, entropy). A **tty** is the canonical +character device — a byte stream plus a *line discipline* (echo, line buffering, +erase handling, Ctrl-C-to-signal) that lives in the kernel. A **pty** is a pair of +character devices (master/slave) that exists so a *userspace* program — a terminal +emulator — can impersonate terminal hardware to the kernel's in-kernel line +discipline. + +The identification asked for and confirmed: yes, tty and pty are character +devices in this taxonomy. + +## The realization that shapes everything: danos already has the mechanism + +A Unix character device is an in-kernel dispatch table: major/minor numbers route +`read()`/`write()` to a driver. danos already has exactly that dispatch — the VFS: +`fs_resolve` routes a path to a mounted backend service, and `Operation.mount` +attaches a backend *endpoint* at a prefix. What is missing is not a device model; +it is **one node kind with stream semantics**. And the protocol already reserved +it: `NodeKind.character_device = 2` sits unimplemented in +[vfs-protocol.zig](../library/protocol/vfs/vfs-protocol.zig), exactly like +`symbolic_link`. + +So the design is small: + +**A character device on danos is a VFS node, served by an ordinary service over +the existing VFS wire protocol, whose read/write have stream semantics.** + +No device numbers, no `/dev` special casing, no new syscalls, no new protocol — +a service mounts itself at a path (per the FSH, e.g. `/device/console`), clients +open it with `runtime.fs` like any file, and `FileStatus.kind` says what it is. + +### Stream semantics (the actual contract change) + +For a node whose kind is `character_device`: + +- **`offset` is ignored** on read and write; there is no seek position. (`lseek`, + when the C layer exists, returns `ESPIPE`.) +- **Reads block** until at least one byte is available, then return what is there — + **short reads are normal**, not EOF. A zero-length read reply means the stream + is closed (hangup), not end-of-file-at-size. +- **`FileStatus.size` is 0** and means nothing; `mtime` may be 0. +- Writes may be short if the service's buffer is full; the client loops as it + already must for the 256-byte message cap. + +This is a semantics note on existing operations, not a wire change — the `Request` +and `Reply` structs are untouched. The one true protocol addition is a **`control` +operation** (appended to `Operation`, values stable): a typed request the stream's +service interprets. Deliberately *not* an `ioctl` grab-bag — the control payloads +are enumerated per protocol, starting with the terminal set below. + +## The first character device is a pseudo-device + +The first device is deliberately **not hardware**: an in-memory **loopback** — a +byte queue served over the stream contract, where bytes written to one end are +read from the other. It is the reference implementation of the semantics above +(blocking reads, short reads, hangup on close, the `control` round-trip), it +tests deterministically with no QEMU serial scripting, and it keeps hardware off +the critical path entirely. `null` and `zero` come along nearly for free as +degenerate cases. This is a decision, not a convenience: the dead-COM1 boot bug +on real hardware already proved serial cannot be assumed present or alive, so +**nothing in this milestone writes to COM1**. (A serial-backed stream node can +exist *later* as one more optional backend for headless debugging; it is on +nobody's critical path.) + +The loopback is also not throwaway — it is the seed of P5's `pipe()`, which is +the same object with two fds. + +## The console service + +A `console` service owns the line discipline — **in userspace**, where a +microkernel wants it, not in the kernel as Unix has it: + +- **The discipline is a pure library first**: bytes and key events in, bytes + out, no I/O of its own — developed and host-tested against in-memory buffers, + then shared verbatim between the console and the future terminal application. +- **Input**: subscribes to keyboard `InputEvent` IPC (the structured events that + exist today) and cooks them into bytes. Cooked mode is the default: echo, line + buffering, backspace/erase, so a line is delivered on Enter. Raw mode delivers + bytes as they come (the REPL's line editor and any full-screen program need it). +- **Output is a pluggable sink**, and the stream contract is independent of it: + the bring-up sink is in-memory (readable back by tests, mirrored to the boot + log), and the real one is the framebuffer text renderer when the display + track's font work lands. +- **Control set** (the `control` payloads): mode raw/cooked, echo on/off, and + window-size query — the minimal termios. Ctrl-C-to-signal joins when M17 + signals-over-IPC lands; until then Ctrl-C is just a byte. +- Mounts itself at `/device/console` as a `character_device` node. + +**fd 0/1/2** then stop being special: spawn hands the child three open handles +(console by default; anything else if the parent chooses), and `runtime`'s fd +table maps 0/1/2 to them. `isatty` is simply "does `status` say +`character_device`" — no side channel needed. + +## The pty answer: there is no pty + +The pty exists in Unix *because the line discipline is in the kernel* — userspace +terminal emulators need a kernel gadget to impersonate hardware. On danos the +terminal emulator is already a userspace server, so the pair collapses: + +**The graphical terminal application serves the VFS stream protocol itself and +hands its own endpoints to the children it spawns as their fd 0/1/2.** + +The terminal *is* the console service for its children — same protocol, same +control set, same line discipline code (shared as a library with the boot +console). No master/slave device pair, no `/dev/pts`, no new kernel object. When +CPython arrives, the libc's `isatty`/read/write see a character device and are +none the wiser; when xonsh eventually wants job control, that lands as control +messages + M17 signals, still with no pty object. + +What this costs: programs that *specifically* manipulate Unix ptys +(`os.openpty()`, `pexpect`-style tools) have no direct equivalent — the danos +answer is "spawn the child yourself with your own stream endpoints," which is the +same capability with less machinery. Accepted. + +## Milestone slicing + +1. **pseudo-devices** — VFS honors `character_device` semantics end to end; + `Operation.control` added; the in-memory **loopback** (plus `null`/`zero`) + as the first device. QEMU test: one client writes, another reads — open, + offsetless read/write, blocking read, short read, hangup on close, control + round-trip. No hardware anywhere. +2. **console-service** — the line-discipline library (host-tested, pure) plus + the console composing keyboard `InputEvent`s with an in-memory output sink; + mounted at `/device/console`. QEMU test injects key events and reads cooked + lines and raw bytes back through the sink. +3. **fd-inheritance** — spawn passes 0/1/2 handles; `runtime` fd table; `isatty` + via `status`; existing binaries' stdout migrates from `debug_write` to fd 1 + (the logger keeps its own path). +4. **terminal-as-server** — deferred to the terminal application milestone + (Python track P3): the terminal reuses the discipline library and serves its + children directly. + +Steps 1–3 are exactly the shared seam that Zig self-hosting Phase 1 and Python +Phase 1 both list; neither track repeats them. + +## Decisions needing sign-off + +- **No pty object; the terminal serves its children directly** (the section + above) — the load-bearing simplification. +- **`control` as an enumerated, typed operation** rather than an ioctl-style + opaque pass-through. +- **Line discipline in userspace services** (console + terminal, shared library), + never in the kernel. + +## Related + +- [python-on-danos.md](python-on-danos.md) — consumes this as its Phase 1. +- [zig-self-hosting.md](zig-self-hosting.md) — ditto ("stdio as fds"). +- [file-system-development/vfs-protocol.md](file-system-development/vfs-protocol.md) — + the wire protocol this note extends. +- [file-system-development/danos-file-system-hierarchy-FSH.md](file-system-development/danos-file-system-hierarchy-FSH.md) + — where `/device/console` lives. +- [device-driver-development/input.md](device-driver-development/input.md) — the + `InputEvent` stream the console cooks. diff --git a/docs/dynamic-libraries.md b/docs/dynamic-libraries.md new file mode 100644 index 0000000..820f311 --- /dev/null +++ b/docs/dynamic-libraries.md @@ -0,0 +1,124 @@ +# Dynamic libraries on danos + +A design note and milestone plan for shared objects: building them, loading them +with `dlopen`, and — the part that needs kernel work — actually *sharing* them +between processes. Directional, post-P5 of +[python-on-danos-milestones.md](python-on-danos-milestones.md); nothing on the +CPython bring-up path depends on it. + +## Reconciling the earlier "rejected" + +Dynamic libraries were evaluated once before and rejected — but as an answer to a +*different question*: whether they could claw back ReleaseSafe's measured ~2× +code size. They cannot (the safety checks inline at every call site; no library +scheme dedups them), and that verdict stands for that question. The reasons to +build them now are the ones that investigation never weighed: + +- **`ctypes` and runtime FFI** — Python calling into a danos library without + rebuilding the interpreter. This is the piece that makes Python prototyping + self-serve: drop a `.so` on the image, `ctypes.CDLL` it, iterate. +- **Loadable CPython extension modules** — today every C extension means + relinking the interpreter (`Modules/Setup`); with `dlopen`, an extension is a + file. +- **One interpreter image, many Python services** — a statically-linked CPython + is tens of megabytes *per process*. A shared `libpython` mapped read-only once + (milestone D3 below) makes Python services cheap enough to be the default way + to prototype one. +- **Plugin-shaped applications** — the UI toolkit and the terminal will want + them eventually. + +The scoping that dissolves the apparent contradiction is the **size doctrine**: +leanness is an *operating-system* property — the kernel and system services stay +small and statically linked, and none of them ever link the loader — while +*applications* have their own budget and may be big. Dynamic libraries are an +**application-layer facility**, full stop. + +What also does **not** change: the public ABI stays the vDSO + the IPC +protocols. Shared objects are artifacts *within* one system image, versioned by +the build — not a new stable ABI surface for the OS. + +## Design + +- **Format and codegen are free.** ELF shared objects with position-independent + code; `zig cc -fPIC -shared` against the [libdanos-c](c-library-compatibility.md) + sysroot already emits them. The work is entirely on the loading side. +- **The loader lives in userspace, inside the libc.** `dlopen` reads the `.so` + through the VFS, maps its segments, applies relocations, resolves symbols + against the process and the `DT_NEEDED` dependency graph, runs constructors, + returns a handle. No kernel loader changes in v1 — segments land in anonymous + `mmap` as private copies. +- **Bind-now, always.** All relocations resolved at `dlopen` time + (`RTLD_NOW` semantics only). Lazy PLT binding buys startup latency danos does + not care about, at the price of a writable GOT dance and a much subtler + loader. Not worth it; keep it out permanently. +- **W^X from day one.** Map, relocate, then flip text pages read-execute — + which requires memory-protection change (`mprotect`-shaped) in the danos + `mmap` surface if it is not already there. No page is ever writable and + executable at once. +- **TLS in shared objects is deferred.** Thread-local storage models + (initial-exec vs. general-dynamic) are the deep end of every dynamic linker. + v1 refuses a `.so` with a TLS segment; revisit alongside the post-P5 pthread + subset, which is when it could matter. +- **Executables stay static until D4.** v1 is "a static binary that can + `dlopen`" — no `PT_INTERP`, no program interpreter, no dynamically-linked + `main` binaries. That keeps process startup untouched. + +## Milestones + +1. **D1 — dlopen in-process.** The `.so` build target; the loader in libdanos-c: + map, relocate (`RELATIVE`/`GLOB_DAT`/`JUMP_SLOT`), resolve, constructors; + `dlopen`/`dlsym`/`dlerror`/`dlclose`; private anonymous mappings; no TLS. + *Test:* QEMU `dlopen-hello` — load a `.so`, call a symbol, unload, reload. +2. **D2 — the FFI payoff.** `DT_NEEDED` dependency graphs; a **libffi port** + (x86-64 SysV assembly is upstream; the port is its closure-allocation paths, + which must respect W^X); CPython's `ctypes` enabled; extension modules + loadable from file. *Test:* QEMU — a Python script `ctypes.CDLL`s a danos + `.so` and round-trips a call; a `.so` extension module imports. +3. **D3 — actual sharing (the kernel milestone).** Shared read-only file-backed + mappings — a page-cache-shaped facility so N processes mapping `libpython` + hold one physical copy. This is the memory-win milestone and the only one + touching the kernel; design it with the existing shm machinery in view + (the shared-fate walks already locked the relevant paths). *Test:* N Python + services up; measure physical pages against N× the static baseline. +4. **D4 — dynamically-linked executables** (optional, evaluate after D3): + `PT_INTERP`, a danos program interpreter, and the spawn path teaching the + loader about it. Only worth it if the image-size or update story demands it. + +## Risks and gotchas + +- **Scope creep is the failure mode.** Every dynamic linker grows toward glibc. + The fences: bind-now only, no lazy binding ever, no TLS until pthreads demand + it, no dlopen-from-memory, no versioned symbols. Each fence removed is a + design discussion, not a patch. +- **Code loading is a security event.** `dlopen` turns file bytes into executable + code, so W^X discipline is table stakes and *what may be dlopened* is a + capability question — the natural danos answer is that loadability follows VFS + readability of the `.so`, and services' images are supervised like any other + artifact. Revisit explicitly at D3 when mappings become shared. +- **`dlclose` is where loaders go to die.** Constructors/destructors, + dangling function pointers, re-open identity. Keep v1 semantics honest and + simple: `dlclose` runs destructors and unmaps; holding pointers past it is + undefined; no reference-counted deferral cleverness. +- **The ReleaseSafe fact still applies to `.so`s** — a ReleaseSafe shared object + carries its inlined checks like any static code; D3's sharing saves *copies*, + not check overhead. Size expectations should be set accordingly. + +## Decisions needing sign-off + +- Dynamic libraries join the roadmap at all (this note exists because the + earlier size-motivated rejection was re-opened for ABI/sharing reasons). +- **Bind-now only; no lazy binding, permanently.** +- **Loader in userspace libc; kernel involvement only at D3** (shared read-only + mappings). +- **Static executables until D4**, and D4 only on demonstrated need. + +## Related + +- [c-library-compatibility.md](c-library-compatibility.md) — the sysroot the + loader ships in; its absence table gains `dlfcn.h` at D1. +- [python-on-danos.md](python-on-danos.md) — the `ctypes` story this unlocks. +- [python-on-danos-milestones.md](python-on-danos-milestones.md) — sequencing; + this work is post-P5. +- [os-development/memory-map.md](os-development/memory-map.md) / + [os-development/paging.md](os-development/paging.md) — where W^X and shared + mappings land. diff --git a/docs/python-on-danos-milestones.md b/docs/python-on-danos-milestones.md new file mode 100644 index 0000000..3a02886 --- /dev/null +++ b/docs/python-on-danos-milestones.md @@ -0,0 +1,192 @@ +# Python on danos: the milestone plan + +The execution plan for [python-on-danos.md](python-on-danos.md). That note holds +the *why* and the design decisions; this one slices the work into milestones with +concrete deliverables, tests, and exit criteria. Milestones are numbered **P0–P5** +(track-local — the global M-series stays with the driver/lifecycle tracks). + +Dependencies at a glance: + +``` +P0 toolchain + mini-libc ──┐ +P1 streams + console + seam ─┴─→ P2 CPython minimal ─→ P3 terminal + REPL + │ │ + └─→ P4 danos module │ + + Python service│ +P5 process control + shell ←─────────────────────────────────┘ +``` + +P0 and P1 are independent of each other and can proceed in parallel. P1 is shared +work — it is also Zig self-hosting Phase 1 and the first three slices of +[character-devices-and-tty.md](character-devices-and-tty.md). + +## P0 — Toolchain + the C library compatibility layer + +**Goal:** a C hello-world, cross-compiled on the host with `zig cc`, runs on danos. + +Design and slicing live in +[c-library-compatibility.md](c-library-compatibility.md): the **libdanos-c** +sysroot (hand-written danos-native headers + `libc.a`) as a `library/c/` build +package — pure computation (string, libm, `strtod`, the printf/scanf engines) +lifted from a vendored, pinned musl subtree; the OS plumbing written in Zig over +the `runtime` surface (re-targeting `runtime.os` when the Zig track authors it); +`malloc` over danos `mmap`; a `crt0` bridging the danos entry shim to C `main`. +Driven by `zig cc -target x86_64-freestanding-none -isystem` (the triple becomes +`x86_64-danos` if the Zig fork lands first; nothing else changes). + +Its five slices (sysroot-skeleton, fd-plumbing, malloc, stdio, +mathematics-and-time) carry their own tests — host-side oracle suites for the +computation layer, QEMU cases (`c-hello`, `c-file-io`, `c-stdio`, `c-time`) for +the plumbing. + +**Exit:** `c-hello` and `c-file-io` green in the QEMU suite; host computation +tests green. + +## P1 — Stream nodes, console, and the seam pieces + +**Goal:** the shared Phase-1 surface exists: byte-stream stdio, cwd, environment, +entropy. Design and slicing live in +[character-devices-and-tty.md](character-devices-and-tty.md); this milestone is +its slices 1–3 plus three small seam pieces: + +- **cwd/chdir** — per-process current directory used by path resolution (the + kernel already anchors a VFS root per `fs_resolve`; the cwd is the same idea, + process-scoped, with `getcwd`/`chdir` exposed through `runtime` and the libc). +- **Environment** — spawn carries an environment block; the SysV entry stack's + `envp` slot ([sysv.md](os-development/sysv.md)) stops being empty; `getenv` + reads it. An empty block stays valid. +- **Entropy** — a kernel `entropy` syscall (RDSEED/RDRAND with a jitter fallback, + mirroring the TSC-reliability posture of not trusting one CPU feature blindly); + the libc exposes `getentropy`. + +- **Tests.** QEMU: the character-device tests from the tty note (offsetless + read/write, blocking read, cooked/raw control round-trip), plus `cwd-basics` + (chdir + relative open), `env-roundtrip` (spawn with env, child reads it), + `entropy-sane` (nonzero, changing, correct length). + +**Exit:** a C program reads a cooked line from fd 0 and echoes it to fd 1 — +injected key events in, bytes read back through the console's in-memory sink, +all under QEMU with no hardware involved — and `getcwd`/`getenv`/`getentropy` +return real answers. + +## P2 — CPython, minimal configuration + +**Goal:** `python -c 'print(2**100)'` runs on danos under QEMU. + +- Pin **CPython 3.13.x**; vendor as `third-party/cpython/` or fetch via the build + (decide with the build-packages conventions). +- Host build-Python of the same version (`--with-build-python`). +- `config.site` cache for the cross answers; `config.sub` patch so + `x86_64-unknown-danos` parses; a small `configure`/`pyconfig` patch set kept as + rebasable diffs, WASI-style. +- `--disable-shared`; static `Modules/Setup`: `posix errno _io _codecs _weakref + time math _stat _collections itertools _functools _locale _sre` plus what the + interpreter core insists on; threadless build (WASI precedent). +- `Lib/` on the FAT image under the FSH (e.g. `/system/python/lib`); + `PYTHONHOME` set accordingly; `.pyc` written with **checked-hash + invalidation** (FAT's 2-second mtime granularity makes mtime-based validation + lie during fast edit-run cycles). +- `PYTHONHASHSEED` pinned only if P1's entropy slipped — otherwise real + hash randomization from day one. +- **Tests.** QEMU: `python-expr` (the exit criterion), `python-file` (run a + script from FAT, write a file, read it back), then a curated slice of CPython's + own suite (`test_int`, `test_float`, `test_io`, `test_dict`) as a + longer-running target — the suite is the porting harness. + +**Exit:** the four QEMU cases green; the CPython test slice green or with a +short, documented skip list. + +## P3 — Terminal + REPL: the first real application + +**Goal:** an interactive `python` REPL in a graphical danos terminal — the +milestone demo for the OS. + +- Depends on the display track's font rendering (its stated next step) — until + that lands, the REPL is exercised end-to-end through the pseudo-device + harness from P1+P2 (scripted input in, output read back), so P2's exit is + never blocked on graphics; the graphical terminal is the *interactive* debut. +- The terminal application: draws with the UI toolkit / display client, consumes + keyboard `InputEvent`s, and — per the tty note's load-bearing decision — + **serves the VFS stream protocol itself** to its children, reusing the console's + line-discipline library. Spawns `python` with its endpoints as fd 0/1/2. +- Raw mode + the control set give the REPL line editing; window-size control + gives it wrapping. +- **Tests.** QEMU: scripted terminal session (inject key events, assert rendered + or captured output). Real-hardware smoke on the Intel box joins the existing + checklist. + +**Exit:** typing `2+2` into the terminal on the QEMU GPU target prints `4`. + +## P4 — The `danos` extension module + a Python service + +**Goal:** Python can speak danos: IPC, capabilities, spawn. + +- The `danos` module, **written in Zig against `Python.h`**, statically linked + via `Modules/Setup`: endpoints (create/send/receive), capability passing, + spawn + exit-notification, and the service bootstrap (announce, supervision + handshake) — the same surface Zig services use, re-exposed. +- UI-toolkit bindings as a second module once the toolkit's API settles. +- Prototype **one real service in Python** — policy-shaped, not data-plane + (candidates: hot-plug policy, a settings service) — speaking an existing wire + protocol, supervised by the device manager like any service. +- **Tests.** QEMU: `python-ipc-echo` (Python service echoes over an endpoint, a + Zig client asserts), plus the prototype service's own protocol test. + +**Exit:** a Python process runs as a supervised danos service exchanging IPC +with Zig peers. + +## P5 — Process control, then the shell + +**Goal:** danos can spawn arbitrary programs with arguments and pipes; a small +Python shell uses it. + +The kernel/VFS cluster a shell forces (any shell, any language): + +- **exec-of-path** — spawn an arbitrary VFS path, not a named ramdisk binary; +- **argv/envp** — carried through spawn onto the child's entry stack (env from + P1, argv new); +- **numeric exit status** — extend the exit record beyond the categorical + `ExitReason` (the gotcha the Zig roadmap flagged: `WEXITSTATUS` must be real); +- **fd inheritance + pipes** — a kernel or service pipe (a character device by + the tty note's definition) and spawn-time fd mapping. + +Then, in order: `subprocess` enabled in CPython (maps onto spawn + the +exit-notification endpoint — no fork, Windows-style); a **small Python shell** (a +few hundred lines over `subprocess` + the console: prompt, argv parsing, pipes, +cwd) as the forcing function that reveals what job control actually needs. + +**Explicitly deferred past P5:** the pthread subset over `thread_spawn`/futex, +signals-in-libc via M17, termios job control (Ctrl-C to foreground child), and +**xonsh** — which wants all three and is the arc's endpoint, not a milestone. + +**Tests.** QEMU: `spawn-argv-exit` (child echoes argv, exits 42, parent sees +42), `pipe-through` (parent → child → parent), `python-subprocess`, and a +scripted shell session. + +**Exit:** the Python shell runs `program | program` typed at the terminal and +reports the exit status. + +## Post-P5 outlook + +Two tracks continue past this plan, each with its own design doc rather than a +P-number here: + +- **Dynamic libraries** ([dynamic-libraries.md](dynamic-libraries.md), D1–D4) — + an application-layer facility (the OS stays static and lean): `dlopen` in the + libc, then libffi + `ctypes` + loadable extension modules, then shared + read-only mappings so N Python services hold one physical `libpython`. +- **The full C compatibility layer** + ([c-library-compatibility.md](c-library-compatibility.md), stages 2–3) — the + standing rule that every system capability ships with its C spelling, draining + the absence table toward "portable C builds on danos"; `fork` is the one + permanent exception. + +## Related + +- [python-on-danos.md](python-on-danos.md) — the design note this executes. +- [c-library-compatibility.md](c-library-compatibility.md) — P0's design. +- [character-devices-and-tty.md](character-devices-and-tty.md) — P1's design. +- [zig-self-hosting.md](zig-self-hosting.md) — shares P1; its fork makes P0's + triple prettier but gates nothing here. +- [os-development/process-management.md](os-development/process-management.md) — + the spawn/exit surface P5 extends. diff --git a/docs/python-on-danos.md b/docs/python-on-danos.md new file mode 100644 index 0000000..fc464c1 --- /dev/null +++ b/docs/python-on-danos.md @@ -0,0 +1,261 @@ +# Python on danos: the CPython milestone + +A design note (not built yet) on bringing **CPython** to danos, compiled with the Zig +toolchain (`zig cc`). Like [zig-self-hosting.md](zig-self-hosting.md), it is +forward-looking: it sets a direction and the decisions that follow from it. + +## Why Python, and why now + +The Zig self-hosting road is gated on a compiler fork and a long std-library seam. +Python is the **stop-gap that removes the wait**: a working CPython gives danos a way +to write programs — services, tools, application prototypes — *without* the Zig +compiler being self-hosted, and it brings the pure-Python package ecosystem along as +a bonus. The intended division of labour: + +- **Zig** — the kernel, drivers, and anything on a data plane (interrupt paths, + DMA rings, block I/O). Unchanged. +- **Python** — the control plane and the prototyping surface: services that are + event loops over IPC, policy logic that changes often, application experiments, + and eventually the shell. + +Python is also the scripting language for the terminal-and-shell arc: the first +real danos application is planned as a terminal, a terminal wants a shell, a shell +wants a scripting language — and [xonsh](https://xon.sh) (a shell written in +Python) marks where that road can end. + +### Non-goals + +- **No drivers in Python.** Interrupt handling, ring management, and DMA stay in + Zig. Python may *supervise and configure* drivers; it does not sit in their hot + paths (interpreter overhead and garbage-collection pauses in an interrupt path + are disqualifying). +- **No dynamic loading during bring-up, no `pip`.** The whole arc here ships + statically linked. Dynamic libraries are a real *later* milestone + ([dynamic-libraries.md](dynamic-libraries.md)) — an application-layer + facility that unlocks `ctypes` and loadable extension modules; the operating + system itself stays static and lean regardless (the size doctrine below). + `pip` stays out either way until a networking track exists. +- **No fork.** `os.fork` will not exist. This costs almost nothing (see "The + spawn model fits"). + +## The realization that shapes everything: the compiler is not the obstacle + +`zig cc` is a full Clang-based C cross-compiler, and CPython is portable C with +official precedent for stranger targets than danos — the WASI port is upstream +tier-2, and it runs **without fork, without dynamic loading, and without working +threads**. Every "CPython can't possibly run there" objection has already been +answered upstream by a target *more* constrained than danos. + +What CPython actually needs is a **C environment**: headers and a `libc.a`. danos +has neither — and that is the whole project. In the language of the Zig roadmap's +three doors, this is the **door-2-shaped work** (the deferred "musl door"), not the +`std.os.danos` seam: CPython never touches Zig's std. + +### The same surface, a third time + +The Zig roadmap observed that door 1 (`std.os.danos`) and door 2 (a libc) implement +the *same* ~30 danos-facing operations at different layers. CPython consumes that +identical surface through C spellings. So nothing here is throwaway: the +danos-native operations backing `runtime.os` are the same ones the libc bottoms out +in, and the gaps this track must close (stdio byte streams, cwd, environment, +entropy) are **exactly the Phase-1 gaps the Zig roadmap already lists**. The two +tracks share a road until Python forks off at "build the libc." + +## Where danos stands: coverage vs. the gaps + +Judged against the minimal CPython configuration (static, WASI-like): + +| CPython need | danos today | Gap | +|--------------|-------------|-----| +| open/read/write/close/lseek, readdir | VFS + FAT via `runtime.fs` | none — wrap in C | +| mkdir / unlink / rename / truncate | done (self-hosting Phase 2) | none | +| stat with mtime | done (`wall_clock` + FAT mtime) | none | +| mmap/munmap (object allocator) | native syscalls | none | +| monotonic + wall clock | `clock` + `wall_clock` syscalls | none | +| a place for `Lib/` | FAT boot image | none — better than WASI has it | +| fork / exec | not needed (subprocess disabled at first) | — | +| dynamic loading | not needed (static extension modules) | — | +| getcwd / chdir | — | **missing** (shared with Zig Phase 1) | +| environment variables | `Init` has no env | **missing** (can start empty) | +| entropy | — | **missing** (hash seed; `PYTHONHASHSEED` pins it meanwhile) | +| byte-stream stdin/stdout (fd 0/1/2) | `debug_write` out; structured `InputEvent` in | **missing** (shared with Zig Phase 1; the REPL needs it) | +| signals | — | stubs suffice (WASI precedent); M17 signals-over-IPC maps on later | +| threads | native `thread_spawn`/futex | build threadless first; a pthread subset later (xonsh needs it) | + +The clustering repeats the Zig roadmap's: **files, memory, and time are done; the +work is the C packaging plus the small seam pieces** (tty bytes, cwd, env, entropy). + +## The libc decision: hand-rolled in Zig, computation lifted from musl + +Two viable shapes were considered: + +| Option | What it is | Verdict | +|--------|-----------|---------| +| **Mini-libc in Zig** | C-ABI-exporting Zig library over `runtime.os`/`runtime.fs`, shipped as headers + `libc.a`. | **Take this.** Reuses the danos-native surface directly; no Linux assumptions to fight. | +| **Port musl** | Full musl with a danos syscall backend. | Defer, again. musl assumes Linux syscall semantics in places; heavier than the need. | + +The trick that makes the mini-libc tractable: musl's `string/`, `math/` (libm — +CPython needs essentially all of it), and number-conversion layers are **pure +computation with no syscalls**. Lift those wholesale (MIT-licensed, designed to +compile standalone) and hand-write only: + +- the OS-facing bottom: fds, `mmap`, clocks, `exit`, `getcwd` — thin C-ABI wrappers + over `runtime.os`; +- a `FILE*` stdio layer (buffered, over the fd layer); +- `malloc` over danos `mmap` (a simple allocator is fine; CPython does its own + small-object arena management above it); +- the headers (`stdio.h`, `stdlib.h`, `string.h`, `math.h`, `errno.h`, …). + +Estimate: **100–150 functions**, of which the hard 40% (libm, string, printf/strtod +cores) are lifted, not written. Correctness hot spots are `strtod`/`dtoa` (Python's +float repr round-trips through them) — another reason to lift musl's, not improvise. + +## C interop: static extension modules, not ctypes + +"Python can interface with C libraries" is true on danos with one important +correction: **`ctypes` does not work at first** — it is built on `dlopen` + libffi, +both of which arrive only with the [dynamic-libraries](dynamic-libraries.md) +milestone (D2). Until then the interop story is the other, older one: + +- **Extension modules statically linked into the interpreter** via CPython's + `Modules/Setup` mechanism (the standard route for embedded/static builds). +- **Zig speaks C ABI natively**, so danos extension modules are written in Zig + against `Python.h` — no C required. Two modules are planned from the start: + - **`danos`** — the system module: endpoints, send/receive, capability passing, + spawn, exit notification. This is what makes a Python *service* possible: an + event loop over IPC, speaking the same wire protocols as Zig services. + - **UI toolkit bindings** — the in-progress danos UI toolkit exposed to Python, + so application prototypes drive real windows. + +The package story follows: **pure-Python packages work** (unpack into +`Lib/site-packages` on the FAT image); packages with C extensions must be +cross-compiled and baked into the interpreter — a curated set chosen per image, +not `pip install`. That is the honest shape of the stop-gap. + +## The roadmap + +### Phase 0 — Toolchain + libc bring-up + +`zig cc -target x86_64-freestanding-none` plus `-isystem` the danos headers and the +mini-libc archive. No compiler fork required — this track deliberately avoids the +Zig roadmap's Phase-0 gate (if the fork lands first, the triple becomes a clean +`x86_64-danos`; nothing else changes). Exit criterion: a **hello-world C program** +compiles on the host and runs on danos, printing via the libc's `write`. + +### Phase 1 — The shared seam pieces + +The same list as Zig self-hosting Phase 1, closed once for both tracks: + +- fd 0/1/2 as console **byte** streams (output exists as `debug_write`; input is a + new small thing — cooked line input first, raw mode when the REPL wants editing); +- `getcwd`/`chdir`; +- environment variables (an empty block is a valid start); +- an entropy syscall or service (until then, builds pin `PYTHONHASHSEED`). + +### Phase 2 — Cross-compile CPython, minimal configuration + +Pin one CPython release (3.13 — strongest WASI-era cross-compile support). The +mechanics are well-trodden upstream since 3.11: + +- a same-version **build-Python on the host** (`--with-build-python`); +- a `config.site` cache answering what configure cannot probe cross + (`ac_cv_file__dev_ptmx=no` and friends); +- a `config.sub` patch so `x86_64-unknown-danos` parses; +- `--disable-shared`, static `Modules/Setup` with a minimal module set + (`posix`, `errno`, `_io`, `_codecs`, `time`, `math`, …); +- `Lib/` shipped on the FAT image; `PYTHONHOME` pointed at it. + +Exit criterion: `python -c 'print(2**100)'` runs on danos under QEMU. + +### Phase 3 — Terminal + REPL: the first real application + +Depends on the display track's font rendering (already its stated next step) and +Phase 1's tty. A terminal emulator drawing a `python` REPL is the milestone demo: +interactive, self-evidently real, and it needs **zero** process-control machinery. + +### Phase 4 — The `danos` module and Python services + +Write the `danos` extension module and the UI-toolkit bindings; prototype one real +service in Python (a policy-shaped one — e.g. hot-plug policy or a settings +service) speaking the existing IPC protocols. This is the payoff phase for +"prototyping a service or application." + +### Phase 5 — Process control, then the shell + +The shell — any shell, in any language — forces the surface danos has deferred so +far: **exec-of-path, argv/envp passing, numeric exit status (`WEXITSTATUS`, not the +categorical `ExitReason`), fd inheritance, and pipes.** That is a kernel/VFS +milestone cluster of its own. Then, in order: + +1. `subprocess` enabled in CPython (maps onto danos spawn — see below); +2. a **small Python shell** (a few hundred lines over `subprocess` + line input, no + job control) — the forcing function that reveals which process-control pieces + actually matter; +3. **explicitly deferred:** a pthread subset over `thread_spawn`/futex + (create/join/mutex/condition/thread-locals), signals via M17 signals-over-IPC, + termios job control — and then **xonsh**, which wants all three. + +### The spawn model fits + +One genuinely good alignment: **CPython does not need fork.** `subprocess` maps +cleanly onto a posix_spawn-style model — exactly what danos has — and the existing +exit-notification-via-endpoint is a *better* fit for `Popen.wait` than Unix's +`wait` semantics. `os.fork` simply won't exist, as on Windows, and almost nothing +in practice cares. + +## Risks and gotchas + +- **Binary size — and the size doctrine that makes it acceptable.** danos's + leanness mandate applies to the **operating system**: the kernel and the system + services stay small (the kernel is measured in kilobytes, not megabytes), and + nothing in this track changes that — Python never enters the OS layer. An + **application** budget is different: a statically-linked CPython with its + module set will be tens of megabytes in ReleaseSafe (the measured ~2× + safety-check factor compounds it), and that is *allowed* — applications live + on the FAT image, not in the kernel's world. It still shapes the image, and it + means every Python service shares one interpreter binary + per-service + scripts, so the spawn model needs **argv** before "run this .py" works at all. +- **FAT mtime granularity is 2 seconds.** CPython's `.pyc` cache validation is + mtime-based by default; a rapid edit-run cycle can see stale bytecode. Use + hash-based `.pyc` invalidation (PEP 552, `--invalidation-mode checked-hash` at + freeze time) or accept the quirk during bring-up. +- **FAT name lookups are case-insensitive.** Long file names preserve case but + match insensitively — the same world Python inhabits on Windows/macOS, so + importlib copes, but two modules differing only by case cannot coexist on the + image. +- **`strtod`/float repr correctness.** Python's float round-tripping is exacting; + lift musl's conversions rather than writing them, and run CPython's float tests + early. +- **Threadless build is load-bearing, initially.** Like WASI, the first builds have + no working `threading`. The escape hatch is real (danos has native threads and + futexes; a pthread subset is Phase-5 work) but keep the configuration honestly + single-threaded until then. +- **The test suite is the porting harness.** CPython ships its own conformance + suite; getting `test_builtin`, `test_int`, `test_float`, `test_io` running on + danos early converts "it seems to work" into a checklist. Budget image space for + the test `Lib/` tree during bring-up. +- **Entropy before exposure.** `PYTHONHASHSEED=0` is fine for bring-up and wrong + forever; hash randomization exists because attacker-controlled dict keys are a + denial-of-service vector. Land the entropy source before any Python service + parses external input. + +## Related + +- [python-on-danos-milestones.md](python-on-danos-milestones.md) — the execution + plan (P0–P5) for this note. +- [c-library-compatibility.md](c-library-compatibility.md) — the mini-libc + (libdanos-c) design behind Phase 0. +- [character-devices-and-tty.md](character-devices-and-tty.md) — the stream-node / + console / no-pty design behind Phase 1. +- [zig-self-hosting.md](zig-self-hosting.md) — the sibling track; shares Phase 1, + diverges at the libc. +- [os-development/syscall.md](os-development/syscall.md) — the kernel ABI the + mini-libc bottoms out in. +- [os-development/vdso.md](os-development/vdso.md) — the public ABI boundary the + `danos` extension module wraps. +- [os-development/sysv.md](os-development/sysv.md) — the entry stack (argv/envp) + the spawn-argv work extends. +- [device-driver-development/ipc.md](device-driver-development/ipc.md) — the IPC + surface Python services speak. +- [file-system-development/danos-file-system-hierarchy-FSH.md](file-system-development/danos-file-system-hierarchy-FSH.md) + — where `Lib/` and `site-packages` land on the image.