danos/docs/python-on-danos.md

Python on danos: the CPython milestone

A design note (not built yet) on bringing CPython to danos, compiled with the Zig toolchain (zig cc). Like zig-self-hosting.md, it is forward-looking: it sets a direction and the decisions that follow from it.

Why Python, and why now

The Zig self-hosting road is gated on a compiler fork and a long std-library seam. Python is the stop-gap that removes the wait: a working CPython gives danos a way to write programs — services, tools, application prototypes — without the Zig compiler being self-hosted, and it brings the pure-Python package ecosystem along as a bonus. The intended division of labour:

• Zig — the kernel, drivers, and anything on a data plane (interrupt paths, DMA rings, block I/O). Unchanged.
• Python — the control plane and the prototyping surface: services that are event loops over IPC, policy logic that changes often, application experiments, and eventually the shell.

Python is also the scripting language for the terminal-and-shell arc: the first real danos application is planned as a terminal, a terminal wants a shell, a shell wants a scripting language — and xonsh (a shell written in Python) marks where that road can end.

Non-goals

• No drivers in Python. Interrupt handling, ring management, and DMA stay in Zig. Python may supervise and configure drivers; it does not sit in their hot paths (interpreter overhead and garbage-collection pauses in an interrupt path are disqualifying).
• No dynamic loading during bring-up, no pip. The whole arc here ships statically linked. Dynamic libraries are a real later milestone (dynamic-libraries.md) — an application-layer facility that unlocks ctypes and loadable extension modules; the operating system itself stays static and lean regardless (the size doctrine below). pip stays out either way until a networking track exists.
• No fork. os.fork will not exist. This costs almost nothing (see "The spawn model fits").

The realization that shapes everything: the compiler is not the obstacle

zig cc is a full Clang-based C cross-compiler, and CPython is portable C with official precedent for stranger targets than danos — the WASI port is upstream tier-2, and it runs without fork, without dynamic loading, and without working threads. Every "CPython can't possibly run there" objection has already been answered upstream by a target more constrained than danos.

What CPython actually needs is a C environment: headers and a libc.a. danos has neither — and that is the whole project. In the language of the Zig roadmap's three doors, this is the door-2-shaped work (the deferred "musl door"), not the std.os.danos seam: CPython never touches Zig's std.

The same surface, a third time

The Zig roadmap observed that door 1 (std.os.danos) and door 2 (a libc) implement the same ~30 danos-facing operations at different layers. CPython consumes that identical surface through C spellings. So nothing here is throwaway: the danos-native operations backing runtime.os are the same ones the libc bottoms out in, and the gaps this track must close (stdio byte streams, cwd, environment, entropy) are exactly the Phase-1 gaps the Zig roadmap already lists. The two tracks share a road until Python forks off at "build the libc."

Where danos stands: coverage vs. the gaps

Judged against the minimal CPython configuration (static, WASI-like):

CPython need	danos today	Gap
open/read/write/close/lseek, readdir	VFS + FAT via runtime.fs	none — wrap in C
mkdir / unlink / rename / truncate	done (self-hosting Phase 2)	none
stat with mtime	done (wall_clock + FAT mtime)	none
mmap/munmap (object allocator)	native syscalls	none
monotonic + wall clock	clock + wall_clock syscalls	none
a place for Lib/	FAT boot image	none — better than WASI has it
fork / exec	not needed (subprocess disabled at first)	—
dynamic loading	not needed (static extension modules)	—
getcwd / chdir	—	missing (shared with Zig Phase 1)
environment variables	Init has no env	missing (can start empty)
entropy	—	missing (hash seed; PYTHONHASHSEED pins it meanwhile)
byte-stream stdin/stdout (fd 0/1/2)	debug_write out; structured InputEvent in	missing (shared with Zig Phase 1; the REPL needs it)
signals	—	stubs suffice (WASI precedent); M17 signals-over-IPC maps on later
threads	native thread_spawn/futex	build threadless first; a pthread subset later (xonsh needs it)

The clustering repeats the Zig roadmap's: files, memory, and time are done; the work is the C packaging plus the small seam pieces (tty bytes, cwd, env, entropy).

The libc decision: hand-rolled in Zig, computation lifted from musl

Two viable shapes were considered:

Option	What it is	Verdict
Mini-libc in Zig	C-ABI-exporting Zig library over runtime.os/runtime.fs, shipped as headers + libc.a.	Take this. Reuses the danos-native surface directly; no Linux assumptions to fight.
Port musl	Full musl with a danos syscall backend.	Defer, again. musl assumes Linux syscall semantics in places; heavier than the need.

The trick that makes the mini-libc tractable: musl's string/, math/ (libm — CPython needs essentially all of it), and number-conversion layers are pure computation with no syscalls. Lift those wholesale (MIT-licensed, designed to compile standalone) and hand-write only:

• the OS-facing bottom: fds, mmap, clocks, exit, getcwd — thin C-ABI wrappers over runtime.os;
• a FILE* stdio layer (buffered, over the fd layer);
• malloc over danos mmap (a simple allocator is fine; CPython does its own small-object arena management above it);
• the headers (stdio.h, stdlib.h, string.h, math.h, errno.h, …).

Estimate: 100–150 functions, of which the hard 40% (libm, string, printf/strtod cores) are lifted, not written. Correctness hot spots are strtod/dtoa (Python's float repr round-trips through them) — another reason to lift musl's, not improvise.

C interop: static extension modules, not ctypes

"Python can interface with C libraries" is true on danos with one important correction: ctypes does not work at first — it is built on dlopen + libffi, both of which arrive only with the dynamic-libraries milestone (D2). Until then the interop story is the other, older one:

• Extension modules statically linked into the interpreter via CPython's Modules/Setup mechanism (the standard route for embedded/static builds).
• Zig speaks C ABI natively, so danos extension modules are written in Zig against Python.h — no C required. Two modules are planned from the start:
  • danos — the system module: endpoints, send/receive, capability passing, spawn, exit notification. This is what makes a Python service possible: an event loop over IPC, speaking the same wire protocols as Zig services.
  • UI toolkit bindings — the in-progress danos UI toolkit exposed to Python, so application prototypes drive real windows.

The package story follows: pure-Python packages work (unpack into Lib/site-packages on the FAT image); packages with C extensions must be cross-compiled and baked into the interpreter — a curated set chosen per image, not pip install. That is the honest shape of the stop-gap.

The roadmap

Phase 0 — Toolchain + libc bring-up

zig cc -target x86_64-freestanding-none plus -isystem the danos headers and the mini-libc archive. No compiler fork required — this track deliberately avoids the Zig roadmap's Phase-0 gate (if the fork lands first, the triple becomes a clean x86_64-danos; nothing else changes). Exit criterion: a hello-world C program compiles on the host and runs on danos, printing via the libc's write.

Phase 1 — The shared seam pieces

The same list as Zig self-hosting Phase 1, closed once for both tracks:

• fd 0/1/2 as console byte streams (output exists as debug_write; input is a new small thing — cooked line input first, raw mode when the REPL wants editing);
• getcwd/chdir;
• environment variables (an empty block is a valid start);
• an entropy syscall or service (until then, builds pin PYTHONHASHSEED).

Phase 2 — Cross-compile CPython, minimal configuration

Pin one CPython release (3.13 — strongest WASI-era cross-compile support). The mechanics are well-trodden upstream since 3.11:

• a same-version build-Python on the host (--with-build-python);
• a config.site cache answering what configure cannot probe cross (ac_cv_file__dev_ptmx=no and friends);
• a config.sub patch so x86_64-unknown-danos parses;
• --disable-shared, static Modules/Setup with a minimal module set (posix, errno, _io, _codecs, time, math, …);
• Lib/ shipped on the FAT image; PYTHONHOME pointed at it.

Exit criterion: python -c 'print(2**100)' runs on danos under QEMU.

Phase 3 — Terminal + REPL: the first real application

Depends on the display track's font rendering (already its stated next step) and Phase 1's tty. A terminal emulator drawing a python REPL is the milestone demo: interactive, self-evidently real, and it needs zero process-control machinery.

Phase 4 — The danos module and Python services

Write the danos extension module and the UI-toolkit bindings; prototype one real service in Python (a policy-shaped one — e.g. hot-plug policy or a settings service) speaking the existing IPC protocols. This is the payoff phase for "prototyping a service or application."

Phase 5 — Process control, then the shell

The shell — any shell, in any language — forces the surface danos has deferred so far: exec-of-path, argv/envp passing, numeric exit status (WEXITSTATUS, not the categorical ExitReason), fd inheritance, and pipes. That is a kernel/VFS milestone cluster of its own. Then, in order:

1. subprocess enabled in CPython (maps onto danos spawn — see below);
2. a small Python shell (a few hundred lines over subprocess + line input, no job control) — the forcing function that reveals which process-control pieces actually matter;
3. explicitly deferred: a pthread subset over thread_spawn/futex (create/join/mutex/condition/thread-locals), signals via M17 signals-over-IPC, termios job control — and then xonsh, which wants all three.

The spawn model fits

One genuinely good alignment: CPython does not need fork. subprocess maps cleanly onto a posix_spawn-style model — exactly what danos has — and the existing exit-notification-via-endpoint is a better fit for Popen.wait than Unix's wait semantics. os.fork simply won't exist, as on Windows, and almost nothing in practice cares.

Risks and gotchas

• Binary size — and the size doctrine that makes it acceptable. danos's leanness mandate applies to the operating system: the kernel and the system services stay small (the kernel is measured in kilobytes, not megabytes), and nothing in this track changes that — Python never enters the OS layer. An application budget is different: a statically-linked CPython with its module set will be tens of megabytes in ReleaseSafe (the measured ~2× safety-check factor compounds it), and that is allowed — applications live on the FAT image, not in the kernel's world. It still shapes the image, and it means every Python service shares one interpreter binary + per-service scripts, so the spawn model needs argv before "run this .py" works at all.
• FAT mtime granularity is 2 seconds. CPython's .pyc cache validation is mtime-based by default; a rapid edit-run cycle can see stale bytecode. Use hash-based .pyc invalidation (PEP 552, --invalidation-mode checked-hash at freeze time) or accept the quirk during bring-up.
• FAT name lookups are case-insensitive. Long file names preserve case but match insensitively — the same world Python inhabits on Windows/macOS, so importlib copes, but two modules differing only by case cannot coexist on the image.
• strtod/float repr correctness. Python's float round-tripping is exacting; lift musl's conversions rather than writing them, and run CPython's float tests early.
• Threadless build is load-bearing, initially. Like WASI, the first builds have no working threading. The escape hatch is real (danos has native threads and futexes; a pthread subset is Phase-5 work) but keep the configuration honestly single-threaded until then.
• The test suite is the porting harness. CPython ships its own conformance suite; getting test_builtin, test_int, test_float, test_io running on danos early converts "it seems to work" into a checklist. Budget image space for the test Lib/ tree during bring-up.
• Entropy before exposure. PYTHONHASHSEED=0 is fine for bring-up and wrong forever; hash randomization exists because attacker-controlled dict keys are a denial-of-service vector. Land the entropy source before any Python service parses external input.

Related

• python-on-danos-milestones.md — the execution plan (P0–P5) for this note.
• c-library-compatibility.md — the mini-libc (libdanos-c) design behind Phase 0.
• character-devices-and-tty.md — the stream-node / console / no-pty design behind Phase 1.
• zig-self-hosting.md — the sibling track; shares Phase 1, diverges at the libc.
• os-development/syscall.md — the kernel ABI the mini-libc bottoms out in.
• os-development/vdso.md — the public ABI boundary the danos extension module wraps.
• os-development/sysv.md — the entry stack (argv/envp) the spawn-argv work extends.
• device-driver-development/ipc.md — the IPC surface Python services speak.
• file-system-development/danos-file-system-hierarchy-FSH.md — where Lib/ and site-packages land on the image.