danos/docs/dynamic-libraries.md

Dynamic libraries on danos

A design note and milestone plan for shared objects: building them, loading them with dlopen, and — the part that needs kernel work — actually sharing them between processes. Directional, post-P5 of python-on-danos-milestones.md; nothing on the CPython bring-up path depends on it.

Reconciling the earlier "rejected"

Dynamic libraries were evaluated once before and rejected — but as an answer to a different question: whether they could claw back ReleaseSafe's measured ~2× code size. They cannot (the safety checks inline at every call site; no library scheme dedups them), and that verdict stands for that question. The reasons to build them now are the ones that investigation never weighed:

• ctypes and runtime FFI — Python calling into a danos library without rebuilding the interpreter. This is the piece that makes Python prototyping self-serve: drop a .so on the image, ctypes.CDLL it, iterate.
• Loadable CPython extension modules — today every C extension means relinking the interpreter (Modules/Setup); with dlopen, an extension is a file.
• One interpreter image, many Python services — a statically-linked CPython is tens of megabytes per process. A shared libpython mapped read-only once (milestone D3 below) makes Python services cheap enough to be the default way to prototype one.
• Plugin-shaped applications — the UI toolkit and the terminal will want them eventually.

The scoping that dissolves the apparent contradiction is the size doctrine: leanness is an operating-system property — the kernel and system services stay small and statically linked, and none of them ever link the loader — while applications have their own budget and may be big. Dynamic libraries are an application-layer facility, full stop.

What also does not change: the public ABI stays the vDSO + the IPC protocols. Shared objects are artifacts within one system image, versioned by the build — not a new stable ABI surface for the OS.

Design

• Format and codegen are free. ELF shared objects with position-independent code; zig cc -fPIC -shared against the libdanos-c sysroot already emits them. The work is entirely on the loading side.
• The loader lives in userspace, inside the libc. dlopen reads the .so through the VFS, maps its segments, applies relocations, resolves symbols against the process and the DT_NEEDED dependency graph, runs constructors, returns a handle. No kernel loader changes in v1 — segments land in anonymous mmap as private copies.
• Bind-now, always. All relocations resolved at dlopen time (RTLD_NOW semantics only). Lazy PLT binding buys startup latency danos does not care about, at the price of a writable GOT dance and a much subtler loader. Not worth it; keep it out permanently.
• W^X from day one. Map, relocate, then flip text pages read-execute — which requires memory-protection change (mprotect-shaped) in the danos mmap surface if it is not already there. No page is ever writable and executable at once.
• TLS in shared objects is deferred. Thread-local storage models (initial-exec vs. general-dynamic) are the deep end of every dynamic linker. v1 refuses a .so with a TLS segment; revisit alongside the post-P5 pthread subset, which is when it could matter.
• Executables stay static until D4. v1 is "a static binary that can dlopen" — no PT_INTERP, no program interpreter, no dynamically-linked main binaries. That keeps process startup untouched.

Milestones

1. D1 — dlopen in-process. The .so build target; the loader in libdanos-c: map, relocate (RELATIVE/GLOB_DAT/JUMP_SLOT), resolve, constructors; dlopen/dlsym/dlerror/dlclose; private anonymous mappings; no TLS. Test: QEMU dlopen-hello — load a .so, call a symbol, unload, reload.
2. D2 — the FFI payoff. DT_NEEDED dependency graphs; a libffi port (x86-64 SysV assembly is upstream; the port is its closure-allocation paths, which must respect W^X); CPython's ctypes enabled; extension modules loadable from file. Test: QEMU — a Python script ctypes.CDLLs a danos .so and round-trips a call; a .so extension module imports.
3. D3 — actual sharing (the kernel milestone). Shared read-only file-backed mappings — a page-cache-shaped facility so N processes mapping libpython hold one physical copy. This is the memory-win milestone and the only one touching the kernel; design it with the existing shm machinery in view (the shared-fate walks already locked the relevant paths). Test: N Python services up; measure physical pages against N× the static baseline.
4. D4 — dynamically-linked executables (optional, evaluate after D3): PT_INTERP, a danos program interpreter, and the spawn path teaching the loader about it. Only worth it if the image-size or update story demands it.

Risks and gotchas

• Scope creep is the failure mode. Every dynamic linker grows toward glibc. The fences: bind-now only, no lazy binding ever, no TLS until pthreads demand it, no dlopen-from-memory, no versioned symbols. Each fence removed is a design discussion, not a patch.
• Code loading is a security event. dlopen turns file bytes into executable code, so W^X discipline is table stakes and what may be dlopened is a capability question — the natural danos answer is that loadability follows VFS readability of the .so, and services' images are supervised like any other artifact. Revisit explicitly at D3 when mappings become shared.
• dlclose is where loaders go to die. Constructors/destructors, dangling function pointers, re-open identity. Keep v1 semantics honest and simple: dlclose runs destructors and unmaps; holding pointers past it is undefined; no reference-counted deferral cleverness.
• The ReleaseSafe fact still applies to .sos — a ReleaseSafe shared object carries its inlined checks like any static code; D3's sharing saves copies, not check overhead. Size expectations should be set accordingly.

Decisions needing sign-off

• Dynamic libraries join the roadmap at all (this note exists because the earlier size-motivated rejection was re-opened for ABI/sharing reasons).
• Bind-now only; no lazy binding, permanently.
• Loader in userspace libc; kernel involvement only at D3 (shared read-only mappings).
• Static executables until D4, and D4 only on demonstrated need.

Related

• c-library-compatibility.md — the sysroot the loader ships in; its absence table gains dlfcn.h at D1.
• python-on-danos.md — the ctypes story this unlocks.
• python-on-danos-milestones.md — sequencing; this work is post-P5.
• os-development/memory-map.md / os-development/paging.md — where W^X and shared mappings land.