danos/docs/dynamic-libraries.md

# Dynamic libraries on danos

A design note and milestone plan for shared objects: building them, loading them
with `dlopen`, and — the part that needs kernel work — actually *sharing* them
between processes. Directional, post-P5 of
[python-on-danos-milestones.md](python-on-danos-milestones.md); nothing on the
CPython bring-up path depends on it.

## Reconciling the earlier "rejected"

Dynamic libraries were evaluated once before and rejected — but as an answer to a
*different question*: whether they could claw back ReleaseSafe's measured ~2×
code size. They cannot (the safety checks inline at every call site; no library
scheme dedups them), and that verdict stands for that question. The reasons to
build them now are the ones that investigation never weighed:

- **`ctypes` and runtime FFI** — Python calling into a danos library without
  rebuilding the interpreter. This is the piece that makes Python prototyping
  self-serve: drop a `.so` on the image, `ctypes.CDLL` it, iterate.
- **Loadable CPython extension modules** — today every C extension means
  relinking the interpreter (`Modules/Setup`); with `dlopen`, an extension is a
  file.
- **One interpreter image, many Python services** — a statically-linked CPython
  is tens of megabytes *per process*. A shared `libpython` mapped read-only once
  (milestone D3 below) makes Python services cheap enough to be the default way
  to prototype one.
- **Plugin-shaped applications** — the UI toolkit and the terminal will want
  them eventually.

The scoping that dissolves the apparent contradiction is the **size doctrine**:
leanness is an *operating-system* property — the kernel and system services stay
small and statically linked, and none of them ever link the loader — while
*applications* have their own budget and may be big. Dynamic libraries are an
**application-layer facility**, full stop.

What also does **not** change: the public ABI stays the vDSO + the IPC
protocols. Shared objects are artifacts *within* one system image, versioned by
the build — not a new stable ABI surface for the OS.

## Design

- **Format and codegen are free.** ELF shared objects with position-independent
  code; `zig cc -fPIC -shared` against the [libdanos-c](c-library-compatibility.md)
  sysroot already emits them. The work is entirely on the loading side.
- **The loader lives in userspace, inside the libc.** `dlopen` reads the `.so`
  through the VFS, maps its segments, applies relocations, resolves symbols
  against the process and the `DT_NEEDED` dependency graph, runs constructors,
  returns a handle. No kernel loader changes in v1 — segments land in anonymous
  `mmap` as private copies.
- **Bind-now, always.** All relocations resolved at `dlopen` time
  (`RTLD_NOW` semantics only). Lazy PLT binding buys startup latency danos does
  not care about, at the price of a writable GOT dance and a much subtler
  loader. Not worth it; keep it out permanently.
- **W^X from day one.** Map, relocate, then flip text pages read-execute —
  which requires memory-protection change (`mprotect`-shaped) in the danos
  `mmap` surface if it is not already there. No page is ever writable and
  executable at once.
- **TLS in shared objects is deferred.** Thread-local storage models
  (initial-exec vs. general-dynamic) are the deep end of every dynamic linker.
  v1 refuses a `.so` with a TLS segment; revisit alongside the post-P5 pthread
  subset, which is when it could matter.
- **Executables stay static until D4.** v1 is "a static binary that can
  `dlopen`" — no `PT_INTERP`, no program interpreter, no dynamically-linked
  `main` binaries. That keeps process startup untouched.

## Milestones

1. **D1 — dlopen in-process.** The `.so` build target; the loader in libdanos-c:
   map, relocate (`RELATIVE`/`GLOB_DAT`/`JUMP_SLOT`), resolve, constructors;
   `dlopen`/`dlsym`/`dlerror`/`dlclose`; private anonymous mappings; no TLS.
   *Test:* QEMU `dlopen-hello` — load a `.so`, call a symbol, unload, reload.
2. **D2 — the FFI payoff.** `DT_NEEDED` dependency graphs; a **libffi port**
   (x86-64 SysV assembly is upstream; the port is its closure-allocation paths,
   which must respect W^X); CPython's `ctypes` enabled; extension modules
   loadable from file. *Test:* QEMU — a Python script `ctypes.CDLL`s a danos
   `.so` and round-trips a call; a `.so` extension module imports.
3. **D3 — actual sharing (the kernel milestone).** Shared read-only file-backed
   mappings — a page-cache-shaped facility so N processes mapping `libpython`
   hold one physical copy. This is the memory-win milestone and the only one
   touching the kernel; design it with the existing shm machinery in view
   (the shared-fate walks already locked the relevant paths). *Test:* N Python
   services up; measure physical pages against N× the static baseline.
4. **D4 — dynamically-linked executables** (optional, evaluate after D3):
   `PT_INTERP`, a danos program interpreter, and the spawn path teaching the
   loader about it. Only worth it if the image-size or update story demands it.

## Risks and gotchas

- **Scope creep is the failure mode.** Every dynamic linker grows toward glibc.
  The fences: bind-now only, no lazy binding ever, no TLS until pthreads demand
  it, no dlopen-from-memory, no versioned symbols. Each fence removed is a
  design discussion, not a patch.
- **Code loading is a security event.** `dlopen` turns file bytes into executable
  code, so W^X discipline is table stakes and *what may be dlopened* is a
  capability question — the natural danos answer is that loadability follows VFS
  readability of the `.so`, and services' images are supervised like any other
  artifact. Revisit explicitly at D3 when mappings become shared.
- **`dlclose` is where loaders go to die.** Constructors/destructors,
  dangling function pointers, re-open identity. Keep v1 semantics honest and
  simple: `dlclose` runs destructors and unmaps; holding pointers past it is
  undefined; no reference-counted deferral cleverness.
- **The ReleaseSafe fact still applies to `.so`s** — a ReleaseSafe shared object
  carries its inlined checks like any static code; D3's sharing saves *copies*,
  not check overhead. Size expectations should be set accordingly.

## Decisions needing sign-off

- Dynamic libraries join the roadmap at all (this note exists because the
  earlier size-motivated rejection was re-opened for ABI/sharing reasons).
- **Bind-now only; no lazy binding, permanently.**
- **Loader in userspace libc; kernel involvement only at D3** (shared read-only
  mappings).
- **Static executables until D4**, and D4 only on demonstrated need.

## Related

- [c-library-compatibility.md](c-library-compatibility.md) — the sysroot the
  loader ships in; its absence table gains `dlfcn.h` at D1.
- [python-on-danos.md](python-on-danos.md) — the `ctypes` story this unlocks.
- [python-on-danos-milestones.md](python-on-danos-milestones.md) — sequencing;
  this work is post-P5.
- [os-development/memory-map.md](os-development/memory-map.md) /
  [os-development/paging.md](os-development/paging.md) — where W^X and shared
  mappings land.