Phase 2a: FAT engine mutations + O_TRUNC (fix the overwrite corruption)

The FAT engine could create, read, write, and grow files, but never free clusters
or make directories — so overwriting a shorter file left a stale tail (a real bug:
corrupt boot-log re-flushes, and later corrupt compiler cache/.o files). This adds
the mutation half of the engine, with the corruption fix wired all the way through.

Engine (system/services/fat/engine.zig), all host-tested:
- freeChain: return a cluster chain to the pool (bounded against a corrupt cycle) —
  the shared primitive under truncate and remove.
- truncate: free the chain and zero the entry's size/first-cluster (O_TRUNC).
- createDirectory (mkdir): allocate + initialise a cluster with "." and ".." and add
  the directory entry to the parent.
- removeFile (unlink): free the chain and mark the 8.3 entry plus any preceding
  long-name entries deleted, so a reused slot can't inherit an orphaned long name.
  Refuses directories.
- createFile now shares a common addEntry helper with createDirectory.

O_TRUNC wired end to end: a truncate open-flag (vfs protocol) that the router already
forwards; runtime.fs.OpenOptions.truncate; and fat's handleOpen calls engine.truncate
on an existing file. The boot-log flush (log-flush + init) now opens with truncate, so
a shorter log on a later boot of the same stick leaves no stale tail — closing the
caveat from the boot-log work.

mkdir/unlink are engine-complete and host-tested but not yet exposed as VFS
operations / runtime.fs methods (they are new path-based ops needing router cases);
rename and the richer stat (mtime/mode, blocked on wall-clock) remain. See
docs/zig-self-hosting.md (Phase 2).

Verified: zig build, zig build test (8 engine host tests, incl. truncate, the
overwrite-no-stale-tail regression, remove, and mkdir), zig build check-fat-image, and
a sequential QEMU sweep — fat-mount, log-flush (DANOS.LOG read back at 11804 bytes,
clean, with the truncate-based final flush), vfs, vfs-client-death, usb-storage,
orderly-shutdown, initial-ramdisk, smoke — all green.

docs/zig-self-hosting.md

@@ -251,6 +251,14 @@ Because `std.fs`/`std.Io` have no per-OS branches, finishing this in `runtime.os
 lights up the whole file tower for the compiler at once. Environment can stay an empty
 map until the kernel populates a non-empty `envp`.
 
+**Status (Phase 2a landed):** `truncate` is done end to end — `engine.truncate` frees
+the cluster chain, an O_TRUNC open flag wires it through the VFS and `runtime.fs`, and
+the boot-log flush uses it (closing the stale-tail bug). The engine also has
+`createDirectory` (mkdir) and `removeFile` (unlink), both host-tested (LFN-aware
+deletion included). Still to do: expose mkdir/unlink as VFS operations + `runtime.fs`
+methods (they are new path-based ops needing router cases); `rename`; and the richer
+`stat` (blocked on wall-clock).
+
 ### Phase 3 — Single-threaded, self-linked compiler bring-up
 
 Build the compiler with **two load-bearing flags**:
@@ -318,9 +326,10 @@ Two current decisions fall out of this roadmap:
    as the concrete backing for that entry — not as a bespoke `std.Io.Writer`-only shim.
    Decide it once, at the seam, and program stdout, diagnostics, and file writes all
    flow through the same danos VFS/console path.
-2. **The boot-log `truncate` caveat graduates from cosmetic to a real bug.** It is the
+2. **The boot-log `truncate` caveat is now fixed** (Phase 2a). It was the same
-   same `writeFile`-only-grows gap; on the self-hosting road it corrupts build output.
+   `writeFile`-only-grows gap that on the self-hosting road would corrupt build output;
-   Fix it in Phase 2, not "someday."
+   `engine.truncate` + an O_TRUNC open flag now free the old chain so a shorter rewrite
+   leaves no stale tail, and the boot-log flush opens with it.
 
 ## Related
 

library/runtime/fs.zig

@@ -42,11 +42,15 @@ pub const OpenOptions = struct {
     create: bool = false,
     /// Open a directory node (for listing) rather than a file.
     directory: bool = false,
+    /// Truncate an existing file to zero length on open (O_TRUNC) — replace its
+    /// contents rather than overwriting in place.
+    truncate: bool = false,
 
     fn wireFlags(self: OpenOptions) u32 {
         var f: u32 = 0;
         if (self.create) f |= protocol.create;
         if (self.directory) f |= protocol.directory;
+        if (self.truncate) f |= protocol.truncate;
         return f;
     }
 };

system/services/fat/engine.zig

@@ -1,6 +1,8 @@
 //! The FAT filesystem engine: mount a block device, walk the FAT and directory
-//! structures, and read / write / create files. FAT12/16/32 (the type is
+//! structures, and read / write / create / truncate files, remove files, and make
-//! detected from the cluster count). Pure logic over a `BlockDevice` interface —
+//! directories. FAT12/16/32 (the type is detected from the cluster count). The
+//! crash-safe write order is data -> FAT -> directory; truncate and remove free the
+//! cluster chain, then update the directory. Pure logic over a `BlockDevice` interface —
 //! no IPC — so it is host-testable against a RAM-backed image (see the tests at
 //! the bottom). The fat.zig server wraps a real `.block` device in a BlockDevice
 //! and serves this over the VFS protocol.
@@ -238,6 +240,21 @@ pub const FileSystem = struct {
         return null;
     }
 
+    // Free every cluster of the chain starting at `first`, returning them to the
+    // pool. A first < 2 (an empty file) frees nothing. Bounded against a corrupt
+    // cyclic chain by the cluster count so it can never loop forever.
+    fn freeChain(self: *FileSystem, first: u32) void {
+        var cluster = first;
+        var guard: u32 = 0;
+        const limit = self.geometry.cluster_count + 2;
+        while (cluster >= 2 and cluster < limit and guard < limit) : (guard += 1) {
+            const next = self.readFatEntry(cluster);
+            _ = self.writeFatEntry(cluster, on_disk.free_cluster);
+            if (self.isEndOfChain(next) or next < 2) break;
+            cluster = next;
+        }
+    }
+
     // --- directory iteration ------------------------------------------------
 
     // The absolute LBA of the `sector_index`th sector of directory `dir`, or null
@@ -546,6 +563,18 @@ pub const FileSystem = struct {
         return consumed;
     }
 
+    /// Truncate a file node to zero length: free its cluster chain and clear its
+    /// size and first cluster in the directory entry. This is O_TRUNC — the fix for
+    /// re-opening and overwriting an existing file, whose old (longer) contents
+    /// would otherwise linger past the new end (a silent-corruption bug for anything
+    /// that rewrites a file in place, like the boot-log flush).
+    pub fn truncate(self: *FileSystem, node: *Node) void {
+        self.freeChain(node.first_cluster);
+        node.first_cluster = 0;
+        node.size = 0;
+        self.updateEntry(node.*);
+    }
+
     // Write a node's size and first cluster back into its 8.3 directory entry.
     fn updateEntry(self: *FileSystem, node: Node) void {
         if (!node.has_entry) return;
@@ -557,11 +586,10 @@ pub const FileSystem = struct {
         _ = self.blockWrite(node.entry_sector, &self.dir_sector);
     }
 
-    /// Create an 8.3-named file in directory `dir`. Returns the new (empty) node,
+    // Add an 8.3 directory entry to `dir` with the given attributes, first cluster,
-    /// or null if the name is not 8.3-representable or no directory slot is free.
+    // and size, reusing a free (0x00 or 0xE5) slot and growing the directory chain
-    pub fn createFile(self: *FileSystem, dir: Node, name: []const u8) ?Node {
+    // if needed. Returns the new node (with its entry location) or null if full.
-        const raw = to83(name) orelse return null;
+    fn addEntry(self: *FileSystem, dir: Node, raw: [11]u8, attributes: u8, first_cluster: u32, size: u32) ?Node {
-        // Find a free directory slot (a 0x00 or 0xE5 entry), growing the directory.
         var sector_index: u32 = 0;
         while (self.dirSectorLba(dir, sector_index, true)) |lba| : (sector_index += 1) {
             if (!self.blockRead(lba, &self.dir_sector)) return null;
@@ -572,13 +600,15 @@ pub const FileSystem = struct {
                 if (existing.isFree()) {
                     var entry = std.mem.zeroes(on_disk.DirectoryEntry);
                     entry.name = raw;
-                    entry.attributes = on_disk.attribute_archive;
+                    entry.attributes = attributes;
+                    entry.file_size = size;
+                    entry.setFirstCluster(first_cluster);
                     @memcpy(self.dir_sector[offset .. offset + @sizeOf(on_disk.DirectoryEntry)], std.mem.asBytes(&entry));
                     if (!self.blockWrite(lba, &self.dir_sector)) return null;
                     return .{
-                        .first_cluster = 0,
+                        .first_cluster = first_cluster,
-                        .size = 0,
+                        .size = size,
-                        .is_directory = false,
+                        .is_directory = attributes & on_disk.attribute_directory != 0,
                         .entry_sector = lba,
                         .entry_offset = offset,
                         .has_entry = true,
@@ -590,6 +620,125 @@ pub const FileSystem = struct {
         }
         return null;
     }
+
+    /// Create an 8.3-named file in directory `dir`. Returns the new (empty) node,
+    /// or null if the name is not 8.3-representable or no directory slot is free.
+    pub fn createFile(self: *FileSystem, dir: Node, name: []const u8) ?Node {
+        const raw = to83(name) orelse return null;
+        return self.addEntry(dir, raw, on_disk.attribute_archive, 0, 0);
+    }
+
+    /// Create an 8.3-named subdirectory in `dir`: allocate and initialise its first
+    /// cluster with "." (itself) and ".." (the parent) entries, then add its
+    /// directory entry to `dir`. Returns the new directory node, or null (bad name,
+    /// no free cluster, or the directory is full). Long names are not created.
+    pub fn createDirectory(self: *FileSystem, dir: Node, name: []const u8) ?Node {
+        const raw = to83(name) orelse return null;
+        const cluster = self.allocateCluster() orelse return null;
+        self.zeroCluster(cluster);
+
+        // ".." points at the parent: 0 for the fixed root on FAT12/16, the root
+        // cluster on FAT32, else the parent's own first cluster.
+        const parent_cluster: u32 = if (dir.first_cluster != 0)
+            dir.first_cluster
+        else if (self.geometry.fat_type == .fat32)
+            self.geometry.root_cluster
+        else
+            0;
+        var dot = std.mem.zeroes(on_disk.DirectoryEntry);
+        dot.name = [_]u8{'.'} ++ ([_]u8{' '} ** 10);
+        dot.attributes = on_disk.attribute_directory;
+        dot.setFirstCluster(cluster);
+        var dotdot = std.mem.zeroes(on_disk.DirectoryEntry);
+        dotdot.name = [_]u8{ '.', '.' } ++ ([_]u8{' '} ** 9);
+        dotdot.attributes = on_disk.attribute_directory;
+        dotdot.setFirstCluster(parent_cluster);
+        var first_sector = [_]u8{0} ** sector_size;
+        const entry_size = @sizeOf(on_disk.DirectoryEntry);
+        @memcpy(first_sector[0..entry_size], std.mem.asBytes(&dot));
+        @memcpy(first_sector[entry_size .. 2 * entry_size], std.mem.asBytes(&dotdot));
+        if (!self.blockWrite(self.clusterSector(cluster, 0), &first_sector)) {
+            self.freeChain(cluster);
+            return null;
+        }
+        return self.addEntry(dir, raw, on_disk.attribute_directory, cluster, 0) orelse {
+            self.freeChain(cluster);
+            return null;
+        };
+    }
+
+    const EntryLoc = struct { sector: u64, offset: u32 };
+
+    // Mark a directory entry deleted in place (its name[0] set to 0xE5).
+    fn markDeleted(self: *FileSystem, sector: u64, offset: u32) void {
+        if (!self.blockRead(sector, &self.dir_sector)) return;
+        self.dir_sector[offset] = 0xE5;
+        _ = self.blockWrite(sector, &self.dir_sector);
+    }
+
+    /// Remove a file named `name` from directory `dir`: free its cluster chain and
+    /// mark its 8.3 entry — and any long-name entries immediately preceding it —
+    /// deleted, so the slots (and the long name) are reusable without a later entry
+    /// that reuses them inheriting the orphaned long name. Refuses a directory (a
+    /// separate rmdir would have to check emptiness). Returns true if removed.
+    pub fn removeFile(self: *FileSystem, dir: Node, name: []const u8) bool {
+        var run: [21]EntryLoc = undefined; // the long-name entries before the 8.3 one
+        var run_len: usize = 0;
+        var long_name: [260]u8 = undefined;
+        var long_len: usize = 0;
+        var sector_index: u32 = 0;
+        while (self.dirSectorLba(dir, sector_index, false)) |lba| : (sector_index += 1) {
+            if (!self.blockRead(lba, &self.dir_sector)) return false;
+            var i: u32 = 0;
+            while (i < entries_per_sector) : (i += 1) {
+                const offset = i * @sizeOf(on_disk.DirectoryEntry);
+                const entry = std.mem.bytesToValue(on_disk.DirectoryEntry, self.dir_sector[offset .. offset + @sizeOf(on_disk.DirectoryEntry)]);
+                if (entry.isEnd()) return false;
+                if (entry.name[0] == 0xE5) {
+                    run_len = 0;
+                    long_len = 0;
+                    continue;
+                }
+                if (entry.isLongName()) {
+                    if (run_len < run.len) {
+                        run[run_len] = .{ .sector = lba, .offset = offset };
+                        run_len += 1;
+                    }
+                    const lfn = std.mem.bytesToValue(on_disk.LongNameEntry, self.dir_sector[offset .. offset + @sizeOf(on_disk.LongNameEntry)]);
+                    const order = lfn.order & 0x1F;
+                    if (order >= 1 and order <= 20) {
+                        var chunk: [13]u8 = undefined;
+                        const n = longNameChars(lfn, &chunk);
+                        const start = (order - 1) * 13;
+                        if (start + n <= long_name.len) {
+                            @memcpy(long_name[start .. start + n], chunk[0..n]);
+                            if (lfn.order & 0x40 != 0) long_len = start + n;
+                        }
+                    }
+                    continue;
+                }
+                if (entry.isVolumeLabel()) {
+                    run_len = 0;
+                    long_len = 0;
+                    continue;
+                }
+                // A real 8.3 entry.
+                var short: [12]u8 = undefined;
+                const display = if (long_len > 0) long_name[0..long_len] else format83(entry.name, &short);
+                if (nameMatches(display, name)) {
+                    if (entry.isDirectory()) return false; // not for directories
+                    self.freeChain(entry.firstCluster());
+                    self.markDeleted(lba, offset);
+                    var r: usize = 0;
+                    while (r < run_len) : (r += 1) self.markDeleted(run[r].sector, run[r].offset);
+                    return true;
+                }
+                run_len = 0;
+                long_len = 0;
+            }
+        }
+        return false;
+    }
 };
 
 // --- tests: a RAM-backed FAT16 image ----------------------------------------
@@ -705,3 +854,112 @@ test "create, write, read back a file through the engine" {
     try std.testing.expectEqualStrings("HELLO.TXT", listing.name_buffer[0..listing.name_len]);
     try std.testing.expect(fs.listEntry(fs.rootNode(), 1) == null);
 }
+
+test "truncate frees the chain and zeroes the file" {
+    const allocator = std.testing.allocator;
+    const bytes = try allocator.alloc(u8, 5000 * sector_size);
+    defer allocator.free(bytes);
+    formatFat16(bytes);
+    var disk = RamDisk{ .bytes = bytes };
+    var fs = FileSystem.mount(disk.device()).?;
+
+    var node = fs.createFile(fs.rootNode(), "BIG.BIN").?;
+    var payload: [2000]u8 = undefined;
+    for (&payload, 0..) |*b, i| b.* = @truncate(i);
+    _ = fs.writeFile(&node, 0, &payload);
+    const cluster = node.first_cluster;
+    try std.testing.expect(cluster >= 2);
+
+    fs.truncate(&node);
+    try std.testing.expectEqual(@as(u32, 0), node.size);
+    try std.testing.expectEqual(@as(u32, 0), node.first_cluster);
+    // The old first cluster is free again.
+    try std.testing.expectEqual(on_disk.free_cluster, fs.readFatEntry(cluster));
+
+    // Re-resolve: the persisted entry is empty, and reads produce nothing.
+    const resolved = fs.resolve("/BIG.BIN").?;
+    try std.testing.expectEqual(@as(u32, 0), resolved.size);
+    var buf: [16]u8 = undefined;
+    try std.testing.expectEqual(@as(usize, 0), fs.readFile(resolved, 0, &buf));
+}
+
+test "overwrite after truncate leaves no stale tail (the O_TRUNC corruption fix)" {
+    const allocator = std.testing.allocator;
+    const bytes = try allocator.alloc(u8, 5000 * sector_size);
+    defer allocator.free(bytes);
+    formatFat16(bytes);
+    var disk = RamDisk{ .bytes = bytes };
+    var fs = FileSystem.mount(disk.device()).?;
+
+    // Write a long file, then truncate-and-rewrite a short one — the O_TRUNC flow.
+    var node = fs.createFile(fs.rootNode(), "LOG.TXT").?;
+    _ = fs.writeFile(&node, 0, "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"); // 32 bytes
+    fs.truncate(&node);
+    _ = fs.writeFile(&node, 0, "bb");
+
+    // Size is the short length — no lingering old bytes past the new end.
+    const resolved = fs.resolve("/LOG.TXT").?;
+    try std.testing.expectEqual(@as(u32, 2), resolved.size);
+    var buf: [8]u8 = undefined;
+    const n = fs.readFile(resolved, 0, &buf);
+    try std.testing.expectEqualStrings("bb", buf[0..n]);
+}
+
+test "remove a file frees its slot and its cluster chain" {
+    const allocator = std.testing.allocator;
+    const bytes = try allocator.alloc(u8, 5000 * sector_size);
+    defer allocator.free(bytes);
+    formatFat16(bytes);
+    var disk = RamDisk{ .bytes = bytes };
+    var fs = FileSystem.mount(disk.device()).?;
+
+    var node = fs.createFile(fs.rootNode(), "GONE.TXT").?;
+    var payload: [1000]u8 = undefined;
+    for (&payload, 0..) |*b, i| b.* = @truncate(i);
+    _ = fs.writeFile(&node, 0, &payload);
+    const cluster = fs.resolve("/GONE.TXT").?.first_cluster;
+    try std.testing.expect(cluster >= 2);
+
+    try std.testing.expect(fs.removeFile(fs.rootNode(), "GONE.TXT"));
+    // Gone from the directory, its cluster free, root empty again.
+    try std.testing.expect(fs.resolve("/GONE.TXT") == null);
+    try std.testing.expectEqual(on_disk.free_cluster, fs.readFatEntry(cluster));
+    try std.testing.expect(fs.listEntry(fs.rootNode(), 0) == null);
+    // Removing a missing file reports false.
+    try std.testing.expect(!fs.removeFile(fs.rootNode(), "GONE.TXT"));
+    // A directory is refused (it is not a file).
+    _ = fs.createDirectory(fs.rootNode(), "ADIR").?;
+    try std.testing.expect(!fs.removeFile(fs.rootNode(), "ADIR"));
+}
+
+test "create a subdirectory with . and .. and a file inside" {
+    const allocator = std.testing.allocator;
+    const bytes = try allocator.alloc(u8, 5000 * sector_size);
+    defer allocator.free(bytes);
+    formatFat16(bytes);
+    var disk = RamDisk{ .bytes = bytes };
+    var fs = FileSystem.mount(disk.device()).?;
+
+    const made = fs.createDirectory(fs.rootNode(), "SUB").?;
+    try std.testing.expect(made.is_directory);
+    try std.testing.expect(made.first_cluster >= 2);
+
+    // It resolves as a directory, with "." and ".." as its first two entries.
+    const dir = fs.resolve("/SUB").?;
+    try std.testing.expect(dir.is_directory);
+    const dot = fs.listEntry(dir, 0).?;
+    try std.testing.expectEqualStrings(".", dot.name_buffer[0..dot.name_len]);
+    const dotdot = fs.listEntry(dir, 1).?;
+    try std.testing.expectEqualStrings("..", dotdot.name_buffer[0..dotdot.name_len]);
+
+    // A file created inside is reachable by its full path.
+    var child = fs.createFile(dir, "INNER.TXT").?;
+    _ = fs.writeFile(&child, 0, "hi");
+    const inner = fs.resolve("/SUB/INNER.TXT").?;
+    try std.testing.expectEqual(@as(u32, 2), inner.size);
+
+    // The root lists SUB as a directory.
+    const listing = fs.listEntry(fs.rootNode(), 0).?;
+    try std.testing.expectEqualStrings("SUB", listing.name_buffer[0..listing.name_len]);
+    try std.testing.expect(listing.is_directory);
+}

system/services/fat/fat.zig

@@ -124,7 +124,12 @@ fn handleOpen(out: []u8, path: []const u8, flags: u32) usize {
         const parent = filesystem.resolve(parent_path) orelse return fail(out);
         node = filesystem.createFile(parent, leaf);
     }
-    const resolved = node orelse return fail(out);
+    var resolved = node orelse return fail(out);
+    // O_TRUNC: replace an existing file's contents rather than overwriting in place
+    // (frees the old chain, so a shorter rewrite leaves no stale tail).
+    if (flags & protocol.truncate != 0 and !resolved.is_directory) {
+        filesystem.truncate(&resolved);
+    }
     const index = allocOpen() orelse return fail(out);
     open_nodes[index] = .{ .used = true, .node = resolved };
     return writeReply(out, .{ .status = 0, .node = index }, &.{});

system/services/init/init.zig

@@ -133,7 +133,8 @@ fn subscribePower() void {
 /// USB volume is not mounted, the open fails and it does nothing. Must run while
 /// the storage services are still alive (see shutDown).
 fn flushKernelLog() void {
-    var file = runtime.fs.open(log_path, .{ .create = true }) orelse return; // no USB volume mounted
+    // Truncate on open so this fuller flush replaces the boot-time one cleanly.
+    var file = runtime.fs.open(log_path, .{ .create = true, .truncate = true }) orelse return; // no USB volume
     defer file.close();
     var chunk: [4096]u8 = undefined;
     var offset: usize = 0;

system/services/log-flush/log-flush.zig

@@ -51,7 +51,9 @@ pub fn main() void {
     }
     if (!ready) return; // /mnt/usb never became available — nothing to persist to
 
-    var file = fs.open(log_path, .{ .create = true }) orelse return; // could not create the file
+    // Truncate on open: each flush replaces the file, so a shorter log on a later
+    // boot of the same stick leaves no stale tail from a previous, longer one.
+    var file = fs.open(log_path, .{ .create = true, .truncate = true }) orelse return;
     const written = drainKernelLog(&file);
     file.close();
 

system/services/vfs/protocol.zig

@@ -82,11 +82,15 @@ pub const reply_size: usize = @sizeOf(Reply);
 /// Largest inline payload that still fits one IPC message alongside a header.
 pub const maximum_payload: usize = message_maximum - request_size;
 
-/// Open flags (danos-native; the POSIX layer maps `O_CREAT` onto `create`).
+/// Open flags (danos-native; `runtime.fs.OpenOptions` maps its booleans onto these).
 pub const create: u32 = 1;
 /// Open a directory (for readdir) rather than a file. A mounted backend uses
 /// this to open a directory node; the flat ramfs ignores it.
 pub const directory: u32 = 2;
+/// Truncate the file to zero length on open (O_TRUNC): replace its contents rather
+/// than overwriting in place, so a shorter new file leaves no stale tail. A mounted
+/// backend frees the old cluster chain; the flat ramfs ignores it.
+pub const truncate: u32 = 4;
 
 test "protocol struct sizes and node kinds" {
     const std = @import("std");