When a Linux desktop breaks, you know one thing: it worked last week. Package history covers packages only, the journal shows events but not state, and snapshot tools show the whole system at a moment but not what changed between two snapshots, who changed it, or which change broke things. There is no git bisect for an operating system.

1. 01

   Two witnesses

   A live witness records who changes each file as it happens, with a filesystem-wide fanotify mark and eBPF tracing every write back to its process and package transaction. A state witness proves what changed between read-only btrfs snapshots from the send stream. When they disagree, for example a change made while the daemon was stopped, the gap is reported instead of smoothed over.

2. 02

   Causes, not files

   Changes are grouped into causes: a dnf transaction, a systemd service, a terminal command, an app writing a setting. Writes a daemon makes on someone's behalf over D-Bus, from tools like timedatectl or nmcli, are credited to the caller. osreflog why shows who changed a file and a readable diff of what it means.

3. 03

   Git's vocabulary on purpose

   log, show, diff, blame, bisect and undo work on causes, and references read like git's reflog: c42, @{1}, @{boot-1}, @{yesterday}. Diffs are computed after each checkpoint and cached, so the diff after a real dnf transaction comes back in 6 ms.

4. 04

   Bisect down to one cause

   Snapshots are coarse, so osreflog rebuilds the states between them by replaying causes into a writable clone. Each step is tested in a throwaway systemd-nspawn container in under a second, or in a QEMU VM that boots the clone's own kernel over virtiofs in about 14 seconds. Every replay says whether it was exact or approximate.

5. 05

   Undo without rolling back

   osreflog undo reverts one cause path by path as a 3-way merge, leaving alone anything that changed since and refusing on conflict. Package causes are handed to dnf. Dry run is the default, a checkpoint is taken before every apply, and an undo can itself be undone. GNOME settings get the same treatment through dconf.

6. 06

   Observe before touching

   Nothing wrote to the live root until a write gate passed: the full undo matrix in a contained scratch root, with symlink-swap races and magic links refused by openat2 resolution. Config parsers run in a sandboxed worker and were each fuzzed for an hour. Secrets are never stored and command lines are redacted.

7. 07

   Cheap enough to leave on

   The idle daemon used 0.0% CPU and 26 MB of memory on the development laptop, and a checkpoint adds under a second to a dnf transaction. The ledger is local, root-only SQLite with a hash chain, so tampering is evident.

8. 08

   Metal bisect next

   Some regressions only show on real hardware. The next phase reboots once into a disposable clone, with the default boot entry checked untouched after every probe, and it stays behind a boot gate tested in a nested VM first. A GTK and libadwaita app and COPR packaging, like drainscope's, come last.

5 of 5

seeded regressions bisected

each found within log2(n) + 1 probes

0.9 s

slowest container probe

nspawn; vm probes boot the snapshot's kernel in 14 s or less

87 of 87

config changes attributed

a scripted workload of about 22 real admin tools

0

parser crashes under fuzzing

8 targets, an hour each, about 3.05 billion inputs