Hostmortem
A flight recorder for Linux edge devices: it keeps the evidence a reboot destroys.
- Role
- Sole developer: scope and research, architecture, the diagnosis rules, the fault lab, and review of the code Claude Code agents wrote under my direction
- Dates
- Nov 2026 – present
- Status
- Replay live · v0.1
- Stack
- C++20, Linux, SQLite · 8 of 8 faults diagnosed
Hostmortem, screen by screen
In short
It says “unknown” when that is the truth
Twelve fixed playbooks score evidence for and against each explanation. If nothing is supported, or two tie, the answer is unknown, with what the agent could not see.
Tested on real faults, not mock ones
A lab breaks a real network inside Linux namespaces eight ways, from a DNS outage to killing the agent itself. All eight were diagnosed as expected.
Loss is recorded, never hidden
The queue between collectors and disk is bounded and never blocks. Every dropped event is counted and written down as an event of its own.
The problem
The moment an edge box misbehaves, the usual fix is a reboot, and the reboot erases what would have explained it.
- Logs in memory, counters and connection state are gone after a restart.
- Small boxes often have no battery-backed clock, so they wake up in 1970 and timestamps cannot be trusted for order.
- A monitoring tool that guesses confidently is worse than none: a wrong cause sends someone to fix the wrong thing.
- The recorder itself must be small and must not fall over when the box it watches is under load.
How it works
It watches five places
Kernel counters, link and route changes, the kernel log, a few active probes and the processes you name.
It writes evidence down
One writer commits batches to SQLite. If something is dropped under load, the drop is written down too.
It opens an incident
A detector notices failed probes, lost packets, slow replies or a dead process, and marks where the incident starts and ends.
It argues with itself
Playbooks score each explanation, cite the events behind it, rule others out and say what could not be seen.
Engineering challenges
-
A queue that never blocks the thing it watches
Every event is either written once or counted as dropped, under ThreadSanitizer. With every commit synced to disk, 10 runs at 20,000 events per second dropped none.
-
An upstream outage that looked like a local fault
Both real DNS scenarios are diagnosed correctly: upstream blocked points upstream, a broken local resolver points local.
-
A lab that passed for the wrong reason
A passing run now means the box was healthy first and the fault was the one injected.
Technical decisions
| Decision | What I chose | Instead of |
|---|---|---|
| Rules a person can read, not a model | Twelve fixed playbooks with whole-number evidence points | A language model or a learned classifier |
| Order by boot, not by the wall clock | Boot count first, then the kernel's time since boot | |
| Plain kernel interfaces first | procfs, netlink, the kernel log and active probes in 0.1; eBPF in 0.2 |
Results
- real injected faults diagnosed as expected, two of them an honest “unknown”
- 8 of 8
- of evidence lost at most when a virtual machine was hard reset mid-incident, over 2 runs (limit 5 s)
- 1.4 s
- events dropped at 20,000 per second in 10 runs, with every commit synced to disk
- 0
- tests, clean under three sanitizers
- 82
Measured on 13 November 2026 in Ubuntu 24.04 arm64 virtual machines on an Apple M1 Pro. The hard reset is a VM reset, not a physical power cut. No Raspberry Pi or x86 hardware numbers yet.
Reflection
What worked
- Cutting the first brief down to one box and one question made a testable first version possible.
- Injecting real faults found bugs that hand-written test files could not, including one in the lab itself.
What I underestimated
- How easily a test harness can be broken in a way that still passes.
- How much sanitizers disagree with their surroundings: one check made a run 600 times slower, and another would not start on a newer kernel until the tests ran with address randomisation off.
What I would do next
- Repeat the reset as a physical power cut, and measure on a Raspberry Pi.
- Publish the prepared 0.1.0 release.
- eBPF probes for per-process and per-connection evidence, and numbers from a Raspberry Pi.
Full engineering notes
Summary
A Linux gateway loses DNS, its link or a service overnight, someone reboots it, and the evidence goes with it. Hostmortem is a small agent that records what happened, in order, across the kernel, the network and the services on one box, and keeps it on disk through the reboot. Five collectors feed a bounded queue that never blocks and counts everything it drops; one writer commits batches to SQLite. When something breaks, a detector opens an incident and twelve fixed playbooks score the evidence for and against each explanation. Every claim cites the stored events it rests on, and when nothing is supported, or two explanations tie, the answer is “unknown”, with a list of what the agent could not see. No AI takes part in the diagnosis. I designed and built it on 13 November 2026, with Claude Code agents writing much of the code under my direction and review. It is version 0.1: the engine, the command-line tool and a lab that injects real faults all work; there is no packaged release yet.
My role
What I owned
- The scope: one host, no fleet server, no packet payloads, cut down from a much larger first brief.
- The architecture and its eight decision records: the queue, the store, time ordering and the playbooks.
- The diagnosis rules: what counts as evidence for and against each explanation, and when to answer unknown.
- The real-fault lab and its pre-flight checks.
- Direction and review of the code: Claude Code agents wrote much of it; I set each task, read the result and sent it back when it was wrong.
What I did not own
- Claude Code agents did much of the typing; the decisions and review are mine.
- SQLite, GoogleTest, nlohmann/json and the sanitizers and fuzzer from LLVM.
- The showcase fonts are customised from the open-licence Martian Mono and Atkinson Hyperlegible.
- No outside users, code review or hardware trial yet.
Constraints
One host, on purpose
The first brief described a platform with a fleet server, a web interface and a microcontroller peer. I cut it to one box and one question, so the first version could be finished and tested.
No hardware, no Linux desk
I develop on a Mac, so everything builds and runs in a Linux container, and the fault lab needs a privileged throwaway one. Numbers come from an arm64 virtual machine and are labelled that way; a Raspberry Pi has not been measured.
How each challenge was solved
A queue that never blocks the thing it watches
- Why it was hard
If the disk is slow, collectors must not stall, or the recorder becomes part of the outage. But silently dropping events would make the record lie.
- What I built
Collectors push into a bounded queue with a high and a low priority lane and never wait. When it is full, the event is counted as dropped, and the writer commits that count as an event of its own, so a gap in the record is visible as a gap.
- Options considered
- An unbounded queue
- Block collectors until the writer catches up
- A bounded queue that drops silently
- A bounded queue with two priority lanes that records its own drops
- Trade-off
Under extreme load low-priority samples are lost first, and the record says so instead of pretending to be complete.
- Validation
A concurrency test with several producers checks every event is either written once or counted as dropped; it runs under ThreadSanitizer. A disk-full test checks a batch fails whole and recovery commits a drop record.
An upstream outage that looked like a local fault
- Why it was hard
When the upstream DNS server was blocked in the lab, the tool answered unknown: the local resolver and the upstream server looked equally guilty.
- What I built
Healthy direct queries from just before the fault were being counted as “direct succeeded while the resolver failed”. Each failure is now paired with the nearest-in-time probe from the other family, so evidence from before the fault no longer argues against it.
- Options considered
- Count any probe within 15 seconds as happening at the same time
- Pair each failure with the nearest probe of the other kind
- Trade-off
The pairing is stricter, so a short fault with very few probes yields less evidence and more often an honest unknown.
- Validation
The upstream and local DNS scenarios in the real-fault lab, after the lab itself was fixed.
A lab that passed for the wrong reason
- Why it was hard
The first real DNS run passed. The DNS server in the lab had never started, so DNS was down from the beginning and the right answer came out by accident.
- What I built
The lab now runs a pre-flight: DNS, ping and a TCP connection must all work before any fault is injected, and a failed pre-flight is a lab failure. The cause was a relative file path that broke when the DNS server moved to the root directory on start.
- Options considered
- Trust the pass
- Check the lab is healthy before breaking it
- Trade-off
Each scenario takes a little longer to run.
- Validation
Re-running the scenario after the fix exposed the real diagnosis bug above, which the false pass had hidden.
Why I made each decision
Rules a person can read, not a model
- Because
Someone acting on a diagnosis at 3 a.m. needs to see why. Each playbook lists the events for and against, the same input always gives the same answer, and a release check rejects any explanation that cites no event.
- Trade-off
Each new kind of fault needs a rule written and tested by hand.
Order by boot, not by the wall clock
- Because
Edge boxes without a clock battery start in 1970 and jump when time syncs. Ordering by boot and by time since boot stays correct across both; wall time is shown only for people to read.
Plain kernel interfaces first
- Because
They work on old kernels with two capabilities and no compiler on the box. What they cannot see, such as which connection retransmitted, is listed as a blind spot in every diagnosis instead of being guessed.
- Trade-off
Per-connection and per-process detail waits for 0.2.
Testing
Unit and integration tests cover the queue, the store, each parser, the detector and the reports, with thirteen recorded scenario files replayed through the whole pipeline. The same tests run under AddressSanitizer with UndefinedBehaviorSanitizer and under ThreadSanitizer, and each parser that reads outside input has a fuzz target; fuzzing found one real crash in the kernel-log parser, now fixed with a regression test. The lab then injects real faults into Linux network namespaces.
- tests, clean under the address, undefined-behaviour and thread sanitizers
- 82
- real injected faults diagnosed as expected
- 8 of 8
- fuzz targets, one per parser of outside input
- 6
Results in detail
Incidents anyone can replay
The showcase page replays the lab's eight recorded incidents on a five-lane tape, event by event, with the verdict under each and the unedited report beside it.
It survives a hard reset, after the test found it did not
The first hard reset of a virtual machine lost a whole boot of evidence. The cause was a database setting that let commits wait in memory; syncing every commit fixed it. Now at most 1.4 seconds are lost over two runs, the store passes its integrity check, and one incident spans both boots.
Stack and release record
Platforms
Linux agent (hmd), run as a dedicated user under systemd, Command-line tool (hm): timeline, incidents, explain, doctor, Incident reports as text, JSON or one self-contained HTML file
Built with
C++20, CMake, SQLite (WAL mode), Linux procfs, rtnetlink, /dev/kmsg, DNS, ICMP and TCP probes, GoogleTest, libFuzzer, AddressSanitizer, UndefinedBehaviorSanitizer, ThreadSanitizer, Docker, GitHub Actions
Testing and delivery
82 tests passed on 14 November 2026 and run clean under three sanitizers, on GitHub runners for both amd64 and arm64; the real-fault lab diagnosed 8 of 8 injected faults as expected. A Debian package builds and installs in a clean Ubuntu 24.04, and the 0.1.0 release is prepared but not published.
Status in full
Version 0.1. The engine, the command-line tool, twelve playbooks, the real-fault lab and a Debian package work, tested in Ubuntu 24.04 arm64 virtual machines. A hard reset of a virtual machine in the middle of an incident is tested; a physical power cut is not. The 0.1.0 release is prepared but not published. Not yet done: eBPF probes (planned for 0.2) and measurements on a Raspberry Pi or x86. The source is private; a public showcase replays the lab's recorded incidents.
Hiring interns for January 2027?
I’m available full-time from 25 January to 23 April 2027, and for graduate roles in Malaysia or overseas from August 2027. Email is the quickest way to reach me.
- Subang Jaya, Malaysia
- local time
yousofselim2@gmail.com