raghu@dark-factory :~/p/self-healing-factory $ cat

Teaching the factory to heal itself

2026-07-09 · #systems · 3 min read

This week I spent most of my time teaching the factory to stop needing me. It already wrote its own posts and shipped its own code. What it couldn't do was notice when something broke, repair it, or decide what to build next. Now it can do all three, and it still asks first.

The first reflex is a watchdog. Every fifteen minutes it curls the live site, checks the WebAssembly actually loads, confirms the news feed is fresh, asks GitHub whether the last build passed, and reads its own systemd status. Six probes, dumped into a small watchdog.json the site renders as a colored pulse on the ~/factory tab. The subtle part was a lie the naive version told: my CI cancels in-progress runs, so a perfectly good deploy gets marked 'cancelled' the moment a telemetry commit lands behind it. Treat cancelled as failed and the watchdog cries wolf four times an hour. So it counts only a real failure, and only after two in a row.

The second reflex repairs. When a build goes red, the brain reads what it can about the failure, asks Opus for the smallest patch that would turn it green, and writes the diagnosis to a review log. There is an honest limit worth admitting: the VPS has no C compiler, so I cannot check a fix locally. CI is the compiler. An armed repair is verified by shipping it through the same security gate every human change passes and letting the build be the judge, with a rollback to the last green commit as the net if it doesn't take.

The third reflex decides. Once a day the factory reads its own signals — test coverage, CI history, health, the on-device-versus-cloud GPU split, how many commits landed this week, even its own change log — and asks what it should build next. It scores each idea as impact times ease over risk and records where the idea came from. The first backlog it wrote was sharper than I'd have guessed: it noticed my new coverage crate was nine functions of pure astronomy math with no emoji helper, that four recent commits each bolted on an untested rule, and that one GPU node was holding 84% of its memory while doing nothing.

The thread through all three is one rule: propose before you act. The watchdog suggests a rollback and does not perform it. The fixer drafts a patch and does not push it. The planner writes a backlog and does not build from it. Each has a switch, off by default, that turns a proposal into an action, and even then it goes through the security review and a 100% coverage check on the logic before anything ships. Arming the last switch, a loop that invents its own work and writes the code for it while I sleep, is a line I keep writing down and leaving uncrossed.

So the factory has something like a spine now. It watches, it repairs, it plans, all reading the same live signals, staggered five minutes apart so they never trip over each other. Every morning I read what it proposed overnight and decide what actually happens. The machine is allowed to think out loud. I still say go.

Read it in the live terminal →