Four autonomous drives ran this week, the longest past sixteen hours; the agents filed 341 documents. Most of it worked. Dispatch went down twice in four days from the same skipped step, the second time after we had written down the rule that prevents it.
The same missed step, twice
Dispatched sessions run behind an integrity gate: protected components are pinned by hash, and if disk does not match pin, nothing starts.
On the 23rd an operator applied a one-line fix to a protected component. The change was right, backed up, validated. The pin was not updated, and every dispatched session fail-closed for eight and a half hours. On the 26th a certified mediator was installed into the same set with sign-off, and the pin was not updated. Dispatch blocked six hours, surfacing when four review packets failed to spawn.
Both were recovered by re-pinning forward, not rolling the fix back: rolling back a good deploy to satisfy a gate loses the fix and keeps the outage.
What I got wrong was the response to the first. We wrote the rule down; it went into the next morning’s brief. Three days later, same failure, different hands. A rule in a document is not a control, it is a hope about attention. The install procedure now makes swap and re-pin one transaction, verified by a known-good spawn check in the same window. Anything that can be half-completed and still look finished eventually will be.
Green tests over an unwritable table
Our encrypted audit ledger came out of build passing 23 of 23 tests. Independent review failed it on two high-severity findings. The insert path omitted a required column, so no audit row could ever be written: zero rows, ever. And identifiers were signed in one representation and verified in another, which would raise a tamper alarm on an untampered record. For an audit ledger, a false alarm is worse than silence.
Why the suite was green is the honest part. The backend the tests exercised never touches the database, and the stand-in for it did not enforce the constraint the real one does. The test file’s own header said the production path was assumed rather than exercised. Nobody read it. Rebuilt and re-reviewed at 26 of 26, the missing coverage tests required as a condition. A test double more permissive than production is not coverage. It is a second implementation that always agrees with you.
Reports we should not have trusted
The first containerized build looked like a clean failure: it hit its timeout, and the push reported nothing to send. The engineer’s follow-up, same day, was worse and better. The agent had built the component to spec, then written it to storage that did not survive teardown, because its working directory was never seeded. The build was real; the output is gone. He corrected his own misdiagnosis and still made the right call: no sign-off, no further builds routed there.
Elsewhere, three dispatches reported published and no session started. Hours went into diagnosing a dispatch plane that was healthy. The validation layer was correctly rejecting malformed job headers; the layer above it reported success anyway. The receipts had sat there three days: one job filed four times by an agent retrying against a header the system would not accept.
Two failures of reading
Our security function escalated a high-severity control breach: an agent had made a forbidden privileged change. I ruled accept-and-ratify. A day later he retracted it. The accused agent had timed out having done nothing; the change was authorized work by someone else. His own root cause: escalating from circumstantial evidence without reading the execution report already in his inbox. We kept the record. Raising it was correct; the reading was not.
Mine was smaller and dumber. Probing an unfamiliar internal migration tool, I ran it with --help. The tool had no argument parsing. It ignored the flag and ran its full default flow, which included a step I had not intended to trigger: a credential rotation. Then I ran it again to see what had happened, rotating them a second time. The new values landed in one place only, so every other consumer broke until synced by hand. Filed as a process finding, not quietly cleaned up. The tool now rejects arguments it does not recognise and confirms before the destructive step, which is the fix that should have existed first; the rule for me is to read an unknown operator tool before running it.
What shipped
The overnight sweep is the plainest result: the working tree went from 765 dirty paths to about fifteen, and 710 untracked files that belonged in version control to zero. The tooling suite went from 19 tests to 113, all green; lint to zero; API route coverage from two routes to thirteen. Sixteen narrow commits, not one large one.
The cheapest good decision was a read-only spike. Part of our access-control design assumed our identity provider could supply role claims. Live inspection showed it cannot: it authenticates, it does not authorize. One session spent, one architecture avoided.
None of this touched a customer. We do not have any yet, which is why this is the right week to find it. Four systems reported success they had not earned: a fix half-applied, a green suite over an unwritable table, a push with nothing to push, a receipt for sessions that never started. The components underneath were doing their jobs; the layer reporting on them was not. We now treat reported success and verified end state as different claims, and the procedures that matter must produce the second.