Devlog
Engineering studies from building deterministic, locally-run AI systems. Architecture reasoning, findings, and the occasional honest negative result — written for people who build the same kinds of things.
The Fabricated Citation That Read Perfectly — And the Check That Costs Nothing
Our first research operation produced a citation to a real page containing a sentence that was not on it. Cross-family review caught it; reading never would have. The check that catches this class of defect requires no model at all.
Read →Believed, Not Verified
Five times in one project, a guarantee we were relying on was not in effect. A structured-output constraint silently ignored. A measurement that skipped a pipeline stage. A fix in the commit message but not the file. An exclusion in the specification but not the code. Each looked like its own bug. All five were one mechanism, and the fix is three greps.
Read →The Wall We Had Already Published
A finding was published on this site, imported by a second project that saved hours with it, and rediscovered the hard way by a third five weeks later. The handbook stated it in the imperative three days before we shipped the bug. The knowledge was not missing — the structure that would have made us consult it was.
Read →How to Fail Legibly
A literature-mining engine was asked, under a frozen bar, whether it could surface drug–disease connections that were validated only after its corpus ended. It couldn't — twice, under two configurations. The useful part is what made that answer worth keeping: a control that could have produced a positive, outcomes written before the data, three causes located by three instruments, and a closure rule that refuses to fire on an unlocated null.
Read →We Cancelled Our Own Audit Because the Number Would Have Been Flattering
We designed a public study measuring how often published citations fail. Adversarial review killed it before data collection: our instrument's 23% false-positive rate would have manufactured a 'one in three citations fail' headline out of a true rate of 10%.
Read →The gate that caught its own author
In one day: my ruling caught a stale premise in Execution's directive, my own exit gate caught my own env-ordering bug, and a Gitea dedup returned the wrong PR while logging success. Three different gates, three different authors, one lesson — differential proof or it didn't happen.
Read →The conscience that got caught unverified
The phase that exists to deepen verification had its own verifier panic on an em-dash — caught not by its tests but by a wall-clock measurement we pre-committed to run before trusting it. Then the floor we'd seeded turned out to be miscalibrated by 5x against a real module. The scores are quarantined. This is what verify-before-measuring looks like when it eats its own cooking.
Read →The Ceiling Was a Floor
We measured a local 9B's pass rate on Rust and read the failures as the model's capacity ceiling. Most of that ceiling was our own infrastructure quietly poisoning the signal — and an honest measurement of a poisoned signal is still a lie about the model.
Read →The Same Mistake, Four Times
Four unrelated bugs in a small local model — a confident hallucination, an anchored retry, an unheard hint, an ignored rule — turned out to be one mechanism. The model's attention is finite, and whatever else is in its context window competes with the signal it needs. The counterintuitive part is the fix: you cannot solve a competing-context problem by adding more context.
Read →The system that learned to know its own limits
A local 9B model hits capacity walls it can't see. The engineering wasn't building the wall — it was teaching the system to notice when the model was stuck on the wrong side of it, and route around it without a human in the loop. Three signals, two timing bugs, and the moment the system became self-escalating.
Read →The seam that four walls make
A local agent needs tools. Some are safe, others touch the world. The question is where to draw the line. The wrong answer is a prompt that says 'don't touch the world.' The right answer is a seam in the code that makes touching the world structurally impossible — and four independent boundaries that happen to land in the same place.
Read →The clock that preempts itself
A shared GPU is a contention problem. When the operator is present and a background batch is running, something has to wait. The wrong answer is 'whatever arrived first.' The right answer is 'whatever the operator is doing, always.' The quartermaster makes presence preempt absence by construction — typed permits, reserved slots, preemption in the type system rather than in a policy.
Read →The loop that doesn't trust the model
An agent is a loop, and a loop will run in circles forever if the model gets stuck. How the commune gives a 9B model agency without trusting it with the keys — five structural guardrails, each aimed at a specific way the model gets stuck.
Read →The memory that conditions the next attempt
A build system that forgets its failures is doomed to repeat them. Anvil records every build attempt as a structured claim and synthesizes a hint that conditions the next attempt — and the measured surprise is that the richer hint didn't lift the pass rate, for reasons that turned out to be two bugs, not capacity.
Read →The pair that measures the gap
A frontier model solving what the 9B can't is a fact, not a training signal — the signal is the labeled pair of the two outputs, captured for DPO. The pipeline is built and the pairs are accumulating; the open question is whether the gap actually closes.
Read →The first breath: what surfaces when the loop runs live
The vertical slice ran live in a test server. The 9B model produced a grounded 4-action scaffolding plan, the role gate denied a non-operator, and the ledger captured everything. A category-nesting bug surfaced on the first run — the execution layer was passing stale server structure to subsequent actions. The bug was structural, not a parameter issue, and the fix was structural. Build-floor findings from the first honest breath of the loop.
Read →The confident hallucination — and the structure that buried it
A local 9B model hallucinated a dispositional belief about the operator at HIGH confidence — higher than it gave cautious truths. Self-reported confidence is not a hygiene layer. We replaced it with structural provenance density, and the same hallucination scored 0.000.
Read →The circuit breaker that could never recover
A circuit breaker that opens on failure but never transitions back to closed — because the recovery call was never wired in. A safety mechanism that becomes a permanent outage. Plus the circular health metrics that hid it.
Read →Two red teams and a spike: designing agent memory that survives the agent
An agent memory architecture proposal that survived two adversarial red team passes and a humanization spike before a single line of implementation. The methodology caught four critical design flaws — including one that would have made semantic search silently useless.
Read →The two clocks, applied to collaboration
The two-clock architecture — a deterministic fast clock that owns the gateway, a slow probabilistic clock that reasons between turns — proven in game loops and embodiment loops, applied here to a Discord collaborator surface. Why the chat-window-wearing-a-bot failure is the default, why inverting it is the discipline, and what the boundary looks like when the agent takes real actions in a real workspace.
Read →Captured from day one, learned from later
A ledger that captures what happened doesn't capture whether it was good. The learning signal — four capture surfaces at different friction levels, all converging on a single ledger entry linked to the action — is specified and captured from the first action, even though the learner that consumes it is a future component. The methodology: design the signal capture before you need the learner, or you'll find the data insufficient when the learner is ready.
Read →Moving a live store, and the two bugs the last gate caught
I moved a live agent system's memory from the old hub to a new kernel — one writer, zero rows lost — and the checks I ran at the irreversible step caught two real bugs before the point of no return. The bugs are the evidence the gates work.
Read →We made the monitor its own first customer. Here's what it found in us.
A monitoring system you haven't deeply consumed is one you don't know works. We pointed ours at its own infrastructure first — and it caught a failure that had been happening once a minute, for weeks, invisible at the log we were watching.
Read →The safety net that paid for itself on the first run
65 HTTP route tests caught two production bugs on their very first run — a deadlock and a runtime panic that 233 unit tests couldn't see. Then the same session imported a vLLM structured output finding from another project, added an HTTP proxy executor for a consumer project, and closed three security attack surfaces. Each piece of discipline paid for itself immediately.
Read →The first off-shape gene
We built a gate to decide what counts as a gene, then built the first thing it wasn't shaped for and ran it through — hoping it would break. It did, and the break told us what the gate had been quietly assuming all along.
Read →Make the unsafe path impossible to build
Eight phases moving a live fourteen-daemon system onto one typed-actor kernel, without breaking it once. The methodology that held: not 'remember not to do the dangerous thing' but 'make the dangerous thing structurally impossible to construct.' The difference is the whole story.
Read →Three walls in vLLM structured output — and the method that found them
A local LLM that only thinks between encounters meant putting Qwen behind a strict JSON contract. Here are the three substrate walls that cost me time — and the working method that surfaced each one, because the method is the part worth copying.
Read →How do you know if your AI actually learned anything?
I built two AI entities that adapt to each other between encounters. Then I had to answer the harder question: is the adaptation real, or does it just look real? The honest answer this time was 'not yet' — and the reason that's the post worth writing is that the system could prove it.
Read →The bug disguised as a parameter
Two findings from Diametric's build floor. Both first looked like 'tune the number,' and both were really 'change the structure.' The useful part isn't the bugs — it's that the system was built so it could hand them to me instead of hiding them. The architecture that can contradict you is the one worth having.
Read →