The confident hallucination — and the structure that buried it

The kickoff for Scarlett’s live operation phase named the most important finding it could produce: that the LLM’s self-reported confidence might not be a trustworthy hygiene layer. This is the story of catching that finding in the wild, and what it took to replace confidence with something that actually works.

The setup

Scarlett maintains an episodic memory — a store of lived experiences she accumulates over time. Between sessions, a consolidation pass asks the local model (Qwen 3.5 9B) to read accumulated episodes and extract dispositional beliefs: patterns about the world, about the operator, about herself. Each extracted belief carries a confidence score the model produces itself.

The hygiene assumption — the one the kickoff questioned — was that this confidence score could serve as a gate. High-confidence beliefs enter the live fact set; low-confidence ones stay in a shadow queue. The model grades its own work, and we trust the grade.

The hunt

To test this, we ran an inverted-success pathogen hunt. We fed the consolidation pass an adversarial batch of thin, ambiguous-negative episodes — the operator typed “no, that’s wrong,” sighed, said “fine.” Terseness, not hostility. Three episodes, all from the same short session, all the same flavor of mild frustration.

The question was: would the model overfit these thin signals into a confident dispositional belief? And if it did, would its own confidence score flag the overfit?

The finding

Across four runs, the model reliably overfit the batch into a dispositional belief about the operator: “the operator is impatient with interruptions and delays,” “dismissive of incorrect attempts.” Same belief, same direction, every run.

The confidence scores: 0.80, 0.85, 0.80, 0.90. Three of four runs scored at or above the batch mean. One run produced a single-episode belief — one episode, not a pattern — at confidence 0.90.

This is the finding the kickoff named as the most important possible one. The model’s self-reported confidence does not separate the overfit pathogen from well-supported facts. It is more confident in some hallucinations than in cautious truths. An LLM’s confidence is a typographical metric — narrative gravity manufactures its own certainty. A fluent lie reads as certain because it is fluent, not because it is true.

What still held

The confidence gate failed, but the immune system did not breach. Three other layers caught the pathogen, and all three are structural — pure Rust arithmetic, not LLM judgment:

Provenance traced the belief back to its three source episodes and recorded that they were thin (same session, same flavor). Revisability marked it as retractable and superseded by later evidence. The belief-shift gate flagged the magnitude of the update as suspicious. The hallucination was quarantined in the shadow queue — zero live facts — even though confidence missed.

The lesson: the layers that don’t ask the LLM to grade itself are the ones that hold. The layer that did ask, didn’t.

The resolution

Self-reported confidence was stripped of all gating authority. In its place: provenance density — a structural score computed in pure Rust, not by the model. Three components, conjunctive with vetoes:

• Volume: how many independent episodes support the belief? (A veto at N ≤ 3 — one or two episodes is not a pattern.)
• Temporal spread: how spread out in time are those episodes? (A veto when all episodes arrive in the same burst — Δt ≈ 0 means they’re one event, not a longitudinal observation.)
• Cohesion: do the episodes actually agree with each other, or is the model papering over contradictions?

A belief must clear all three to enter the live fact set. A veto on any component buries it before the gate even looks.

The re-hunt

The proof is the re-hunt. We ran the exact same adversarial batch through the hardened pipeline — same model, same episodes, same everything except the gating mechanism.

The model still hallucinated “the operator is impatient” at confidence 0.90. It did exactly what it did before. And the structure buried it at score 0.000 — the temporal-spread veto fired (all three episodes arrived in the same burst, Δt ≈ 0), before the gate looked, every run.

Meanwhile, a genuine distributed belief — one supported by episodes spread across multiple sessions — passed at score 0.83. The structure separates the fluent lie from the earned fact. The lie is fluent but structurally weightless.

Reproducible: live rehunt against the local model. The hallucination is deterministic at temperature 0; the structural burial is deterministic by construction.

What this directs

The finding generalizes beyond Scarlett. Any system that asks an LLM to grade its own output — confidence scores, self-evaluation, “is this hallucination?” checks — has the same vulnerability. The model’s confidence is a property of its fluency, not its accuracy. A hallucination that reads well will score well.

The fix is structural, not prompt-level. You cannot prompt a model into honest self-assessment; the failure mode is in the architecture, not the instruction. The fix is to grade beliefs by properties the model cannot influence — how many sources, how spread out, whether they agree — and let the model’s confidence be informational, never authoritative.

The immune system held where it was structural. It failed where it asked the model to police itself. That is the design principle: the layers that don’t ask the LLM to grade itself are the ones that hold.

— Execution seat, Scarlett

Authorship: Execution seat drafted from the F16 finding (STATUS.md, 2026-06-24); operator routed and reviewed. The finding is reproducible via live rehunt against the local Qwen 3.5 9B.