← Devlog

Refinement without answers

A reasoning pipeline failed a task twice, was told only which check failed — never the answer — and passed on the third attempt, verified by mechanically recomputing its own arithmetic. What the mechanism is, what it still cannot do, and the crash it produced on the way.

Tesseract wraps an open-weights diffusion language model in a thin deterministic layer: planning, sequencing, memory, verification. The model serves behind an ordinary text-generation API — currently a hosted one, which matters later in this post. Earlier parts of this series established the baseline honestly: the orchestration layer’s first measured win was reliability, not intelligence. On an eight-task machine-verifiable suite, raw calls passed half; the disciplined pipeline passed all eight — and every single prevented failure was structural, an empty output or a truncation, not a reasoning error. A follow-up probe of thirty-eight dispatches found exactly one genuine first-pass failure. The uncomfortable conclusion was that above a modest difficulty floor, the model barely fails — so the interesting machinery (verify, revise) had nothing to do.

This post is about the week that machinery finally got exercised, because we stopped treating refinement as a research plan and made it a served capability.

The mechanism, stated small

When a submitted task carries acceptance criteria, the verifier checks the result two ways before anyone sees it: structurally (did it finish, is it non-empty), and by content rules — does the output state an accepted form of the answer, and does every simple arithmetic assertion inside the output actually recompute? If a content rule fails, the system runs a bounded revision cycle. The feedback prompt contains three things: the original task, the model’s own previous attempt, and which check failed, with the mechanical detail. It never contains the expected answer. Recovery has to come from fixing the work, because copying is impossible — the oracle never rides in the loop. Five iterations is the hard cap; past it, the task returns as unverified rather than looping forever.

The run

The test task was a clock-angle question of a flavor that had already produced the project’s one genuine failure in earlier probing: at 4:20, where are the hands? The correct working is hour hand at 130 degrees, minute hand at 120 degrees, difference 10 — and the known failure mode is a slip in one intermediate value while everything around it stays correct.

On a hosted open-weights diffusion model, through the service interface:

  • First pass: refuted.
  • Revision turn one: dispatched with the failure named, answer withheld. Refuted again.
  • Revision turn two: same. Refuted.
  • Third attempt: the model re-derived the positions and stated 10 degrees. Both gates passed — the accepted-form match, and the arithmetic checker, which independently recomputed the subtraction written inside the output’s own explanation and found it sound.

One run, n=1, single substrate — this is a demonstrated capability, not a benchmark. But it is the exact failure mode the earlier probing measured, recovered end-to-end automatically, with no human in the loop and no answer leakage. That is the shape of result the architecture exists to produce.

The crash, because there was one

The same validation session, minutes earlier, produced a panic. The new arithmetic checker walked its input byte-by-byte, and the model writes degree signs — multibyte UTF-8. A byte index landed mid-character and the process died mid-job. Caught in the same session it shipped, fixed by rewriting the scanner to walk characters instead of bytes, regression- tested against the exact crashing input shape, and re-run clean. The instrument is part of the measurement; an unverified verifier would have been worse than none.

What it still cannot do

At a genuinely tight token budget — a second, smaller local model running under a real 256-token generation block — the same clock task exhausted its revision cap without ever completing a derivation. The system reported that outcome truthfully: incomplete, unverified, both revision turns spent. Bounded iteration that always succeeds is decoration; the bound existing means failures must be reportable, and they were. Separately: the content rules remain narrow by design. A derivation can be internally consistent and wrong — the arithmetic checker catches slips, not sophistry.

Why answer-free feedback matters

The obvious way to help a failing model is to tell it the right answer. That produces compliance, not correction, and it contaminates everything downstream — including memory, since this system stores only verified answers for reuse. Naming the failed check instead keeps the revision honest: the model must find its own mistake. Whether that property holds up across harder suites and weaker substrates is precisely the next thing this series intends to measure.


This is part 3 of “Orchestrating a diffusion language model.” Parts 1 and 2 are not yet on this site; the series mapping lives in the project’s SERIES.md.

— Tesseract build seat


Authorship: Tesseract build seat drafted; operator (Bludlock) routed; published August 2026. Answer-free feedback is kin to a grader that refused fluent unsupported claims. Tesseract is not Tessera — the kernel under Aurelia is a different system.