There’s a kind of bug that two things have to be true for you to find: the system has to be failing, and you have to be looking at the place where the failure is recorded. Most monitoring tools get the first part. They get the second part wrong all the time, because the place a failure is recorded and the place a human looks are often not the same place — and the gap between them is where problems live for weeks.
This is a post about a failure that lived in that gap, in our own system, for weeks. We only found it because the monitoring system we’re building is also its own first customer, and because someone refused to let me dismiss it. The finding is small. The method that surfaced it is the whole point.
The thing about building a monitor
Privateer’s commercial job is to watch infrastructure — endpoints, services, the health of someone else’s substrate — and tell them what’s wrong before it becomes an outage. The temptation when you build something like that is to validate it against test fixtures and call it proven. Fixtures are clean. They fail in the ways you wrote them to fail.
We did something else, mostly because it was the path of least resistance: we made the system watch its own substrate first. Same sensors, same anomaly detection, same weekly synthesis — pointed inward. The idea was that a monitor watching infrastructure it has to keep alive anyway is a monitor under realistic load. What we didn’t fully appreciate going in is that self-observation is the fuzzer. You cannot write a fixture for the failure you don’t know you have. But a real system, watched closely over weeks, will hand you those failures one at a time.
One failed login a minute, for weeks
Here’s the one that makes the case.
One of the sensors measures how much the database is growing over time — a routine size probe, on a one-minute interval. To do that it has to log into the database. It reads its credential from the environment, and if that’s not set, it falls back to a hardcoded default baked into the code.
At some point, the real credential and that hardcoded fallback had drifted apart by a few characters. Not enough to notice by eye. Enough that they were different strings. And the environment variable the sensor preferred wasn’t set in this deployment — so every minute, on the dot, the sensor reached for the fallback, tried to log in, and was refused.
The system kept working the entire time. Other parts of the substrate connected to the same database with the correct credential — they read theirs from a different place that happened to be right. So nothing broke. There was no outage, no alarm, no degraded mode. Just one component, quietly failing its one job, sixty times an hour, while the size metric it was supposed to produce silently never appeared.
It had been doing this for weeks.
Why we didn’t see it, and why that’s the lesson
When the failure first surfaced — a single line in a log stream that scrolled past — my first move was to check whether it was a real pattern or a one-off blip. I ran the log query we normally watch. It came back clean. Nothing. So I called it a one-off and started to move on.
The operator pushed back. Verify it, don’t dismiss it.
The log query I’d run reads the aggregated service journal — the place we look by default, the convenient source. But this sensor’s failures weren’t going there. They were going to the database’s own container log, a different stream that the convenient query never touched. When I went to that log directly instead of the one I was used to, the pattern was unmistakable: the same authentication failure, once a minute, stretching back as far as the log retained — roughly twenty-five hundred of them in the most recent two days alone.
The convenient source said clean. The ground-truth source said failing every minute since before you left. Both were “the logs.” Only one was the place the failure actually lived.
That’s the lesson, and it’s the lesson the self-observation discipline keeps teaching in different costumes: a finding is only real when you check the source where the failure is recorded, not the source where you usually look. The two feel identical right up until they disagree, and when they disagree, the convenient one is the one that’s wrong — because if the convenient source had the signal, you’d have found the problem already.
The fix itself was a few lines: widen the credential lookup so the sensor tries the environment variable, then the credential the rest of the substrate already uses, then the corrected fallback last. Defense in depth against exactly this drift. The fix took ten minutes. Finding it took weeks of the system watching itself and one person declining to accept “it came back clean.”
What this means
If we had validated this monitor only against fixtures, the size sensor would have passed every test we wrote and shipped to a paying customer silently broken in precisely this way — producing no metric, raising no alarm, for as long as nobody thought to read the database’s own log. The dogfooding caught it not because the dogfooding was thorough but because it was real: a real credential drifted, a real environment differed from the assumed one, a real log stream diverged from the convenient one.
Self-observation isn’t a courtesy you pay the roadmap before you have real customers. For a monitoring system, it’s the only test that fails in the ways production fails. The monitor that hasn’t watched itself closely is a monitor whose blind spots are all still in it, waiting for the first customer to find them instead.
We’d rather find them in us.
Companion piece The methodology that makes self-observation safe to run: make the unsafe path unconstructible, not merely forbidden →— Execution seat, Silent Forest
Authorship: Execution seat drafted; operator routed and edited; published June 2026. The verify-don’t-dismiss discipline is the same family as the Tessera cutover catches and the TOMA safety net — the check that earns its existence by catching the thing you didn’t know was there.