Making the Deploy Wrong on Purpose

Last night I ended the post about what my deploy believes on a question, can I make a belief false on purpose and watch the pipeline refuse, and this morning it was still sitting there, so this is the answer I am going to try to earn, written down before any of it runs.

Twice this fall my pipeline was wrong about itself with every light green. A flag that said dirty on every build, a crawl that expected the wrong thing from sixty pages, and neither one showed up as a failure, they showed up as weather, the kind of thing you stop seeing. The pipeline reported SUCCESS and FAILURE the whole time and was right about almost everything except the part that mattered.

So what I want to test is small and plain, that something sitting outside the pipeline, only watching and never touching anything, can catch release mistakes that SUCCESS and FAILURE never show.

Why write it down first

Because I already know how this goes when I don't. I build a check, it catches something, I feel good, and nobody ever asks what it missed or what it would have caught anyway. Writing the hypotheses and my guesses here, dated, before the first fault goes in, is the only way the results can surprise me. If I am wrong I want it on the page next to the part where I said I'd be right.

It is also the third loop from last night, done in public. The experiment is a test of how my pipelines gather their evidence, and the post is a test of how I gather mine.

What gets watched

Not only this blog, the same protocol runs on everything I own outright, and nothing else.

System	What a release is	Why it is in
joshuaayson.com	an artifact built on staging, compared file by file, promoted to production	the deploy from last night's post, with six beliefs already written down
7helm	a tagged release, built once, copied to staging, promoted to production by me	a different shape entirely: a site, a game server, an event API, real players
the other Organic Arts sites	smaller static deploys	to see whether the protocol holds on something simple

For every production release the watcher writes one record. What the pipeline reported. Which artifact was approved, which one staging served, which one production serves. The jobs it expected to run before the promote and the ones that actually did, in what order and when. What each target is actually running. And what the watcher concluded, which is never a bare pass or fail. It is one of three things.

Finding	Means
violation	a belief I wrote down was broken
anomaly	something unusual happened that broke no belief I know of
insufficient evidence	it can't tell, and that is never counted as a pass

That last row is the one I care about most. A check that can't see and says green is how I lost six days in October.

The hypotheses

	I think	The boring answer	What would convince me
H1	a watcher that checks the artifact, its path and its timing catches mistakes the pipeline's own status and checks miss	it catches nothing they don't	it catches more, by enough that the interval leaves zero out, without crying wolf on more than one release in twenty
H2	when a miss happens, fixing the assumption behind the check helps more on new kinds of mistakes than just adding another check	both fixes do the same on mistakes they never saw	assumption fixes catch more on fault families held back from the fixing
H3	an AI-assisted watcher adds something beyond plain rules	everything it adds comes from the rules	it beats the rules-only watcher on the same cases, without the false alarms going up past the line in H1

Every miss also gets one more label, whether the gap was a missing check or a wrong assumption about how the deploy works. The flag in October was a wrong assumption. I think most of them are. That split is the thing I actually want to know.

Making it wrong on purpose

Real mistakes are rare, which is good for the site and bad for an experiment, so most of the evidence comes from mistakes I put there myself. Never in production. On staging, and in Fidelitron, the small reduced worlds I build to test exactly this kind of question, where 7helm's delivery path is the next world on the list.

Fault	What it looks like
stale artifact	production gets bytes from an earlier build than the one approved
unexpected rebuild	the artifact is rebuilt along the way instead of copied
missing predecessor	a job that should have run before the promote never did
out of order	the promote runs before the thing it depends on finishes
config drift	a target runs the right build with the wrong settings
late evidence	a check samples after the thing it measures, the October flag on purpose
wrong definition	a check expects the wrong thing of a kind of page, the sixty redirects on purpose

The faults are split by kind, not at random. I fix assumptions using some families and score on the families the fixes never saw, and that test set gets frozen before any fixing starts.

Each case is scored three ways. The pipeline as it is today. A watcher with plain rules about the artifact, its path and its timing. And the same watcher with a model in the loop, proposing what might be wrong and pulling the evidence together. The model is a tool here, not the judge of what is true. A person labels every case, and the labeler doesn't know which watcher flagged it.

My predictions

	My guess
H1	the plain-rules watcher catches most of the injected faults the pipeline misses
H2	wrong assumptions outnumber missing checks among the misses, about two to one
H3	the AI-assisted watcher adds little to detection and more to the time it takes me to understand a finding
the hardest fault	late evidence, because the October flag hid for weeks in plain sight
the first surprise	7helm, because its shape is the one I have written the fewest beliefs about

Frozen on 2026-11-10, before the first fault goes in. Whatever the results say, this table does not change.

How it gets judged

Measure	For
how many faults each watcher catches, and how many of its alarms are real	H1, H3
false alarms per release	the line in H1
how long until it noticed	all three
how often it says insufficient evidence	keeping the watchers plain about what they can't see
minutes it takes me to understand a finding	H3
missing check versus wrong assumption	the question underneath

The watchers are compared on the same cases, pair by pair. To tell a difference of a fifth in catches from noise takes somewhere around fifty to a hundred faults per comparison, which is why the injected faults carry the test and the real releases carry the examples.

What happens next

Step
1	the protocol, the record and this plan go into the public Complexence repo and get a version tag
2	the watcher runs in the shadow on this site, 7helm and the other OA deploys, read only, for four to six weeks, and the data gets published
3	the fault suite gets built in Fidelitron and on staging, and the held-back families get frozen
4	the three-way comparison runs, the results go up next to these predictions, and the protocol gets fixed where it broke

The watcher never gets to stop, block or fix anything. If it could, it would be part of the pipeline, and then I would need something to watch it.

The questions I am carrying in

The same ones from last night, now with a way to answer them. What does each pipeline believe about its release? Where does each belief get its evidence, and is that evidence gathered before or after the thing it measures? Which beliefs have never once been tested? And when I make one false on purpose, does anything say so?

I'll tag each record with that day's decan and the moon too, because my log does, and I expect them to have no effect at all. No tests get run on them. They ride along so that if something strange shows up there I see it, and so that I am not tempted to go looking.

Where this sits

If you want	Go to
the six beliefs and the two that were wrong	What the Deploy Believes About Itself
the checks that could not fail	Every Check Passed
a loop made wrong on purpose and measured	The Regulator Was Fed Intents
observed systems and observing systems	Anything Said Is Said by an Observer
the reading path	Cybernetics