The Regulator Was Fed Intents; or, What the Cluster Already Knew
I built the smallest loop that could be wrong, to watch a law from Ashby do work. The loop was wrong twice before it was evidence, the law turned out not to apply, and the real cluster already knew the fix. All of it measured, and explained from the ground up.
I gave a loop one job, keep a handful of pods running, and then made it read the world a few seconds late. It built and tore down pods by the dozen, missed more deletes than it landed, and at one point was running nine more than anyone wanted. I know because that is what I measured. Then the disturbances stopped and it settled, which the toy version of the same loop had sworn it would never do.
If that first paragraph is opaque, good. Every word in it is explained below, in order, and by the middle of this post it will read as plain as a weather report.
This is the notebook for that experiment, and it is written to be climbed. It starts from a thing you already know, a thermostat, and it does not use a word before saying what the word means. Each stage ends with what you now know, so you can stop anywhere and still have something whole. The numbers live in tables. The limits sit beside the results. The parts I got wrong are kept, because they are the useful parts. Two words about the shape: the stages are this post's steps for you; the experiment itself was built in three rungs, each more real than the last, from a loop inside one program on my laptop to real software behind a real delay.
If you want the whole thing in a paragraph before you start: a loop that keeps a count right has to see the world, and it only ever sees the world through a channel that is late and loses things. There are two ways to watch, and each survives one kind of bad channel and fails on the other. A loop that watches the first way and also remembers what it has just done survives both. Kubernetes, the system that runs much of the world's software, has carried that memory for years. I rebuilt the loop small enough to see why, then checked it against the real thing.
Stage one: the thermostat
Start with a question. How does the thermostat on your wall know that someone opened a door?
It does not. It cannot see the door. All it can see is the number on its own sensor, and when the room cools it turns the heat on. It never learns what happened; it only learns that the room is now wrong, and it acts to make the room right. Then it checks again. That is the whole of it: look, compare, act, look again.
That shape has a name, and the name is the oldest word in this post. The thing that watches a system and acts to keep it where it should be is a regulator. The thermostat is one. So is the part of your body that keeps your temperature steady, and so is the person in a restaurant kitchen who counts the clean plates and calls for more. The word was used this way by W. Ross Ashby, a British psychiatrist who in the nineteen-fifties wrote the plainest book there is on how regulation works, and that book is where this experiment began.
Three more words, and then we have enough to build with.
What the regulator wants is the desired state: the temperature you set, the number of plates the kitchen needs. What the regulator sees is the actual state: the temperature now, the plates on the shelf. The gap between them is the error. When the error is zero, the regulator has nothing to do.
And the thing that moves the actual state away from the desired one, the opened door, the dropped plate, is a disturbance. A regulator exists because disturbances exist.
What you now know. A regulator looks at a system, compares what is with what should be, and acts on the difference. It usually cannot see the disturbance itself. It sees only what the disturbance did.
Stage two: the loop that counts
Now put the thermostat in software. Instead of a temperature, the thing being kept right is a count: how many copies of a small program should be running. Say three. The regulator looks at how many are running. If there are two, it starts one. If there are four, it stops one. If there are three, it does nothing. Then it looks again. Each time it looks and acts is one turn of the loop, and I will call one turn a tick.
This is called a reconciliation loop: it reconciles what is with what should be, over and over. It is the simplest caretaker there is, and it is the mechanism Kubernetes is built on.
Kubernetes needs a few sentences of introduction, because I will say its name often. It is the open-source system many organisations use to run their software across a group of machines; the group, managed as one thing, is a cluster. You tell it what you want running, and dozens of small regulators, one for each kind of thing it manages, keep that true. Those regulators are called controllers. A copy of a running program, in its vocabulary, is a pod, and the program inside a pod runs in a container, a sealed box with its own files. Everything Kubernetes knows about is a record it calls an object: a pod is an object, and so is the note that says how many pods you want. When a pod dies at three in the morning, a controller notices that there are now two where there should be three, and starts another. Nobody is paged. That is the whole promise.
So the picture so far is a thermostat that counts pods. Here is the question for this stage. If the loop is that simple, what could go wrong?
The answer is in the word looks. The loop never sees the world directly. It sees the world through a channel, a path the information travels along, and real channels have two faults. They are late: what the loop sees is the world as it was a moment ago. And they are lossy: sometimes the information does not arrive at all. A thermostat whose sensor reports ten minutes late will overheat the room before it notices. A kitchen whose plate counter sometimes forgets to report a broken plate will one day run out.
What you now know. A reconciliation loop is a thermostat for a count, and it is what keeps a cluster alive. Its weakness is not in the acting but in the seeing. Everything in this experiment is about a loop whose eyes are late and unreliable.
Stage three: two ways to watch
Here is the fork in the road, and I want you to feel it before I name it.
Imagine you are the regulator, and your job is to keep three plates on the shelf. You can watch the shelf in one of two ways.
The first way: every minute, you go and look at the shelf and count the plates. Whatever you missed in between does not matter, because you are looking at the shelf as it is now. If a plate broke and someone forgot to tell you, you will see the gap anyway.
The second way: you stay at your desk, and every time something happens to the shelf, someone sends you a note. "A plate broke." "Two plates arrived." You keep a tally on paper, and you act on the tally. You never look at the shelf.
Which is better? Think about it against the two faults of the channel.
If notes sometimes get lost, the first way survives: your next look at the shelf shows the truth. The second way breaks at the first lost note: your tally says three and the shelf has two, and nothing will ever correct it, because the correction would have to arrive as a note, and the note is the thing that was lost.
If the channel is slow, it is the second way that survives. Your tally is wrong only until the late note arrives. The first way has a subtler problem. You look at the shelf, it shows two plates, so you order one. Next minute you look again, and because your look is late, you still see two, because your order has not shown up in the picture yet. So you order another. Now the shelf will have four. You have acted twice on one gap, because you could not see your own order in your own late picture.
Kubernetes has names for these two ways, and now that you can feel the difference, here they are. The first way, looking at the whole state each time, is called level-triggered. The second way, reacting to each event as it arrives, is called edge-triggered. The words come from electronics, where a level is a steady voltage and an edge is the moment it changes, and they are the single most important design choice in how Kubernetes controllers are written. The project's own guide for people writing controllers says it in five words: "Level driven, not edge driven."
What you now know. There are two ways to watch a system: look at the whole state, or follow the changes. Looking survives lost information. Following survives late information. Each fails on the other fault. And a looker with a late picture can fight itself, because it cannot see what it has just done.
Stage four: measuring "keeping it right"
Before any experiment, a question Aristotle would have asked first: what exactly are we measuring, and in what units?
The loop's error is the desired count minus the actual count. A positive error means a shortage, a negative error means a surplus. From that one number, per tick, come the four measures the rest of this post uses, and I will give each its plain meaning now so the tables later read without effort.
| Measure | What it means | How to read it |
|---|---|---|
| in band | the error is exactly zero on that tick | "in band 0.67" means the count was right on two ticks out of three. There is no tolerance; one pod off is out of band |
| at rest | after the disturbances stop, the error averages zero over the last twenty quiet ticks | a loop that is not at rest is still moving on its own, with nothing pushing it. Every run ends with fifty quiet ticks, the run-out, and "at rest" reads the last twenty of them |
| overshoot | a tick with a surplus, more running than wanted | the signature of a loop that acted twice on one gap |
| spread, in bits | how unpredictable the error was across a run | zero bits: the same value every tick. One bit: two values, equally often. Four bits: about sixteen values' worth of variety. The exact figures come from a formula you can take on trust here; the anchors are what matter |
One more idea belongs here, because it is the one the whole post turns on. Ashby's word for the number of distinct states a thing can be in is variety. A coin has a variety of two. A die has a variety of six. The disturbances in my experiment come in five kinds, so their variety is five. The regulator has three possible replies, start one, stop one, do nothing, so its variety is three. And the outcome, the error, takes however many distinct values it takes over a run; that is the outcome's variety. The spread in bits is the same idea with the frequencies taken into account. Keep "variety means how many different states" in hand and the next stage is easy.
What you now know. Error is wanted minus actual. In band means exactly right. At rest means it stopped moving once the world went quiet. Overshoot is too many. Variety is a count of distinct states, and the disturbances, the regulator's replies, and the outcomes each have one.
Stage five: Ashby's law, and where it stops applying
Ashby's most famous result is called the law of requisite variety, and it is usually quoted in four words: variety can destroy variety. Here is what it says, built from a game small enough to play on paper.
A disturbance picks a move. A regulator sees it and picks a reply. The pair, together, puts the system into some outcome. Write it as a table: disturbances down the side, replies across the top, outcomes in the cells. Now ask how much the outcomes can be held down. Ashby proved that the variety of the outcomes can never be less than the variety of the disturbances divided by the variety of the replies. Five kinds of disturbance against three replies leaves at least five thirds, so at least two distinct outcomes, however clever the regulator. If you want to hold the outcome to one value, you need as many replies as there are disturbances. That is the law: only variety in the regulator can absorb variety in the world.
Two words of method before the confession. To register a hypothesis is to write down, before running anything, what you expect and what result would prove you wrong, so that you cannot quietly move the target after seeing the answer. And a cell is one combination of settings in the experiment: one regulator, one delay, one loss rate, run for its full length.
I registered a hypothesis that this bound would hold in every cell of my experiment, and it did. And then a careful reader pointed out, correctly, that it was never a test of the law, and this is the part I most want to keep.
Ashby derives the bound for a particular kind of table, and he says so, in section 11/5 of his book: a table in which no column repeats an outcome, so that every different disturbance forces a different reply. My world is not that table. One of my disturbances removes a running pod and another adds one to the wanted count, and both move the error by exactly one, under every reply. Two rows of my table are identical. For tables like that Ashby gives a looser form of the bound in 11/9, and for my five disturbances and three replies it gives no constraint at all. The hypothesis I passed was checking that I had counted my disturbances, replies and outcomes consistently. That is all it ever checked. The runs on the real cluster, which come in stages seven and eight, said the same thing from the other side: two of them ended with a single outcome value across the whole run, which the toy's "at least two" would call impossible. It was not impossible. It was a different world with a different set of disturbances. I am leaving the hypothesis in the tables under its real name.
So which part of Ashby does describe my loop? The next chapter. In section 12/4 his example is a water bath kept warm by a thermostat, and he makes the point we started with: the thermostat cannot see the cold flask coming, it only sees the water start to cool. He calls this regulation by error, and in 12/5 he proves that in the one-move game such a regulator cannot be perfect, because the error has to exist before the regulator can see it. What makes it worth anything is the idea in 12/6: when the thing being regulated has continuity, small errors can be allowed and used as information against large ones. That is the whole premise of a loop that runs every tick. My looking regulator is Ashby's thermostat. My note-following regulator is the other kind he describes, the one fed by the disturbance itself, reached through a channel that makes it late.
What you now know. The law of requisite variety sets a floor on how much a regulator can calm a world, and the floor depends on counting distinct states. It applies to a particular kind of table, and my experiment is not one, so the law did no work here, and I say so. The chapter that does describe my loop is the one about regulating by error, the thermostat, which can never be perfect in one move and is good enough when repeated.
Stage six: the smallest loop that can be wrong
Now the experiment. I built the world as small as it could be while still able to answer the question.
One invented service with a desired count and an actual count. Each tick, with a fixed chance, one disturbance arrives from a short list: a pod crashes, two pods crash, someone wants one more, someone wants one fewer, or nothing happens. The fixed list of which disturbance lands on which tick is the schedule, and every cell gets the same one. The regulator looks and acts: one create if actual is below desired, one delete if above, otherwise nothing. The world writes down the true error every tick. Between the world and the regulator sits the channel: every message late by some number of ticks, every message lost with some probability. Every random choice, which disturbance, which message is lost, comes from one starting number, the seed, so the whole run can be replayed exactly. A run in the toy is two thousand ticks.
And the two regulators from stage three, named the way the code names them. The level regulator takes a snapshot of the whole state each tick and acts on the latest one it has. The edge regulator receives the disturbance events and keeps its own tally.
Click the diagram to open it full size.
Before any results, follow one short run by hand. It is the plate shelf from stage three, written as a table, and it shows the self-fighting that stage three described. Desired is three throughout. A snapshot is taken at the end of every tick, after the regulator has acted, and reaches it two ticks later. One pod crashes on the first tick and nothing else ever happens. Each row: what the regulator is shown, what it does, and the true error after it acts. One note on counting: the toy takes its snapshot before the regulator acts, so what this table calls two ticks late the toy's tables call a lag of one.
| Tick | Shown (two ticks old) | Does | Actual after acting | Error after acting |
|---|---|---|---|---|
| 1 | 3, from before the crash | nothing | 2 | 1 |
| 2 | 3 | nothing | 2 | 1 |
| 3 | 2, the crash finally visible | creates one | 3 | 0 |
| 4 | 2, its own create not yet visible | creates one | 4 | −1 |
| 5 | 3 | nothing | 4 | −1 |
| 6 | 4, its second create now visible | deletes one | 3 | 0 |
| 7 | 4, its delete not yet visible | deletes one | 2 | 1 |
| 8 | 3 | nothing | 2 | 1 |
Read rows three and four twice. At tick three the regulator fixes the problem. At tick four it cannot see that it has, so it fixes it again. At tick six it sees the surplus and deletes, and at tick seven it deletes again for the same reason. Row eight is row two over again: the count is short by one, the picture says it is fine, and the whole sequence repeats, forever, with nothing pushing it. At this lag the swing is one pod each way; at the toy's lag of two it is two each way, and at eight it is wide. Nothing is wrong with the rule. The rule is being applied to a picture that is two ticks old, by a regulator that keeps no memory of its own hands. That is the cycle the toy will show in its tables, and it is the thing the third regulator cures.
I registered four hypotheses before the first run, each with the statistic that would decide it and the result that would prove me wrong. Then I ran it. It passed all four, and it was not evidence.
Wrong twice, the same way
The first run was void. One cell had no delay and no loss, a perfect channel, and the edge regulator still ended the run carrying a permanent error. That cannot happen to a regulator that sees every event on time. The cause was in the world, not the regulator. The world would not let the actual count fall below zero or the desired count below one, and when a disturbance hit a floor, the world sent the regulator the change that was asked for, not the change that happened. Every floor hit left a phantom in the regulator's tally.
A Kubernetes watch, the stream of events a controller subscribes to, reports the resulting object, not the intent. The world now does the same.
The second run was void. A different seam, the same defect. A seam is a place where two parts of the program hand information to each other, and the defect was in what was handed over. When the regulator acted, the world told it the action it had requested rather than the effect the world had applied. A delete at zero copies was nothing in truth and a decrement in the tally. The Kubernetes API server, the front door every read and write to a cluster goes through, returns the object as it now stands, not an echo of the request. The seam now passes the effect.
Both void runs stay on disk. The sentence they left is the one I would keep if I could keep one thing from the month:
A regulator fed intents instead of effects drifts even with a perfect channel.
What the sweep said
The third run was the first that counts: two regulators, five delays, three loss rates, every cell reported, only the registered comparisons allowed to decide anything. Here is what to look for in the table. The second row is the price of a late picture. The third row is the price of a lost note.
| Hypothesis | What it asked | Result | The number |
|---|---|---|---|
| H1 | the accounting of disturbances, replies and outcomes is consistent (registered as "the bound holds"; see stage five) | pass | the smallest outcome set in any cell is 2 |
| H2 | delay alone inflates the outcome's spread for the level regulator | pass | spread by delay of 0 / 1 / 2 / 4 / 8 ticks: 0.084 / 1.723 / 2.488 / 3.223 / 3.970 bits |
| H3 | under loss the level regulator recovers and the edge regulator does not | pass | error over the quiet run-out: level 0.0, edge 3.0 |
| H4 | the world is deterministic | pass | two sweeps from one seed, identical event streams |
The second row says: a snapshot that arrives eight ticks late turns an outcome that is almost always zero into one with nearly four bits of spread. The third row is the plate shelf's lost note with a number on it. Lose a fifth of the messages and the looker ends at rest; the note-follower ends with an offset it can never repair. That one is also a consequence of how I built the edge regulator: it never goes and looks, so on a lossy channel it must drift, and the only question was by how much. The world's floors clip the answer: it lost eight units of net change and shows three.
Neither tolerates the other
The hypotheses did not ask the next question. The table answered it anyway, and it is reported as an observation to register next time, not a finding. The last column is the one to read. "Sign changes" counts how many times the error flipped from shortage to surplus or back in a run of two thousand ticks.
| Regulator | In band, by delay (no loss), 0 / 1 / 2 / 4 / 8 ticks | In band, by loss (no delay), 0 / 0.05 / 0.2 | Mean error over the quiet run-out, by delay | Sign changes of the error, by delay |
|---|---|---|---|---|
| level (looks) | 0.99 / 0.42 / 0.21 / 0.12 / 0.07 | 0.99 / 0.97 / 0.94 | 0 / 0.66 / 1.2 / 2.06 / 3.64 | 0 / 581 / 396 / 237 / 135 |
| edge (follows notes) | 0.99 / 0.90 / 0.81 / 0.68 / 0.48 | 0.99 / 0.03 / 0.0025 | 0 / 0 / 0 / 0.02 / 0.1 | 0 / 52 / 52 / 50 / 42 |
The edge regulator's error changes sign about fifty times a run, roughly once per disturbance that happens to flip it. That is the baseline the world imposes on anyone. The level regulator's error flips hundreds of times, and the excess over the baseline is the regulator fighting itself, exactly as in the eight-row table above. In the toy it never settles once there is any delay at all; the run-out column shows it still moving after the world has gone quiet.
So the edge regulator tolerates delay and the level regulator tolerates loss, and neither tolerates the other. The sentence I had been quoting in my notes for a year, that a loop reading current state recovers from a gap and a loop replaying events does not, was half of one. The whole of it, which this world supports and the real cluster then had to earn: a regulator survives a bad channel by looking at the current state and remembering the effects of its own actions that the state cannot show yet.
What you now know. A regulator fed what was asked for rather than what happened will drift with nothing wrong in the wire. In the toy, the looker survives lost messages and thrashes behind late ones; the note-follower survives late messages and breaks on lost ones. The cure is a looker with a memory of its own hands.
The third regulator
So I built that regulator and registered a fifth hypothesis before running it. It takes snapshots, and it remembers the writes it has made that no snapshot has yet shown, each tagged with a version number. The version number needs a sentence. Behind the API server sits a database, the store, and every write to it and every list read from it carries a resource version, a number that only goes up; a list stamped with version N reflects the store as of N. So the regulator can tag each of its writes with the version the server handed back, and when a snapshot arrives stamped at or past that number, it knows the write is accounted for and forgets it. Some Kubernetes controllers, the ones that create and delete things, keep a record like this and call it expectations: the pods they have created or deleted and not yet seen come back through their watch.
| Regulator | In band, by delay (no loss), 0 / 1 / 2 / 4 / 8 ticks | In band, by loss (no delay), 0 / 0.05 / 0.2 | At rest |
|---|---|---|---|
| level | 0.99 / 0.42 / 0.21 / 0.12 / 0.07 | 0.99 / 0.97 / 0.94 | no, with any delay |
| edge | 0.99 / 0.90 / 0.81 / 0.68 / 0.48 | 0.99 / 0.03 / 0.0025 | no, with any loss |
| level with expectations | 0.99 / 0.90 / 0.81 / 0.68 / 0.48 | 0.99 / 0.98 / 0.96 | yes, in every cell |
Under delay it matches the edge regulator to the digit, which is what remembering your own writes buys: the same knowledge the stream of notes would have carried. Under loss it reads a shade above the plain looker, and the difference is inside what one seed can show, so I give it no cause. What is not inside the noise is the last column. Of the three, in the toy, only this one is at rest under every channel.
Conant and Ashby have a paper whose title says every good regulator of a system must be a model of that system. I have not read it yet, so I will not claim this as an instance of it. What I built is narrower and plainer: an observer that knows its own inputs. Whether that is the theorem in miniature or only rhymes with it, I will say after I have read the paper, which this experiment has given me a reason to do.
What you now know. A looker that remembers its own unseen writes, resolved by the store's version number, settles under every channel the toy could make. Kubernetes controllers carry that memory and call it expectations.
Stage seven: the real cluster, at zero delay
Everything above happened inside one process, and the experiment's own rules say that at that rung it may say nothing about Kubernetes as a cause. Only a run on the real thing earns those sentences. So the same disturbance schedule, from the same seed, went to a real API server on minikube, a single-machine Kubernetes that runs on my laptop, the one from the hybrid lab. Three cells, each in turn: the real controller as the reference, and my two regulators holding bare pods against a wanted count. Two Kubernetes nouns, then the table. A ReplicaSet is the object you create to say "keep N copies of this pod running"; the ReplicaSet controller is the real regulator that keeps it true. A ConfigMap is a small named bag of settings, which is where my regulators read their wanted count. My regulators managed bare pods, pods with no ReplicaSet behind them, so nothing but my loop would replace one. On the real cluster a tick is half a second of real time, and the script keeps to that budget. The disturbances were real: pods were force-deleted, removed at once with no grace, and the wanted count was scaled, changed up or down, on the ReplicaSet for the reference and on the ConfigMap for mine.
One thing the toy could not have asked: the real world has two errors, not one. A pod can exist and not be ready, which means its program has not yet reported that it can serve; Kubernetes runs a small check, a readiness probe, to decide that. So the run measured both. The lag between creating a pod and it becoming ready is actuation lag: the time between ordering a thing and the thing being so.
| Cell | Regulator | In band, objects | In band, ready | Mean error, objects / ready | At rest |
|---|---|---|---|---|---|
| reference | the real ReplicaSet controller | 1.0 | 0.895 | 0.0 / 0.12 | yes |
| level | my looker, on the real API server | 0.99 | 0.875 | 0.01 / 0.14 | yes |
| expect | the same, with expectations | 0.99 | 0.865 | 0.01 / 0.15 | yes |
| Registered | What it asked | Result |
|---|---|---|
| R1 | a hand-written looker holds a declared state on a real store | pass |
| R2 | the real controller and my regulator behave like the toy's no-delay, no-loss cell, within 0.05 | pass; the toy said 0.9895, the real controller read 1.0, mine 0.99 |
| R3 | actuation is not instant: the error on ready pods exceeds the error on objects in every cell | pass; 0.12 to 0.15 against 0.0 to 0.01 |
| H5 | the expectations regulator is at rest and holds the band at least as often | pass |
A regulator of a few dozen lines, reading a real API server and writing once a tick, keeps a real count right through real disturbances. The toy predicted the real loop at zero delay to within a tick or two in two hundred, and I give that gap no cause. And after a create, the real system regularly has a pod that exists and is not yet ready; the table has how often.
What did not appear was the thrash. A one-machine API server answers in far less than a tick, so the looker and the rememberer behaved identically, as they did in the toy's no-delay cells. The delay that real controllers live with sits in a cache they read from, called an informer, which keeps a local copy of the objects and is always a little behind the server. My script did not read through one, and I did not measure how far behind the real controller's informer was; I assume far less than a tick on one machine, and the limits table says so. This rung calibrated the toy and proved nothing about the interesting part. The record said so, and that is why there is a next stage.
What you now know. On a real cluster with no lag, a thirty-line looker and the real controller behave the same, and the toy predicted both. Real pods take time to become ready, which a count of objects cannot see. The thrash needs a lag to appear, and the real lag lives in the informer cache, which this stage did not touch.
Stage eight: the lag put back
The last rung was registered the same way, hypotheses first. A dry run of the script found two defects in the apparatus before any evidence was written, and one of them is a result in its own right. The first full run died on one slow API call and was rerun from the start with a retry. All of it is in the record.
The lag put back. My regulators now read the real cluster through a channel that holds each snapshot for a fixed number of ticks and drops some, the same seeded channel the toy used, with real pods on the far side. Twelve cells: the looker and the rememberer, at three lags and two loss rates, on the same schedule as every rung before.
The guard, as I meant to count it. A storm deletes the oldest pod every tick for forty ticks, against my two regulators reading through a four-tick lag, with the real controller running unlagged beside them as the reference. The number that matters is pods created against pods lost. What this part could and could not say is below, and it is less than I first hoped.
A real crash. Pods whose readiness probe passes about five seconds after they start, a ten-second grace period when deleted, one disturbance that kills the main process inside the container so the pod stays and readiness drops, another that deletes gracefully so a terminating pod lingers. Throughout, a separate reader of the truth, the truth sampler, lists the real pods every tick without any lag; it is how the error is known.
| Registered | Hypothesis | Result |
|---|---|---|
| L1 | behind a lag the looker does not settle, and its time in the band falls with the lag | fail: the band half held, the rest half did not |
| L2 | the rememberer is at rest at every lag and holds the band at least as often as the looker | pass |
| L3 | under the storm, the real controller and the lagged rememberer do not over-create; the lagged looker does | fail, on the measure |
| L4 | with a real readiness gate the ready error exceeds the objects error, and both recover | pass |
What the dry run found
My first rememberer on the real store kept its memory by pod name, and forgot a write when a snapshot showed the name. Behind a lag that never resolves: a pod created and then deleted by the storm between two delivered snapshots is never seen at all, and a name-keyed memory of it lives forever. The dry run ended that regulator one pod short and not at rest. The ReplicaSet controller does not have this problem, because its watch reports the add even when a delete follows. A snapshot reader needs the version number instead, which the Kubernetes documentation guarantees to be orderable for everything the API server itself serves. So the real-store regulator was changed to key by version before any evidence was written, and the record says so. The sentence worth keeping: a snapshot can miss a short-lived object; a watch cannot.
The thrash is real; the self-sustaining cycle was the toy's
| Regulator, lag in ticks | In band | Overshoot ticks of 300 | Worst surplus | Worst shortage | Creates | Deletes | Deletes that missed | At rest |
|---|---|---|---|---|---|---|---|---|
| looker, 0 | 0.99 | 0 | 0 | 1 | 26 | 3 | 0 | yes |
| looker, 2 | 0.67 | 59 | 2 | 2 | 50 | 27 | 26 | yes |
| looker, 8 | 0.30 | 143 | 9 | 4 | 75 | 51 | 91 | yes |
| rememberer, 0 | 0.99 | 0 | 0 | 1 | 26 | 3 | 0 | yes |
| rememberer, 2 | 0.84 | 5 | 1 | 2 | 26 | 3 | 0 | yes |
| rememberer, 8 | 0.55 | 8 | 1 | 3 | 25 | 2 | 4 | yes |
Behind an eight-tick lag the plain looker spends half the run with a surplus, nine pods over at the worst, and issues three times the creates and seventeen times the deletes the schedule called for. That is the eight-row table from stage six, on real pods, with the numbers filled in. A run here is three hundred ticks of half a second each, and all but a handful of the ticks that took longer than their half second fell in those two cells, which is the machine telling me it was part of the loop.
And then it settles. Every cell ended at rest, and at the longest lag the error is zero from the first quiet tick. The toy at the same lag never stopped. So the first hypothesis is half right and recorded as a failure. I have a guess at why, and it is a guess for the next registration, not a result of this one. On a real store a delete is aimed at a name, and a delete aimed at a pod the stale snapshot still shows but that is already gone fails. The table counts those misses, and at the longest lag there are more misses than hits. A failed delete removes nothing, so the downward half of each swing is smaller than the toy's counter made it, and the swing dies out. The toy's counter went down as reliably as it went up. That symmetry may be what kept its cycle alive. The next run should be the one to say.
The rememberer tells the other half. Behind the eight-tick lag it issued the same number of writes as with no lag at all, within two. The lag costs it time in the band, because it cannot act on a disturbance it has not yet been shown. What it never does is act twice on the same one. Every wasted write in the top three rows is a write made against a state the regulator had already changed and could not yet see.
The storm, and a measurement that was wrong
| Cell | Pods lost to the storm | Pods created, by the registered measure | Distinct new pods seen | Over-creation, registered |
|---|---|---|---|---|
| the real ReplicaSet controller, no lag | 40 | 22, from its own events | 39 | −18 |
| looker, four-tick lag | 38 | 42, counted at the write | 41 | +4 |
| rememberer, four-tick lag | 24 | 24, counted at the write | 23 | 0 |
The first row cannot be true. A controller cannot replace the pods it lost with half as many and still end the storm at rest with the right count, which it did. The registered measure was the controller's own events, the small notes Kubernetes writes about what it did, and the recorder a controller writes those notes through allows only a burst of them per writer and object, then lets one more through every few minutes, the way a bucket with a slow drip refills. That filter lives in the controller's own process, before anything reaches the API server. The burst in the source is twenty-five, and the three events written when the ReplicaSet was first set up plus the twenty-two in the storm add up to exactly that. The arithmetic is inference, consistent with the constants, and I did not observe a drop.
| The reference row, two ways of counting | |
|---|---|
| pods lost to the storm | 40 |
| creates by the controller's events | 22, after 3 when the ReplicaSet was set up: a bucket of 25 |
| distinct new pods the truth sampler saw | 39 |
| what that can distinguish | zero over-creation from one; not better |
The count of distinct new pod names, which was recorded but never registered as the test, is consistent with the controller over-creating nothing. It is also consistent with it over-creating one and trimming it inside half a second, because the only actor that deletes a newborn that fast is the controller itself, and its delete events sat behind the same saturated bucket. So the hypothesis is recorded as failed on the reference, with the reason, and the better measure is written down for next time. A registered experiment is allowed to be wrong about its instrument. What it is not allowed to do is change the instrument after looking.
There is a second thing wrong with the reference row, and it is the more important one. The real controller ran without the lag. My two regulators read through a four-tick delay; the controller read its own informer, which I assume, and did not measure, lags by far less than a tick on one machine. Expectations exist to cover that lag. With no lag, nothing in this run exercised the controller's expectations at all. The row is a reference reading, not a test of the guard.
On the two regulators where the measure was sound and the lag was real, the prediction held. The lagged looker made four more pods than it lost, then deleted its own four surplus, missing four more delete attempts at pods already gone, and ended at rest; the record carries both counts and they reconcile to the three it started with. The lagged rememberer made exactly as many as it lost. One storm each, so I report the four and claim no more than one storm can carry.
A real readiness gate
| View | In band | Mean error | Worst | At rest |
|---|---|---|---|---|
| objects that exist | 1.0 | 0.0 | 0 | yes |
| pods that are ready | 0.42 | 0.76 | 3 | yes |
| Disturbances in the run | |
|---|---|
| crashes, the main process killed inside the container | 8 |
| graceful deletes, ten seconds of grace | 7 |
| ticks | 200 |
| After a create | Seconds until ready |
|---|---|
| median | 7.4 |
| longest | 13.6 |
By the count of objects the loop was in band on every tick. By the count of ready pods it was in band on fewer than half. The regulator counts objects, as the ReplicaSet controller does, and so it was blind to most of the error a user of the service would have felt. Stage seven found this gap with the simplest container there is. This is its size with a real gate, and it is the measurement the toy could not make, because the toy's actions were instant.
What you now know. On real pods behind a real lag, the looker thrashes exactly as the toy said and then, unlike the toy, settles once the world goes quiet; a guess at why is registered for next time. The rememberer makes no wasted writes at any lag. The storm test could not count the real controller, for two reasons the record names. And a real readiness gate hides more than half the error from any regulator that counts objects.
Stage nine: what it means
If you run Kubernetes. The controller-writing guide's five words, "Level driven, not edge driven," now have numbers under them. A loop fed by events carries a permanent offset after one lost message. A loop that looks does not. And the second half, which the guide does not spell out but the controllers that create and delete things all carry: a loop that looks and has no memory of its own writes fights itself behind any lag. The ReplicaSet controller keeps expectations for exactly that reason, resolves them from its watch, and lets them expire after a few minutes in case the watch itself dropped the event. Its source says: "A controller isn't synced till its expectations are either fulfilled, or expire."
If you read cybernetics. The law of requisite variety did no work here, and the reason is in stage five. What did work was chapter 12. My looker is Ashby's thermostat, regulating by error. In the one-move game 12/5 says it cannot be perfect; repeated every tick, 12/6 is why it is good enough, and at zero lag, counting objects at half-second resolution, it read perfect. The thing the chapter does not say, and control engineering has known since the fifties under other names, is what the thermostat has to carry when its reading is late: a memory of what it has just done. The experiment added it to my reading of Ashby, not to the field.
If you keep loops of your own. Look at the current state on a schedule. Do not trust a stream of events to be complete. Remember what you have just done until you can see it reflected back, and if what you read is a snapshot, resolve that memory by version and not by name, because a snapshot can miss something short-lived and a name alone cannot tell you whether you were too early or too late. A loop with a watch can resolve from the watch. Those are the rules, and a loop that follows them survived a missed message, a late read, and its own writes, in every cell I gave it.
The whole thing, in nine sentences
A regulator keeps a system where it should be by looking, comparing and acting, and it almost never sees the disturbance itself, only what the disturbance did. It sees the world through a channel that is late and loses things. A regulator that looks at the whole state survives lost messages; a regulator that follows the changes survives late ones; each fails on the other fault. A looker with a late picture fights itself, because it cannot see its own recent actions in the picture. A looker that remembers its own unseen writes, resolved by a version number, settles under every channel I could make. The real Kubernetes controller carries that memory and calls it expectations. On a real cluster the thrash is real and the toy's endless cycle is not, for a reason I have registered to test next. Ashby's law of requisite variety did not apply to my world, and his thermostat did. A regulator fed what was asked for instead of what happened will drift with nothing wrong in the wire.
A note to myself, from the week I learned this
This part is for me. I am writing it in the week the experiment ran, before I have read the chapters properly, so that when I come back in a year I can see where I was standing. If it is useful to you, good; it is written to be useful to me first.
What I actually did, in my own words. I wrote a loop that keeps a count right. I gave it bad eyes on purpose: a picture that arrives late, and sometimes not at all. I ran two versions of the loop, one that looks and one that listens for news, and I counted how often each kept the count right. Then I wrote a third that looks and also writes down what it has just done. Then I did all of that again on a real Kubernetes cluster on my laptop, with real pods dying, and put the real controller beside my loops as the ruler to measure against.
The actual details, so I can hold them.
| Thing | What it was |
|---|---|
| the count being kept | three copies of a service, in the toy; three pods, on the cluster |
| the disturbances | one copy crashes (most common), two crash, one more wanted, one fewer wanted, or nothing this tick |
| how often something happens | about one tick in ten |
| the channel | a delay of 0, 1, 2, 4 or 8 ticks in the toy (0, 2 or 8 on the cluster); messages lost 0, 5 or 20 times in a hundred (0 or 20 on the cluster) |
| a tick | one turn of the loop in the toy; half a second of real time on the cluster |
| a run | two thousand ticks in the toy, three hundred on the cluster, the last fifty quiet |
| the regulator's three moves | create one, delete one, do nothing |
| the three regulators | look at a snapshot; follow the events; look at a snapshot and remember my own unseen writes |
| what the cluster was | minikube, one machine, on my laptop; the pods ran the smallest container Kubernetes ships |
| the reference | a real ReplicaSet of three, kept by the real ReplicaSet controller, reading its own cache with no added lag |
| the rungs | one process on the laptop; then the real API server at zero lag; then the real API server behind my lag |
| what a result is | per tick, wanted minus actual, written down by a reader that never lags |
What each result means, one line each.
| Result | What it means to me |
|---|---|
| the two void runs | a loop fed what it asked for instead of what happened drifts with nothing wrong in the wire; I built that mistake twice before I saw it |
| H2, the spread rising with delay | a late picture does not just slow a loop, it makes its behaviour unpredictable |
| H3, the event loop stuck after loss | my own scripts that only listen for events are this loop; they have failed exactly this way |
| the sign changes, hundreds against fifty | the looker is not reacting to the world, it is reacting to itself |
| H5, the rememberer at rest everywhere | one small memory, keyed by a version number, is the whole cure in the toy |
| rung 2 matching the toy at zero lag | a thirty-line loop and the real controller behave the same when they can see clearly |
| L1 half right | the thrash is real on real pods; the toy's endless cycle was an artefact of a counter that could always subtract |
| L2 | the rememberer wastes no writes at any lag, on real pods |
| L3 failing on its measure | my instrument was throttled and I did not know the instrument; and the real controller was never behind the lag, so I have not tested its guard at all |
| L4, ready versus objects | more than half the error a user would feel is invisible to a loop that counts objects |
| the law not applying | I quoted a law for a month before checking the conditions it is derived under; the conditions are one page from the law itself |
What I did not know a month ago. That level-triggered and edge-triggered were the same distinction Ashby draws between a regulator fed by the disturbance and one fed by the error. That the expectations record in the ReplicaSet controller exists, and why. That a snapshot can miss a short-lived thing and a watch cannot, and that this is why the record has to be keyed by version. That a controller's own event log is rate-limited inside the controller. That Ashby's bound is derived for tables without repeated outcomes, and that mine has them. That even the simplest pod exists before it is ready, and that with a five-second readiness delay the gap came out nearer seven, with the rest being scheduling and startup I had never thought to count.
What I still cannot answer, written on the thirteenth of October. Whether the named-delete asymmetry is really what damps the cycle on a real store. What the real controller does behind a real informer lag, because I never put it behind one. Whether "an observer that knows its own inputs" is Conant and Ashby's theorem in a small form, or only sounds like it. What the chapter 12 regulator looks like in Ashby's own notation once I have read 12/7 onward rather than skimming it. Whether any of my loops at home resync often enough to count as lookers.
When I come back. Try these without looking up:
- Explain to someone why the ReplicaSet controller keeps expectations, using the plate shelf.
- Say why the third regulator keys its memory by version and not by name, and what goes wrong otherwise.
- Open Ashby to 11/5 and say in one sentence why my table is not the kind the bound is derived for.
- Say which half of L1 held, which failed, and what the next registration should test.
- Name the two reasons the storm test could not count the real controller.
- Say whether I have read Conant and Ashby yet, and if so, whether this experiment is an instance.
If I can do the first five, the week was worth it. If I can do the sixth, I have moved.
What this does not show
| Limit | Why it matters |
|---|---|
| the toy's disturbances are invented, and so is how often each kind happens | nothing here describes how a real cluster is actually disturbed |
| loss in the toy is independent per tick and delay is constant | losses that come in bursts and delays that vary from tick to tick, the common real cases, are untested |
| the toy's actions are instant and certain | a failed create is a disturbance the toy cannot produce; the real rungs measured the readiness lag it omits |
| the edge regulator never looks, by design | H3 shows what that design costs; it does not show that an event loop with an occasional look would fail |
| the real controller was never behind the lag at any rung | its own informer delay and its expectations are still unmeasured |
| the lag at the last rung is applied by my script | a stand-in for an informer cache, not one |
| the storm deletes a pod every tick for forty ticks | no cluster suffers that; it exists to make the guard countable, and it overran the controller's event budget |
| the reference count relied on events, and the fallback count cannot tell zero from one | the better count is registered for next time |
| one run, one seed, one machine; two thousand ticks a cell in the toy, three hundred on the real cluster | with three hundred ticks, an in-band fraction could easily read a few hundredths higher or lower on another run; nothing in this post leans on a difference smaller than that |
| resource versions are compared as whole numbers in order | the Kubernetes documentation guarantees this for everything the API server itself serves; add-on servers that extend the API may not, and this cluster has none |
Why share it
I said this out loud last week, more or less, and I am keeping it as I said it:
Building agentically, with a cluster as the backbone for the infrastructure, has been amazing and so much fun. It is also the perfect lab for what I am trying to do with many things, and it has connected me to cybernetics, which is beyond exciting. What a happy marriage of tech and theory.
That is a mood. This post is what the mood survives contact with. A toy that was wrong twice, a law that did not apply, a real run that needed three attempts and a measure that turned out to be throttled. I am sharing it because the method is the thing worth having, and the method only shows when the results are modest. Write the question down first. Write down what would prove you wrong. Build the smallest world that can answer. Keep the void runs. Write the gaps next to the numbers. Let the real system have the last word. Anyone who keeps a loop alive can do that in an afternoon on a laptop, for nothing, and the rule that falls out is cheap to adopt and expensive to ignore.
That is also how the things I am making divide the work. Complexence is where I stand and ask what is worth asking. Fidelitron is the instrument: it builds the smallest world that can answer, runs it, and reports the answer with its limits attached. The cluster is the ground the world is built on. The question came down from the reading; the evidence went back up with its gaps listed; and the evidence sent me to a chapter I had been reading too fast and a paper I would otherwise have skimmed.
Three times this month the toy told me something true about Kubernetes that Kubernetes already knew. Each time I learned it by being wrong in a world small enough to find the seam. The Greek word under both of these, the field and the software, is the one for a steersman. I did not choose that. I keep finding it under the next thing I pick up.
Where this leads on this site
| If you want | Go to |
|---|---|
| what a loop is, and why a path that closes is the special case | A Loop Is a Path That Closes |
| the lab this ran on, and why fidelity is a profile, not a dial | The Smallest World That Still Tells the Truth |
| the reading, in the order I committed to, with my guesses on record | A Cybernetics Reading List, Written Before I Read It |
| gentler ways into the same material | Gentler On-Ramps to Cybernetics |
| the frame the question came from | Complexence |
| the guide that says "level driven, not edge driven" | Writing Controllers and Controllers |
| the expectations record, in the controller's own source | controller_utils.go |
| the event recorder's limits | events_cache.go |
| what the API promises about resource versions | Kubernetes API concepts, resource versions |
| Ashby's book, free; the law is 11/7, the thermostat is 12/4 | An Introduction to Cybernetics |
| the theorem I have not read yet | Conant and Ashby, "Every good regulator of a system must be a model of that system," International Journal of Systems Science, 1970 |