One Writer Per Artifact

There is a directory in my automation that took me longer to understand than any code I have ever written, and it contained no code. It was a data directory, the one my dashboard reads from, and once I finally saw it the reason it resisted understanding was simple. More than a dozen different jobs wrote into it. To answer the smallest possible question about that folder, what is in here and can I trust it, I had to hold every one of those writers in my head at once, know their schedules, and reason about who wrote last and what they overwrote. One folder, a dozen-plus owners, which is to say no owner at all. That directory was the soup at the center of my system, and it was soup because it broke a rule I did not know existed.

The rule is that every artifact gets exactly one writer. One file, one job responsible for it, and no exceptions you did not choose on purpose.

Rules that come free, and the one that does not

I found the rule while typing my loops, and the typing is what makes its absence visible. Most of the rules a pipeline needs turn out to ride along with the type, free, the way grammar rides along with a part of speech. An emit has to be safe to run twice, a report has no side effects, a watch must never write, and the essay on the types has the rest. I did not compose any of that. The types carry it, and every future loop I write will inherit its contract at the moment of naming.

Single writer per artifact is different, and the difference is why it is the rule that actually gets broken. It is a fact about the space between loops. Nothing in a transform's own contract prevents a second transform, written eight months later on a busy day, from writing to the same file. Each loop on its own is correct and the collision lives in the pair, and I was never standing where the pair was visible. That is how my soup directory happened. A dozen individually fine decisions, each one adding a writer, and the fog built up a decision at a time with nowhere to stand to see it.

Chosen twice, or drifted into

Two writers can happen. I have one deliberate violation, a job that seeds a file and a second job that fills in the rest, colliding by design, and the design is written down where the next reader will find it. That violation is fine, and the reason it is fine is what the whole rule turns on. I chose it and I wrote it down, so I can see it, I know what it costs, and I can revisit it. A violation I drifted into is fog with a commit history. So the rule as I actually hold it is never two writers by accident. The constraints in a system I build alone are there to stop the accumulation of reasonable Tuesdays, since there is no rogue teammate to stop.

I did not invent the rule. In data engineering it is close to doctrine, and I wrote separately about where I found it, in Dagster's assets, and why I took the idea without the tool.

Rules you remember, rules the structure carries

A rule I have to remember is a rule I will eventually break, and not through carelessness. I hold a rule in attention a hundred times and lapse once, and the system keeps the lapse. Every rule stored in my memory decays at the rate of my worst day. This is the same reason I build maps outside my head instead of trusting recall, and it took me embarrassingly long to see that it applies to laws as much as to maps.

When the contract rides on the type, and the type is declared in a registry, and a check reads the registry, the rule outlives my attention to it. The next loop I add, months from now, in a hurry, gets named, and the name hands it its obligations, and a violation shows up as a red line in a report instead of as a mystery in a folder. The system has begun to govern loops that do not exist yet. A pile is ruled by the memory of whoever built it and degrades as that memory does, and this one has started to carry its own law.

Looking back it was one move, run three times. Typing the loops made each one legible. Drawing the graph made the whole visible. Writing the contracts into the structure made it hold when I get busy, which is the only time legibility is ever really tested. It took names, a drawing, and a place outside my head for the rules to live, and no rewrite. The territory is the same as it was. There is a chart of it now, and the chart does the checking.