You Reinvented Data Engineering

The best thing anyone told me about my own system this year was that I had not invented it.

For months I had been treating my automation as a private, shapeless thing, a system with no architecture and no best practices because I had made it up as I went. Thirty-two jobs, grown one at a time, each solving whatever that day's problem was, none of them designed against a plan because there was no plan, only a series of Tuesdays. It felt like mine in the way a mess feels like yours. Then I sorted the jobs by what they do to data, seven clean types fell out, and the thing that surfaced next should have been deflating. This is a solved discipline. It has a name, a body of practice, and a shelf of tools, and I had walked the whole road without noticing it was paved.

The field is data engineering. The business of moving data from where it is produced to where it is used, transforming it on the way, keeping the pipes from clogging. That is not a metaphor for what I built. It is what I built, poorly, alone, without the words. Even the shape has a name. When jobs feed each other, output becoming input, and nothing circles back on itself, you have a directed acyclic graph. A DAG. Pull the world in, transform in stages, push the results out, watchers off to the side reporting on the health of the whole. Every serious pipeline on earth is a version of that picture, and so was mine, drawn on no wall, living only in the order my scripts happened to run.

Uncharted usually means unnamed

My instinct the whole time had been that I was in territory with no map. That instinct was wrong in a specific and common way: I mistook unnamed for uncharted. The territory was thoroughly charted. I had simply never looked up the chart, because I did not know my situation was an instance of anything, because the thing you build yourself never announces its own category. It just accretes, Tuesday by Tuesday, until it feels too particular to be anyone else's problem.

Finding the name changed the whole posture of the work. A mess you believe is unique is a mess you defend alone. A pattern with a name comes with lineage, prior art, a vocabulary for the exact failure you are living through, and other people's decade of mistakes already paid for. I did not lose anything when I learned my system was ordinary. I gained the entire field that had been standing behind it, and the fog I had blamed on the system turned out to be, one more time, a fact about words. You cannot navigate by a chart you have refused to look up, even when you are the one who built the territory it describes.

What the tools are actually selling

The shelf of tools came with the name. Airflow, Dagster, dbt, each built to govern graphs of exactly my shape. So the question arrived on schedule: do I adopt one, or keep hand-rolling? There are two easy answers. Professionals use Airflow, so use Airflow. You are one person, keep it simple. Both skip the reasoning, and the reasoning is where the useful part lives.

Every one of these tools is a frozen answer to somebody else's scale, the same way every AWS service is a frozen answer to a distributed-systems trade-off. Read what the tool assumes and you can read the problems its makers had. Airflow assumes hundreds of tasks, teams of operators, backfills over years of history, service-level agreements with consequences. To hold all that it brings a scheduler, a metadata database, a web server, and a fleet of workers. At thirty-two jobs on a personal machine, that is a second system larger than the first, adopted to govern it. dbt assumes the wrong world entirely: SQL transforms over a data warehouse, when mine are files over a repository. Adopting it would mean reshaping the problem to flatter the tool.

Dagster assumes something more interesting, and it earned real study before I set it down. Its unit is not a task but an asset: a specific artifact with exactly one job responsible for producing it, lineage between artifacts tracked for free. That idea is not machinery. It is a way of seeing, and it sees precisely the thing my system got wrong, the directory that a dozen jobs all wrote into so that nobody owned it. So I took the idea and left the installation. Think in artifacts with single owners. Enforce it by hand. If I ever do adopt the tool, the system will already be shaped the way it expects, and the migration will cost nearly nothing. Borrowing the concept and skipping the machinery is not a consolation prize. Most of what a mature tool knows is in its concepts, and the concepts are free.

An engine, or a map

Underneath the tool question there was a sharper question, and answering it is what actually settled the matter. What did I want?

I thought I wanted orchestration. I did not. My loops already run; the operating system's own scheduler fires them and has never been the problem. What I wanted was to see the system. Which loop feeds which. What everything depends on. Where the load-bearing pieces are. That is not an engine. That is a map, and I already held every fact the map needed, sitting in a registry I maintain anyway, waiting to be drawn instead of listed. The drawing took an afternoon. No daemon, no database, nothing new that can break at three in the morning.

That is the oldest rule I trust, the one that has survived every domain I have carried it into: the gains are in subtraction, and the fastest way to make a small system fragile is to bolt a large one onto it for problems it does not have. An orchestrator in my basement would not be sophistication. It would be a new tenant, eating maintenance attention forever, hired for a job that does not exist here yet.

The word that matters is yet, and a yet without a tripwire is just a mood. So I wrote the tripwires down. The day I am doing backfills by hand and hating it. The day I catch myself reinventing retries and alerting instead of configuring them once. The day a second machine or another person needs this on a shared clock. Any one of those, and the answer flips, and the tool earns its weight, and I will adopt it without ceremony because the decision was already made and dated. Until then the engine stays on the shelf.

The map came off the shelf immediately, and what it showed me, the actual graph of my own system, is the next essay: the view that turned thirty-two things I maintained into one system I could finally stand inside and read.