The Smallest World That Still Tells the Truth

Every development environment I have ever used lied to me in one of two directions.

The first lie is the clone. Reproduce production faithfully and you get something slow to provision, expensive to run, fragile to maintain, and shared with everyone else who needs it. It is accurate in principle and unavailable in practice. People stop using it, or they use it so carefully that it never gets stressed, which is the same as not having it.

The second lie is the mock. Replace the hard parts with stubs and you get something instant, cheap, and confidently wrong. The tests pass. The thing that broke in production was living inside one of the parts you stubbed, and it broke in a way the stub had no vocabulary to express.

Both failures come from the same bad assumption: that fidelity is one dial running from fake to real, and the whole argument is about where to set it.

It is not a dial. It is a profile.

Fidelity is a profile, not a dial

The same system needs high deployment fidelity and near zero data volume when the question is "does this release install correctly." It needs high data fidelity and near zero deployment fidelity when the question is "does this query survive the real distribution." An environment tuned for both is tuned for neither, and costs more than either.

So stop building environments and start building answers. For a specific question, construct the smallest runnable world that preserves the behaviors that question depends on, and collapse everything else on purpose.

I have been calling this System Fidelity Engineering, and the rule at the center of it is:

Build the smallest runnable world that preserves the truths required by the experiment.

Every component gets exactly one of four representations.

Representation	Meaning	Choose when
REAL	the actual technology, running	its behavior is the question, or faking it convincingly costs more than running it
SUBSTITUTE	something cheaper with the same semantics	the contract matters, the implementation does not
SIMULATE	only the relevant behavior or contract	you need the shape of the interaction, not the thing
OMIT	absent	it cannot affect the answer

Here is the part that does the actual work. A dimension marked low is an optimization when it is chosen and written down. That identical dimension, at that identical value, is a defect when it is an accident nobody noticed. The only difference between the two is whether it is recorded.

Everything else in this post is technique. That distinction is the method.

So I built one

Talk is cheap, so here is a working example I stood up in a day. I have written before about AWS being math and Kubernetes being physics, two ends of one spectrum. This is what it looks like to wire those two ends together on purpose and make them talk. The question was deliberately small, because the substrate had to exist before anything interesting could run on it:

Can a real Windows runtime behind a real AWS load balancer exchange traffic with a Kubernetes cluster on my laptop, in both directions, cheaply enough to leave running while I work?

That sentence has some jargon in it, so before the diagram, the short version of the vocabulary. AWS is Amazon Web Services, Amazon's cloud. A VPC is a Virtual Private Cloud, which is your own private slice of network inside it, carved into subnets, which are smaller address ranges with their own routing rules. A CIDR is just a way of writing an address range, so 10.42.0.0/16 means the roughly sixty-five thousand addresses starting at 10.42.0.0. NAT is Network Address Translation, the trick that lets machines with private addresses reach the internet by borrowing a public one on the way out. Everything else I will define as it shows up.

The shape:

Every address in that diagram is a private one. The whole thing fits on a laptop and a cloud account you already have.

A few decisions in there matter more than they look.

The Windows host has no public IP address at all. It lives in a private subnet. You reach it three ways: over the encrypted overlay network, through the internal load balancer, or through AWS Systems Manager Session Manager, which works by having the instance dial out rather than by opening anything inbound.

The load balancer is internal. No public DNS record resolves to it, meaning no name on the public internet points at it. It exists to be genuinely in the path, not to be reachable from the internet.

Total inbound exposure to the internet across the whole lab: one UDP port, carrying WireGuard, where every packet is authenticated against a device key before anything else looks at it.

The overlay runs inside the cluster, not on the laptop. This one surprised me. With minikube on the Docker driver, the node network lives inside the Docker Desktop virtual machine, and macOS has no route to it. A subnet router running on the Mac has nothing useful to advertise. The router has to run as a pod, on the correct side of that boundary, where it can see the cluster networks natively.

It works in both directions. A pod on my laptop pulls a page off the Windows host through the internal load balancer in 250 milliseconds. The Windows host fetches a Kubernetes Service back on my laptop, and the response names the exact pod that served it. I can open Remote Desktop and look at the Windows desktop, delivered through an internal load balancer, from a machine with no public address.

The nine cents

Running cost is about $0.087 an hour. A three hour session is roughly a quarter. Idle it is zero, because teardown deletes the load balancer and terminates both instances while keeping the VPC, subnets, security groups, and the IAM role (Identity and Access Management, the permissions system that says which machine is allowed to do what), all of which are free to leave standing.

The interesting line item is the one that is missing. A managed NAT Gateway costs about $32 a month, standing, whether you use it or not. It is frequently the single largest bill in a small AWS account.

I replaced it with a NAT instance: a t4g.nano running Amazon Linux, doing IP forwarding and masquerade with a handful of iptables rules, at roughly $3 a month, and only while the lab exists. The same box also runs the overlay subnet router, so one instance is both the network's way out and its way in.

That trade is not free, and being clear about the cost is part of the method. A NAT Gateway is managed, redundant within its availability zone, and scales without you. A NAT instance is a single point of failure that you patch yourself. For a lab that exists for three hours at a time, that trade is obviously correct. For production it usually is not. Knowing which question you are answering is what makes it a decision instead of a shortcut.

There is one setting that makes the whole thing work, and if you have studied for an AWS certification you already know it:

aws ec2 modify-instance-attribute --instance-id i-xxxx --no-source-dest-check

An EC2 instance (Elastic Compute Cloud, which is what AWS calls a virtual machine) drops packets that are neither from it nor to it. That check exists to stop spoofing. A NAT instance's entire job is forwarding packets that are neither from it nor to it. Leave the check on and you have a beautifully configured router that moves nothing.

What broke, which is the useful part

I kept a log. Fifteen entries by the end of the day, and every one produced a system that looked correctly configured and moved zero packets. A few worth passing on.

Do not put your router on spot. My first t4g.nano was a spot instance. AWS reclaimed it four minutes after launch, no capacity in that availability zone. That box was simultaneously the private subnet's default route and the overlay path, so when it died, everything died. Spot was saving me $0.003 an hour for the privilege of a single point of failure that can vanish with two minutes of warning. It runs on demand now.

The useful part is that I did not have to simulate that failure or schedule it. The cloud did it to me, unprompted, in the first four minutes, and the lab got better because of it. That is the gain from disorder argument in its smallest possible form.

Order matters with iptables, twice. Amazon Linux 2023 ships a FORWARD chain that ends in a blanket REJECT. I appended my ACCEPT rules, which put them below the reject, where they were never evaluated. Use -I FORWARD 1 to insert above it, not -A to append below. Then I made it worse by running systemctl enable --now iptables after adding rules, which restored the packaged ruleset over the top and silently erased everything I had just done. Start the service, then add rules, then save.

set -x will print your secrets. My bootstrap script ran with tracing on, so the line tailscale up --authkey=<the actual key> was echoed in full into /var/log/cloud-init-output.log. That key had to be revoked and regenerated. Secret handling now sits between set +x and set -x, and the key travels from a password store into AWS Parameter Store as an encrypted string, read at boot by the instance's own IAM role. It never touches userdata, a command line, or a file.

The AWS command line tool is not installed on the EC2 Windows AMI. An AMI is an Amazon Machine Image, the disk template a new instance boots from. Only the AWSPowerShell module. My script called aws.exe ssm put-parameter, which failed quietly, and the administrator password never got stored. Use Write-SSMParameter.

A network load balancer will not accept a TCP listener and a UDP listener on the same port. The create fails and leaves you with a target group sitting in unused. Use the TCP_UDP protocol when you want a port both ways.

A network load balancer does not answer ping. One hundred percent packet loss to the load balancer while the host behind it answers in 0.38 milliseconds. Nothing is wrong. Knowing that in advance saves you an evening.

The measurement I got wrong twice

I wanted to know what putting a load balancer in the path actually costs. First run said 40 percent. That was wrong, for two separate reasons, and the correction is more useful than the number.

A burstable instance is a bad measuring stick. A t4g.nano bursts to 5 Gbit/s but its sustained baseline is around 32 Mbit/s. Running back to back ten second tests drains the credit, and the decline gets blamed on whatever you happened to be testing.

A cold load balancer measures its own cold start. When I interleaved the two paths with rest between them, the direct path stayed flat between 4.2 and 4.8 Gbit/s across four rounds, while the load balanced path climbed steadily: 2759, then 3637, then 4164, then 4210 Mbit/s. It was not paying a fixed tax. It was scaling up, which is what these things do when traffic arrives.

Warm, the real cost of the hop is 3 to 6 percent.

Then I tried the same test from my laptop across the overlay, and both paths pinned to the same 45 Mbit/s, because that is my home internet connection and not anything AWS is doing. A 5 percent effect is invisible underneath a bottleneck a hundred times larger.

The general lesson, which cost me an hour and I would like to save you: your vantage point has to be faster than the thing you are measuring. Otherwise you are characterizing your own instrument and writing it down as a finding.

What it cannot do

This section is the one that makes the rest of the post trustworthy, so here it is in full.

Everything is in one availability zone, and the router is a single point of failure for both routing and connectivity. That makes this a high fidelity failure injection rig and a low fidelity failover rig. I can break things realistically. Nothing recovers, because there is nothing to recover to. Any high availability conclusion drawn here does not transfer.

There is an MTU cliff. MTU is Maximum Transmission Unit, the largest packet a link will carry in one piece. The VPC interface runs at 9001 bytes, the encrypted overlay at 1280, and pods still believe they are on 1500. Payloads over roughly 1240 bytes do not cross the boundary. This is the most likely way this lab produces a result that is wrong rather than absent.

There is no DNS in either direction. The router cannot resolve cluster names, and pods cannot resolve AWS names. Everything is IP literals. Any application that depends on service discovery is simply not represented here.

Source NAT erases the caller. The cluster sees the router pod's address for all AWS traffic, never the Windows host's. Anything doing IP allowlisting, client affinity, or per caller auditing is unrepresented.

No data layer, no identity, no pipeline. Nothing stateful exists. There is no directory and no realistic authorization. Nothing is attached to a build.

None of that is an apology. Each one is a dimension I collapsed on purpose, and each is written down in the fidelity profile that ships next to the code. That is the whole discipline: a low value you chose is an optimization, and a low value you did not notice is a defect.

The most dangerous world you can build is one that is highly realistic in every dimension you look at and quietly absent in one you do not. Everything you check behaves correctly, so you trust the parts you never checked.

How this fits with studying for the exam

I am working toward the AWS Solutions Architect Associate exam, and I want to be careful with the claim here, because the obvious version of it is wrong.

This lab did not replace practice tests. Practice tests do a job it cannot. They cover the entire surface of the exam, including the services I would never reach for on my own, and they train recall under time pressure, which is a separate skill from understanding. I still do them, and I would not skip them. I have written about what that grind actually feels like in Same Score, Half the Time.

What the lab does is different, and the two work well together. It takes a small number of topics and makes them permanent. Nearly every networking topic on that exam is in here as something I built and broke rather than something I read: VPC design and address planning, public and private subnets, route tables, internet gateways, NAT instance versus NAT gateway including the source and destination check, security groups, network load balancers, target groups and health checks, spot instance lifecycle and interruption, Systems Manager as a management plane that needs no inbound access, IAM instance profiles, and Parameter Store for secrets.

I did not memorize that a NAT instance needs the source and destination check disabled. I spent twenty minutes staring at a router that answered ping perfectly and forwarded nothing, and now I could not forget it if I tried. That is not faster than reading the line in a study guide. It is just durable in a way that reading is not.

There is also a simpler reason I work this way. I like building complex systems, standing them up, and watching them run. Constructing a system and then observing its actual behavior is the version of working with AWS that I genuinely enjoy, and the studying I enjoy is the studying I keep coming back to. A practice test tells me what I do not know yet. A lab I built tells me how the thing behaves when I poke it, and lets me poke it again tomorrow.

Take the idea, not the repository

You will not reproduce my lab exactly, and you should not want to. Take the shape:

1. Write the question down first. One sentence. If you cannot write it, the work is not ready to be modeled. An environment built for "development" has no stopping condition, so it grows until it is a clone.
2. Choose a representation for every component, and be able to defend each one out loud. "We mocked it" is not a representation decision.
3. Write the fidelity profile down as data, next to the code, per dimension, with the known divergences recorded beside the values.
4. Keep real the things that are expensive to fake convincingly. Windows behavior, load balancer behavior, and cloud network routing all fall here. They are cheaper to run than to simulate believably.
5. Collapse everything else deliberately, and make teardown the default. A world that persists becomes a shared environment, and a shared environment becomes production with worse uptime.
6. Say what you did not represent whenever you report a result.

If you want to hand this to a coding agent, that numbered list plus the diagram above is enough of a specification to work from. The pieces are all public, documented, vendor taught technology: a VPC, a NAT instance, an internal network load balancer, a spot instance, a WireGuard overlay, a single node Kubernetes cluster. There is nothing exotic in here. The idea is the part worth taking.

Where this goes

Making a complex system legible is where this starts, not where it stops. A diagram is understanding you can look at. You cannot run a diagram, you cannot inject a failure into a dependency graph, and you cannot ask a picture a question it was not drawn to answer.

I ran into this from the other direction a while back, when I gave every job in my automation a receipt and a green light and the system still would not fit in my head. A dashboard shows the state of the parts. It does not answer a question about the shape of the whole, and neither does a diagram.

What I actually want is the whole loop: observe the real system, model it small, run the model, work it, break it, learn from it, and feed what you learn back as constraints and design decisions on the real thing. Take anything large and complex, shrink it to something you can hold and run, and use it.

Miniaturization is the mechanism. Control is the goal.

Right now the shrinking is done by hand, one careful decision at a time, and automating that is the hard unproven part. But the substrate exists, it works in both directions, it costs nine cents an hour, and it produced fifteen documented discoveries on its first day.

That is a floor, not a building. It is a good floor.