Same Score, Half the Time

I failed another practice exam today.

Forty out of sixty-five. That's 61 percent, and the real exam wants roughly 72. My whole record is three timed sets: a 55 on May 10, my first time out; a 61 on May 24, on a different set; and today's 61, a retake of the same set that gave me the 55. Today ties my best score, and my best score is still a fail. The ceiling has read 61 twice now. I booked the actual test for the morning of July 31, twelve days from now, so the timing wasn't lost on me.

But the score doesn't show the clock. The set gives you two hours and ten minutes. I used one hour and six. Last time I needed nearly all of it and came out with my head buzzing. This time I moved through the questions the way you move through a neighborhood you've started to live in, and when I hit submit I wasn't tired.

Same number, half the time, none of the strain.

The number moves last

I run, slowly, a couple of miles at lunch when I can. Anyone who runs knows the thing where your pace doesn't improve for weeks and then you notice your heart isn't pounding at that pace anymore. The stopwatch says nothing changed. Your body knows everything changed. The stopwatch is a lagging indicator. The heart rate is a leading one. Fitness shows up in effort first and in time second.

I think learning works the same way, and I think this is easy to forget when a deadline is staring at you. The score is the stopwatch. What moved today was the effort: questions that used to require me to reconstruct half of Amazon's cloud from first principles now read like sentences I could just answer. The recall got fast and cheap. The subscores moved too: the cost domain, a flat zero on that first May set, came in at a third today. What's left isn't a fog anymore, it's a list. Twenty-five specific things I didn't know.

There's a mechanical reason the number sits still, too, and it helped to spell it out for myself tonight. A score is a threshold laid over the whole exam. Every question I almost know still grades as a zero, and the almosts show up nowhere on the report. Understanding can climb for weeks while the number holds flat, because the number only moves when enough almosts flip all the way over to yes. The learning is continuous and the score is a step function, and a step function spends most of its life looking flat.

And a list of specific things you don't know is the single best input a study system can receive.

The misses become cards

The system I study with is built around exactly this. Every miss from today went through a small pipeline and came out the other side as flashcards, fourteen new ones, in a spaced repetition deck. Spaced repetition is the flashcard method that shows you a card again right before you would have forgotten it, so every review arrives right where it does the most good. The practice exam's job is not to grade me. Its job is to find the gaps and feed them to the deck, and the fourteen newest cards in it are today's twenty-five misses, distilled.

Tonight, before bed, I ran it: seventy-five cards, forty-nine right, 65 percent. Nearly the exam's number again, which makes sense, since they're measuring the same holes. The difference is what happens next. The exam prints the score and walks away. The deck takes each card I missed and schedules it to come back right before I'd forget it again.

So the loop is closed. Read, test, harvest the misses, drill the cards, test again. Seven practice sets left in the bank, twelve days, one set every other day. The plan doesn't change because a number stayed flat. The plan was never about the number.

Writing the pressure down

The tie stung, and twelve days doesn't feel like plenty. I wrote the pressure down in my journal tonight the same way I'm writing it here, because pressure that stays in your head steers the car, and pressure that gets written down just rides along.

Pass or fail, I'll report back on August 1. My money says the score moves last, and then all at once.

Update, August 31: the report back

I said I would report back on August 1. This is thirty days late, and the delay is the first half of the report.

A week after I wrote this, the timed set I had set a gate against came back 34 out of 65. Fifty-two percent, my lowest score of the whole study and my slowest time, five days before the exam. So I moved the exam out a month rather than taper into a date I had already outgrown. The reasoning got written down the same week, in the decan where the harvest date moved.

The review of that bad set found the thing this post missed. I had been reading a flat surface score as a plateau, and it was not a plateau. I was drilling one exam domain at a time, and each domain rotted the moment my attention left it. Security climbed to 76 percent while I worked it and fell to 61 as soon as I moved on. The domain I had never once focused on came in at 41 and pulled the whole number down. The score was flat because I kept trading one gain for one loss, which looks exactly like a ceiling from the outside.

So the leading indicator I was so pleased about in this post was real but incomplete. Effort dropping is a genuine signal. It does not tell you that the ground you covered last week is quietly going.

The month went into interleaving all four domains daily, and into reps rather than review: a working model of the exam's world for nine cents an hour that I could break on purpose, and the ten decision regions cut into audio for the car. The last valid simulation, on August 16, scored 70.77 percent, one question below the practice bank's pass line, with 47 seconds left on a 130 minute clock.

This morning I sat the real exam and scored 799, against a line of 720 and a ceiling of 1000.

The bet in this post was that the score moves last and moves all at once. It did move that way. I would just make the claim smaller now: the number was flat because of a rotation problem I could not see, and the fluency I could feel was real in whichever domain I had touched most recently. The full write-up of the study system, and what it got wrong, is in The Mind Places Things on Paper.