Say you open a coffee shop.
You have done the arithmetic, and the arithmetic is beautiful. $0.56 of espresso, $0.31 of milk, $0.22 of cup and lid and sleeve. $1.09 of goods in a latte you are going to sell for $6.
Almost $5 of margin on every cup. 200 cups a day is $982, every day, before you have even opened on Saturday. Call it $29,000 a month.
You sign the lease.
Six months later there is nothing in the bank.
Sit with that before you read on. The arithmetic was not wrong. Nobody stole from you. 200 cups a day is about what you are doing.
So what do you do?
Most people reach for the obvious lever. $5 of margin is not enough, so make it more. Raise prices 10%, $6 becomes $6.60, and that is another $132 a day for the cost of reprinting a menu.
Play it forward, though. Imagine you are the customer here. You buy a latte on the way to work, and one morning it costs $0.60 more. Most days you still buy it. But some days you order a drip coffee instead to save a few dollars, and some days you walk the two blocks to the place that has not raised its latte price. Nobody decides to stop coming. They just come slightly less.
At the higher price, say the daily count falls to 170 cups.
Here is the part that stings. Your rent did not go down. Your baristas are on the clock for the same 12 hours. The costs that used to be spread across 200 cups are now spread across 170, so each remaining cup has to carry more of them. You gained $0.60 of price and handed most of it back.
Which means the lever was never the problem. Where is the money actually going?
Two baristas at $20 an hour, 12 hours a day, is $480 before you sell anything. Rent on a decent corner in Jersey City is $7,000 a month, call it $230 a day. Spread across 200 cups, that is $2.40 of labor and $1.15 of rent in every latte.
Against $1.09 of coffee.
Read that again, and ask what those two numbers depend on. Not how much coffee is in the cup. Labor depends on how long someone stands at a machine. Rent depends on how many square feet that machine sits in. Pour a large instead of a small and neither one moves. Only a few cents of milk moves.
The beans are the only input that tracks volume, and the beans are the smallest line on the list.
So the thing you are selling, it turns out, is a minute of someone’s attention and a square foot of somewhere to stand. The coffee is what gets handed to you at the end of it.
And if that is true, it predicts something you can check without opening anything. A large should cost barely more than a small, because upsizing adds milk and nothing else. And the shop should push the large every time, because everything above that milk is nearly free.
Go look at a menu. Past the base price of the drink, the jump from small to medium to large is usually a matter of cents, not dollars.
There’s a second way to check it, and it doesn’t share a single assumption with the first. Forget the inputs entirely. Take six months of bank statements, subtract everything that went out from everything that came in, and divide by the number of cups you sold. That number doesn’t care what you think a barista costs or how you estimated volume. It’s just what happened.
If it lands near what you worked out from the components, you’re on to something. If it doesn’t, something in the way up was wrong, and the size of the gap tells you roughly where to look.
Your first idea was never wrong about the beans. It was wrong about what the business was.
You made the same move last week
That was a coffee shop, and you don’t run a coffee shop. But look at what actually went wrong, because it wasn’t a math error.
The first approach took a familiar frame, cost of goods plus a markup, and applied it to a situation that didn’t fit it. When the numbers came back wrong, the instinct was to pull the one lever that frame offers. Raise the markup. The frame was never examined, only operated.
You did a version of this recently. A service got slow, so you added instances. Latency was better for a week and then it wasn’t. Or a query was slow, so someone added an index, and the plan didn’t change. Or a deploy kept failing in one environment, so it got a retry.
None of those are stupid. Each one is the obvious lever on a frame nobody said out loud.
A quick piece of vocabulary, since the rest of this leans on it. A model is the story you’re carrying about how a thing works. Not a diagram, not a document. The thing in your head that tells you which lever to reach for. Cost of goods plus markup was a model. So is “the service is slow because it needs more instances.”
Everyone has one for everything. Most of them were absorbed rather than built, and the absorbed ones are the ones that break quietly, because you never watched the assumptions go in.
So where do the absorbed ones come from?
Mostly, from learning products
You learned Kafka because a service you owned used Kafka. You learned it the way anyone learns a thing they need on Tuesday. The topic was created with 12 partitions because that’s what the other topics had. Consumer lag became a real concept the morning the lag alert fired. You found out about consumer groups and rebalancing because a deploy made every consumer drop and repick its partitions at once, and you found out about max.poll.interval.ms because that’s what someone told you to raise so it would stop. You know acks=all is the safe setting. You’d have to look up what it’s safe against.
That’s a real model and it works, right up until something happens the runbook didn’t cover.
It also has a shelf life, and this is the part worth being selfish about. Whatever you know about Kafka today is worth less next year. Not because you’ll forget it, but because the thing moves and eventually gets replaced. Every specific technology you’ve ever learned is on that slope.
What doesn’t depreciate is the layer underneath.
Kafka, Cassandra, DynamoDB, and Flink look like four unrelated products. Decompose them and they’re four arrangements of a short list: split the data by a key so work can spread across machines, replicate it so a lost machine isn’t a lost record, decide what ordering and consistency you promise and what you refuse to promise, and work out what happens when one shard gets hot or a node disappears mid-flight.
That list has barely changed in twenty years. It’ll outlive all four products.
Which means the two ways of learning Kafka have wildly different half-lives. Learn it as a product and you’ve learned Kafka. Learn it as one arrangement of those primitives and the fifth distributed system you meet takes a week instead of a quarter, because you already know what questions it has to answer. You’re just looking up which answers it picked.
It’s also what people are pointing at when they say a senior engineer has good instincts. It looks like fast pattern matching, and it is, but it isn’t the kind a junior engineer does when they copy a pattern from another service because the situation looks similar. Both feel like recognition from the inside. Only one of them survives a case where the surface matches and the mechanics don’t.
The difference is whether a decomposition happened first. The patterns worth having are what’s left over from one.
Everything that sank the shop was on the second list
Go back to the moment before the lease got signed, and split what was in your head into two piles.
Things you knew:
Beans cost $14 a pound. A double shot is two of the twenty-five you get out of it. Milk is $5 a gallon and a latte takes eight ounces. Rent on the corner is $7,000 a month. Baristas cost $20 an hour and you need two of them for twelve hours.
Things you assumed:
That the cost of a drink lives in the cost of what goes into it. That price is the lever that moves profit. That 200 cups a day stays 200 cups a day when the price changes.
Look at those two piles again. From the inside, on the day you signed, they felt identical. Every line in both was something you’d have said with the same confidence if someone asked. That’s the whole problem. Nothing marks an assumption as an assumption while you’re standing in it.
And every single thing that sank the shop is in the second pile.
The first pile is fine. The beans really do cost $14. Those numbers were never the issue, which is why checking them harder would never have helped, and why “I ran the numbers again” is the most comforting useless thing you can do.
So the move is to take the pile that nobody checks and push on it:
1) Split what you know from what you assume. Two lists, written down, actually separate. The test for which pile a thing goes in is whether you could show someone. A number you pulled this morning is knowledge. The same number remembered from six months ago is an assumption wearing knowledge’s clothes.
2) Ask what each assumption is sitting on, until you hit something that doesn’t move. Cost of goods was sitting on the idea that a drink is its ingredients. Underneath that: a minute of someone’s time, a square foot of floor, a payment due whether anyone walks in or not. Those don’t go any further. You’ve hit the floor when the answer stops being about the thing you sell and starts being about what you’re paying for.
3) Make it predict something specific. If the cost lives in the transaction and not the ounces, a large should cost barely more than a small. Direction and rough size. Specific enough to be wrong.
4) Check it, and then check it from somewhere else. Go look at a menu. Then find a second route to the same answer that doesn’t share assumptions with the first, because a prediction can come out right for a reason you never thought of.
Step four is where this stops being a story you tell yourself.
The dashboard already had a hit rate
We wanted to know whether the cache was earning its place.
There’s a metric for that, sitting right on the dashboard. We couldn’t use it.
CacheHitRate is computed as hits over hits plus misses, and a miss is any read-only lookup that doesn’t find a key. Our caching feature was behind a flag, on for a subset of users, but every request into the service attempted a lookup regardless of eligibility. So each ineligible request landed in the denominator as a miss. The number was real. It just wasn’t an answer to our question. It was measuring the flag rollout.
That’s the kind of assumption that doesn’t announce itself. A hit rate metric is supposed to tell you whether your cache is working. Somebody computed it from a premise about how caches get used, that premise held for most people, and it didn’t hold for us.
So we went at it directly. Count what we put in, count what we got out. SetTypeCmds against the hits.
That’s where it fell apart, because our reads are destructive. The service reads an entry and immediately deletes it. And SetTypeCmds isn’t a count of writes, it’s a count of every mutative command, everything that changes data rather than just looking at it. DEL qualifies. So a single cached item that gets used increments the set counter twice: once when we write it, once when the read removes it.
The counter we’d picked as our denominator was being driven by the very thing we were trying to measure against it.
We did the arithmetic anyway, and it didn’t close. Not by a rounding error, by enough that something was going on in that cache we didn’t have a model for. Valkey had gone in months earlier and the rollout stretched well past what anyone planned, so my own hands-on time with it was measured in days. It would have been strange if there weren’t gaps.
Here’s the move that mattered. We didn’t keep pushing on the layer we couldn’t see. We went and found a layer we could.
Every one of these events also passes through our own application. The prefetch endpoint that writes to the cache logs when it’s called. The read path logs a cache hit when it finds something. Both land in Splunk, both are instrumented by us, and neither one needs us to understand a single counter Valkey emits.
Prefetch calls against cache hit lines came out to 71%.
Then we backed up and asked what can actually happen to one of our cache entries.
Something hits the API. The prefetch populates the cache. After that there are exactly two exits. Either a request comes in and reads it, or nobody asks and the TTL runs out. We don’t invalidate, we don’t overwrite, and we’re nowhere near memory pressure, so eviction isn’t in play. Two doors, no third.
That’s a model we could count against, because both doors have their own counter. Reclaimed is the number of keys that expired. Hits are the number that got read. Everything we wrote left through one of them, so the hit rate is just hits over hits plus expirations. No set counter, no lookups from ineligible users, none of the machinery we couldn’t account for.
That arithmetic landed at the same place the Splunk count did.
Which is the part that made it trustworthy. One number came from our own application logging its own behavior, the other from two cache counters and a claim about the only two things an entry can do. They share almost nothing. The Splunk count doesn’t know the cache exists as anything but an endpoint, and the cache count doesn’t know what a prefetch is.
We never did untangle what the set counter was made of, and as far as I know it’s still open. It stopped mattering once two paths that share no assumptions arrived at the same answer.
One measurement can be right by accident. It’s much harder for two to be wrong in the same direction.
And notice what we still don’t know. There are Valkey internals I couldn’t walk you through today. None of that was blocking anything, because the question was never “how does this cache work in full.” It was “is the cache doing its job for the users who have it turned on.” Those are different questions, and only one of them was in the way.
Knowing which one you’re answering is most of the skill.
Nobody schedules the day an abstraction leaks
That Valkey metric was an abstraction. Somebody built a number that means “is your cache working,” and to build it they had to assume something about how caches get used. The assumption was reasonable. It just wasn’t ours.
This is not a Valkey problem. Joel Spolsky named it in 2002: all non-trivial abstractions, to some degree, are leaky. Every layer you work on top of is hiding something, and eventually the thing it’s hiding shows through. The consequence he draws is the one that matters here. Abstractions save you time working, but they don’t save you time learning.
You can’t pick the day that happens. It arrives during an incident, in the layer you never looked at, while people are waiting.
Which is the honest answer to why any of this is worth the time. Not that decomposition makes you a better thinker in the abstract. It’s that the understanding has to already exist when the leak shows up, because there’s no time to build it then.
It applies above your code as well as below it. A requirement is an abstraction over a business need, built by someone else from assumptions you can’t see. “The user needs X” is a compiled artifact. It leaks the same way CacheHitRate did.
A second source, not a shortcut
Worth mentioning what else was in the room during that cache work.
One of the corroborating numbers came from a tool a teammate built with Claude, a small matcher that tied two disjoint requests together in the logs so we could count something the existing instrumentation didn’t track. It worked. I don’t think either of us could have written it that fast.
But notice what made it usable. It wasn’t that the tool was well written or that the model sounded certain. It was that its output landed where two other counts had already landed, by routes that had nothing to do with it. That’s the same convergence test, pointed at a source we had the least reason to trust on its own.
This is the part to be careful about, because the failure mode here is the one this whole piece is about. An AI answer arrives with reasoning attached. It is fluent, it cites mechanisms, and it feels nothing like copying a pattern from another service. It is much easier to accept than a coworker saying “just set it to 4,” and for exactly that reason it’s easier to accept without checking.
So calibrate. An AI answer is a claim from somewhere, same as a metric on a dashboard or a sentence in a doc. It goes on the assumptions list, not the knowledge list. Cheap to produce, cheap to check, and worth checking precisely because it’s cheap. What you cannot hand off is knowing which layer your question lives at and what would tell you the answer is wrong.
Use it to get to a second number faster. Don’t use it to skip having a first one.
This has a name, and it’s older than software
What you’ve watched twice now is first principles thinking. Strip a problem down to the things you can actually establish, then build back up from there instead of from the frame you inherited.
The idea goes back to Aristotle, who was after the propositions underneath a thing that couldn’t be derived from anything else.
The word for the taking-apart is decomposition, and it’s what step two is. Not summarizing the problem, not restating it more carefully. Going underneath each claim to the thing holding it up, then underneath that.
Which raises the obvious question: for how long?
The coffee shop went three levels. A latte costs $6. Underneath that, the cost of a latte is what goes into it. Underneath that, no, the cost of a latte is a minute of labor and a square foot of floor. Then it stopped.
It could have kept going. Why is rent $7,000? Because of the commercial real estate market in Hudson County. Why is that? And so on, for as long as you like, straight down to ones and zeros in every direction. Nothing in the method tells you to stop.
Two things do. The first is the floor: you’ve gone far enough when the next layer down stops changing your answer. Hudson County real estate is real, but you can’t act on it, and knowing it wouldn’t have changed a single thing about what to do with the shop. Labor per transaction was actionable. That’s where it ends.
The second is the question you started with. We never untangled the set counter, and it’s still open. That’s not a decomposition left half done. It’s a layer that turned out to sit outside the question, which was whether the cache was working for the users who had it on. A different question would have pushed further down, and the same stopping point would have been the wrong one.
So you’re not decomposing until you hit bedrock. You’re decomposing until you hit the layer where your question lives, which is usually two or three down rather than ten.
The five whys is the same instinct with a fixed count attached, and five is a reasonable default for the same reason it’s a bad rule. It’s roughly how deep you tend to need to go, and it will happily march you past the layer you wanted.
Now, the part I’d add to the usual account.
Nearly every treatment of this stops at the decomposition. Break it down, reason back up, done. That’s the method as it’s normally taught, and it has a hole in it.
You can decompose carefully, reason impeccably, and be completely wrong, because one of the things you took for bedrock wasn’t. SetTypeCmds looked like bedrock. It was a counter with two different kinds of event in it. Careful reasoning on top of that produces a confident wrong answer faster than sloppy reasoning does.
That’s why steps three and four exist. Prediction is what makes a decomposition capable of being wrong. Convergence from a second, independent direction is what catches the case where it came out right by luck. Without them you haven’t reasoned from first principles. You’ve been thorough about your assumptions, which is not the same thing.
Before or after, not whether
The usual advice at this point is to save this for the big decisions and pattern match the rest, because decomposition is expensive and you can’t afford it everywhere.
I don’t think that’s right.
Start with the cost, because it gets misread. Decomposition is not a fee you pay per decision. It leaves something behind. The afternoon spent working out what a partition key actually does is an afternoon you spend once, and then you have it for Cassandra and DynamoDB and whatever replaces them. Priced as overhead on a single decision it looks expensive. Priced against everything it pays for afterward, it’s one of the cheapest things you do.
Then there’s what each approach leaves you holding.
Pattern matching hands you an answer and nothing else. It worked before, so do it again. When the situation quietly changes and the answer stops being right, nothing tells you. The pattern still matches. It just doesn’t work anymore, and you find out from an incident.
Decomposition hands you the answer plus the conditions it depends on. That second part is the whole difference. Two years later, when someone proposes doubling the partition count, you know whether your understanding still applies, because you know what it was resting on. You can check whether the ground moved. There is no version of pattern matching that gives you that.
So the question was never whether to decompose. It’s when.
Sometimes the clock genuinely wins. It’s 2am, the graph is going the wrong way, and a decision is needed in ninety seconds. Take the heuristic. Roll back, add the instance, flip the flag. That’s the right call, and nothing here says otherwise.
Then decompose it in the postmortem, while the evidence is warm and nobody’s waiting. You get the same understanding. All that changed is whether it arrived before the decision or after it.
What doesn’t work is skipping it and calling that a choice. The decision you skipped it on comes back. Usually in a different system, wearing different words, and by then nobody remembers it’s the same problem.
Start with one thing
Pick one thing you use every day and don’t actually understand. The metric you trust without knowing how it’s computed. The config value someone told you to set. The pattern you copy from service to service because it’s what the other services do.
Write down what you know about it and what you assume. Then go one layer underneath the assumptions and see what’s holding them up.
You’ll probably find it’s fine. Once in a while you’ll find a counter with two different things in it.
You can borrow an answer. You cannot borrow the reason.