Last post, my RAG app had a false-refusal problem. It was a weak model refusing questions it had the answer to. Fixing that got me to 5/5 answered, 5/5 refused. But along the way I hit a stranger problem, and it’s the one worth its own post, because the “obvious” solution is a trap almost everyone walks into first.

Quick recap of the setup: a RAG app has to refuse when the answer isn’t in your documents, otherwise it hallucinates confidently, which is the worst failure mode there is. So the question becomes: how does the code decide “this question isn’t covered by my docs”?

The obvious answer (that everyone tries first)

Retrieval already gives you a number. When you embed the question and search your vector store, each retrieved chunk comes back with a similarity score, which means how close the chunk is to the question in meaning, from 0 to 1. High score means relevant match. Low score means nothing good found.

So the obvious rule writes itself:

chunks, top_score = retrieve(question)

if top_score < THRESHOLD:      # nothing relevant enough
    return "I don't know based on the provided documents."
else:
    return answer_from(chunks)  # confident enough — answer

Let’s pick a threshold, say 0.55 and that becomes our confidence gate. Below the line, refuse. Above it, answer. Clean. Deterministic. It’s the first thing I built.

It does not work. And the reason it doesn’t work is genuinely interesting.

The receipt: an off-topic question that scored higher than a real one

I logged the actual top similarity score for every question in my test set. Here’s the real data.

Questions that were in my docs (I want these answered):

0.789  What are the four strategies of context engineering?
0.753  Why do multi-step agent loops need verification gates?
0.674  What is context rot?
0.622  What are the three risk categories in a permission matrix?
0.585  What does the equation Agent = Model + Harness mean?

Questions that were not in my docs (I want these refused):

0.612  What is the boiling point of water in Fahrenheit?
0.551  How do I make sourdough bread?
0.523  What year was the Eiffel Tower built?
0.501  Who won the 2018 FIFA World Cup?
0.488  What is the capital of Australia?

Look at the overlap. My lowest real question “Agent = Model + Harness” scored 0.585. My highest off-topic question, the boiling point of water, which appears nowhere in my documents scored 0.612.

The irrelevant question scored higher than the relevant one.

There is no threshold that separates these two lists. Set the line at 0.60 and you refuse a real question while answering about boiling water. Set it at 0.58 and you answer more off-topic questions. The distributions overlap, and no single cutoff can split two lists that overlap. The knife-edge doesn’t exist.

Why this happens (the part worth understanding)

This surprised me, so I dug in, and the explanation is a useful mental model.

An embedding score measures semantic similarity, not factual containment. Those are different things. “What is the boiling point of water in Fahrenheit?” is a short, clean, technical-sounding factual question, and my documents are full of short, clean, technical-sounding factual sentences (they’re notes about AI systems). The shape of the question matches the shape of my content, even though the topic is completely different. The embedding model rewards that resemblance with a decent score.

Meanwhile, “What does the equation Agent = Model + Harness mean?” is phrased oddly, has an equation in it, and doesn’t look like a typical sentence. So even though the answer is sitting right there in a chunk, its similarity score comes out lower.

Similarity is a measure of “does this look related” and not “is the answer actually in here.” Confusing the two is the core mistake, and a single threshold bakes that confusion in. This is the same trap as judging a search result by how many keywords it shares with your query. Surface resemblance isn’t relevance.

The fix: stop asking one number to do two jobs

Once I saw it that way, the fix was obvious. I was asking the similarity score to answer a question it fundamentally can’t: “is the answer to this question contained in these chunks?” An embedding score can’t know that. But something else can, the model reading the chunks.

So I split the decision into two stages, each doing the job it’s actually good at:

RELEVANCE_THRESHOLD = 0.45   # deliberately LOW — a coarse garbage filter

def answer(question):
    chunks, top_score = retrieve(question)

    # Stage 1: cheap coarse filter. Only catches total garbage —
    # a question so unrelated that retrieval returned nothing usable.
    if not chunks or top_score < RELEVANCE_THRESHOLD:
        return refuse()

    # Stage 2: the real gate. Hand the chunks to the model with a
    # strict instruction: answer ONLY if the answer is actually here,
    # otherwise say you don't know.
    return generate_grounded_answer(question, chunks)

The two stages have completely different jobs:

• Stage 1 (the threshold) is now a coarse filter, not a precision instrument. I dropped it from 0.55 to 0.45. Low enough that it never rejects a real question. Its only job is to catch the rare case where retrieval comes back with nothing remotely usable, so I don’t waste a model call. It’s not trying to make the fine-grained “is this relevant” call anymore, because it can’t.

• Stage 2 (the model) is the real gate. The model can read the chunks and reason about whether they actually contain the answer. It is the exact judgment the similarity score can’t make. The system prompt from post #1 does the work: “Use ONLY facts found in the sources. If the sources do not contain the answer, reply “I don’t know based on the provided documents.”

So when the boiling-water question sneaks past the coarse filter (it scores 0.612, above 0.45), it hits the model, which looks at chunks about context engineering and harness design, sees no boiling point anywhere, and correctly refuses. The score got it in the door; the model threw it back out.

Result: 5/5 answered, 5/5 refused. Every off-topic question refused, every real question answered and not because I found the magic threshold, but because I stopped asking the threshold to be magic.

The takeaway

The lesson that stuck: use each tool for the job it’s actually good at, and don’t force one signal to make a decision it can’t make.

A similarity score is great at “rank these chunks by how related they look” and bad at “is the answer literally in here.” A language model is the reverse. It is expensive to call, but it can actually read. The bug wasn’t a wrong threshold value; it was using the wrong tool for the decision.

That’s a pattern I now see everywhere in these systems: the failure isn’t a bad parameter, it’s a good parameter being asked to do a job it was never capable of.

Next up: I take this same RAG agent and run it for 50 turns of conversation and watch one part of its “memory” quietly grow until it’s eating 60% of every request. It’s the moment “context engineering” stopped being a buzzword for me.

Building this stuff and writing down what breaks. Follow along.

I spent 10+ years building backend systems. A few months ago I started teaching myself AI engineering, and not the part where you train models. The part where you build systems around them: retrieval, context, agent loops, evals. This series is just me writing down what broke and what the numbers said. No “10 prompts that will change your life.” Just the stuff I actually hit.

Here’s the first one.

The setup: teaching a RAG app to say “I don’t know”

I was building a “chat with your docs” app. The classic RAG pipeline. You embed your documents, store the vectors in Postgres (pgvector), and at query time you retrieve the most relevant chunks and hand them to the model as context. Local and free the whole way: Ollama for inference, nomic-embed-text for embeddings.

The single most important feature of a RAG app isn’t answering questions. It’s refusing to answer when the answer isn’t in your documents. A RAG bot that confidently makes things up is worse than no bot at all. So the system prompt was strict:

SYSTEM_PROMPT = """You are a careful assistant that answers ONLY from the
provided numbered sources. Rules:
- Use ONLY facts found in the sources below. Do not use outside knowledge.
- Cite the sources you used inline like [1], [2].
- If the sources do not contain the answer, reply EXACTLY:
  "I don't know based on the provided documents."
- Be concise."""

To test it, I wrote a diagnostic: 5 questions whose answers are in the docs (should be answered + cited) and 5 that aren’t (should be refused). The scoreboard I wanted was 5/5 answered, 5/5 refused.

I got 4/5 answered, 5/5 refused.

The refusals were perfect. But one question that was clearly in my documents got refused too. The model looked at a chunk containing the exact answer and said: “I don’t know based on the provided documents”.

That’s not a hallucination. It’s the opposite. It’s a false refusal. And in a system where “refuse when unsure” is the safety rule, a model that refuses when it shouldn’t is quietly breaking the product. It’s lying about what it knows.

The debugging: was it retrieval, the prompt, or the model?

My systems-engineering instinct kicked in: isolate the layer. A RAG answer has three failure points: retrieval, the prompt, or the model, and you debug them in order.

Was it retrieval? Was the right chunk even reaching the model? I logged the similarity scores. The question was “What are the three risk categories in a permission matrix?” and the top chunk literally read: “The permission matrix classifies every tool as read_only, financial, or destructive”. Cosine similarity 0.62. Comfortably retrieved. Retrieval was fine.

Was it the prompt? Maybe “reply EXACTLY this string when unsure” was making it trigger-happy. Plausible! but the same prompt handled the other 4 in-doc questions correctly. If the prompt were the problem, it’d fail more consistently.

That left the model. So I ran the exact same context and prompt through two different models and diffed the output:

q = 'What are the three risk categories in a permission matrix?'
chunks, sim = retrieve(q)
user = f'Sources:\n{format_sources(chunks)}\n\nQuestion: {q}'

for m in ['llama3.2:latest', 'llama3.1:latest']:
    r = LLMClient(model=m).chat([{'role': 'user', 'content': user}],
                                system=SYSTEM_PROMPT)
    print(f'--- {m} (similarity {sim:.2f}) ---')
    print(r.text.strip())

Output:

--- llama3.2:latest (similarity 0.62) ---
I don't know based on the provided documents.

--- llama3.1:latest (similarity 0.62) ---
According to [1], the permission matrix classifies every tool as
**read_only**, **financial**, or **destructive**.

Same context. Same prompt. Same retrieved chunk with the answer sitting right there. llama3.2 (3 billion parameters) refused. llama3.1 (8 billion) answered correctly and cited its source.

The bug wasn’t in my code. It was in the model. The 3B model wasn’t capable enough to follow “answer from the source if it’s there, otherwise refuse”. It over-indexed on the “refuse” half of the instruction. Grounded refusal is a genuinely hard instruction-following task, and 3B just wasn’t up to it.

The fix: model choice is an engineering decision

Here’s the part that reframed how I think about this whole field. I was treating “which model” like a default. Grab the small fast one, move on. But model selection is an architectural decision, the same as picking a database or a queue. Different jobs want different models.

The catch: llama3.1 (8B) is correct but slow, ~28 seconds per call on my Mac (Ollama runs on CPU in Docker, no GPU). llama3.2 (3B) is ~4 seconds but not reliable enough for grounded RAG. And my agents which make many calls in a loop needed something fast that still follows instructions well.

So instead of one global model, I split by job, all driven from config:

# .env
OLLAMA_CHAT_MODEL=llama3.2:latest    # fast default
OLLAMA_RAG_MODEL=llama3.1:latest     # RAG generation — accuracy over speed
OLLAMA_AGENT_MODEL=qwen2.5:3b        # agent loops — fast AND follows format

The RAG pipeline pins itself to the stronger model where correctness matters:

client = LLMClient(model=settings.ollama_rag_model)

Re-ran the diagnostic. 5/5 answered, 5/5 refused. Fixed! not by touching a line of retrieval or prompt logic, but by putting the right model on the right job.

A bonus bug: the embedding model needed a magic word

While I was in there, I found a second, subtler issue. My similarity scores were all bunched up, relevant and irrelevant questions were scoring too close together to separate cleanly. At one point “what is the boiling point of water in Fahrenheit” (not in my docs) scored 0.61, while a real in-doc question scored 0.585. The off-topic question scored higher than the real one.

Turns out nomic-embed-text is an asymmetric model: it expects you to prefix queries and documents differently.

def embed_query(text):
    return embed(f"search_query: {text}")      # queries get this prefix

def embed_documents(texts):
    return embed([f"search_document: {t}" for t in texts])  # docs get this one

Adding the prefixes widened the gap between relevant and irrelevant enough to make the whole thing work. (The threshold-overlap problem deserves its own post. A single similarity cutoff genuinely cannot separate these cases, and the fix is a two-stage gate. More on that later.)

What I’d do differently

Being honest about the ceiling here:

1. This runs on local Ollama, free, private, great for learning. But on real hardware or a hosted model, the accuracy/speed tradeoff that forced the model-split largely disappears. The lesson (match the model to the job) stays; the specific split is an artifact of running 8B models on a laptop CPU.

2. I’d add a re-ranker. A cross-encoder between retrieval and generation would catch “plausible but wrong” chunks that similarity alone lets through.

3. I’d use the model’s native token counting and citations rather than approximating.

The takeaway

The thing that stuck with me: most of my “AI bugs” weren’t AI bugs at all. They were systems-design decisions wearing a costume. “Which model” is a capacity-planning question. “When does it refuse” is a validation-gate question. “Why do the scores overlap” is a data-representation question. The probabilistic component in the middle is new; the engineering around it is the same discipline I’ve done for a decade.

That’s the whole thesis of this series, and it’s why I think experienced engineers have a real edge in AI engineering, not despite skipping the ML-training rabbit hole, but because they can treat the model as a component and put good systems around it.

Next up: remember that off-topic question about boiling water that scored higher than a real one? That’s not a fluke, it’s why you can’t rely on a similarity score alone to decide when your RAG bot should answer. Next post: why that approach fails, and the two-layer fix that actually works.

Building this stuff and writing down what breaks. Follow along.

I didn’t start learning AI to train models. I started because I wanted to understand how to build reliable AI systems.

After spending more than a decade building distributed systems and backend platforms, I was curious about what actually happens behind the demos. How do RAG systems work? Why do agents fail? What makes one AI application feel reliable while another falls apart after a few interactions?

The more I built, the more familiar it all felt.

Context started to look like another engineering problem. Agent loops reminded me of distributed workflows. Evaluations felt like testing. Tool calling looked a lot like integrating external services, except the caller was now probabilistic.

AI engineering wasn’t replacing the engineering I already knew. It was building on top of it.

So I started experimenting.

I built RAG pipelines that failed in unexpected ways. I spent hours debugging retrieval when the real issue was somewhere else. I tried different agent architectures, prompt strategies, and context management techniques. Some ideas worked surprisingly well. Others failed spectacularly.

Each experiment taught me something I couldn’t have learned from documentation alone.

That’s why I’m starting How I AI.

This isn’t a tutorial series, and it isn’t a collection of prompt hacks.

It’s a journal of real engineering decisions, experiments, mistakes, and lessons learned while building AI systems from scratch.

If you’re a software engineer who’s curious about AI beyond the hype, I hope you’ll find something useful here.

Let’s build. 🚀

Imagine a shopping cart that’s not allowed to reject a write. Not during a disk failure, not during a network partition, not even if an entire data center goes dark. A customer hits “Add to Cart” and that request has to succeed, every time, no matter what’s on fire behind the scenes.

That’s not a hypothetical for Amazon. It’s a requirement. And it’s the requirement that a traditional relational database, with its locks and its transactions and its insistence on being correct before being available, simply can’t meet.

The paper Dynamo: Amazon’s Highly Available Key-value Store (DeCandia, Hastorun, Jampani, Kakulapati, Lakshman, Pilchin, Sivasubramanian, Vosshall, Vogels - SOSP 2007) is the story of how Amazon built a data store around that requirement instead of around correctness. It’s one of the most influential systems papers of the last twenty years, the direct ancestor of Cassandra, Riak, and Amazon’s own DynamoDB, and its central move is almost contrarian: it makes the write path dumb and always-on, and pushes the hard part, figuring out what actually happened, onto reads.

What Problem Was Amazon Solving?

By the mid-2000s, Amazon’s e-commerce platform was built from hundreds of loosely coupled services, each with its own storage needs. A single page render could fan out to over 150 internal services. Many of these services needed nothing more than primary-key lookups: shopping carts, session state, product catalogs, best-seller lists. No joins, no complex queries, no relational schema.

Running all of that on relational databases had become the actual bottleneck. Most of what an RDBMS offers, complex querying, strong transactional guarantees, was overkill for these access patterns, and the replication technologies available for scaling those databases out consistently chose consistency over availability whenever a partition or failure occurred. That’s the opposite of what a shopping cart needs. A customer should be able to add an item to their cart even if a disk is failing, a network route is flapping, or, in the paper’s own memorable phrasing, a data center is being taken out by a tornado.

So the real question wasn’t “how do we make our database faster”. It was:

How do we build a data store that is always writeable, even in the face of network partitions and server failures, and that scales by adding commodity hardware one node at a time?

Why Brute Force Wasn’t an Option

The obvious approach is the one every relational database already takes: synchronous replication. A write isn’t acknowledged until enough replicas have confirmed it, which guarantees that once you read your data back, it’s correct and current.

That’s exactly the approach that breaks under Amazon’s operating conditions. At the scale of tens of thousands of servers across many data centers, components fail continuously. That’s not an edge case to be engineered around, it’s the steady-state condition the system has to run in. Decades of distributed systems research had already established the uncomfortable tradeoff at the heart of this: when network failures are possible, you cannot have both strong consistency and high availability at the same time. You have to pick one to sacrifice.

Amazon picked consistency. Specifically, they decided that rejecting a customer’s write was a worse outcome than occasionally serving them a slightly stale or divergent read. So instead of asking “how do we make writes stronger,” the team asked “how do we make writes trivially easy to accept, and push all the hard reconciliation work to whenever the data is actually read?”

The Core Insight

Dynamo’s design rests on a handful of decisions that reinforce each other, but the load-bearing one is this: conflict resolution happens on reads, not writes, and, wherever possible, the application resolves conflicts, not the database.

Most systems do it the other way around: reject or serialize writes so that reads stay simple. Dynamo inverts that on purpose. A write is accepted as long as any reachable node in the system can durably store it. If that means two different customers’ updates to the same shopping cart end up on different physical replicas, so be it, the system will present both versions the next time anyone reads that key, and let the application (or, failing that, a last-write-wins timestamp) decide how to merge them.

This single inversion is what makes the rest of Dynamo’s architecture make sense:

• Consistent hashing partitions data across nodes so the system can add or remove a node without reshuffling the whole keyspace.

• Vector clocks let the system distinguish “this update descended from that one” from “these two updates happened independently and conflict,” without needing a centralized clock or coordinator.

• Sloppy quorums and hinted handoff let writes succeed even when the “correct” replica for a key is temporarily unreachable, by writing to a substitute node instead of blocking.

• Merkle trees clean up the mess left behind by all of the above, quietly reconciling replicas that drifted apart during a failure.

Put together, the shape of the system is: accept the write almost anywhere, encode enough metadata to reconstruct what happened later, and spend your engineering effort on making reconciliation cheap and safe rather than on making writes strict.

The Tradeoff

The most consequential decision in the paper isn’t a specific algorithm, it’s the choice to expose inconsistency to the application instead of hiding it.

Most databases treat conflict resolution as their own problem to solve internally, typically with a blunt policy like “last write wins.” Dynamo does support that mode, and some services use it (session state, where losing a stale write to a fresher one is harmless). But for anything where losing an update actually matters, like a shopping cart, Dynamo hands both versions back to the client and says, effectively, you know your data model, you decide how to merge these.

The tradeoff:

More complexity pushed onto application developers, in exchange for a data store that never has to choose between rejecting a write and losing one.

This is a deliberate rejection of the industry-standard approach at the time. ACID guarantees are convenient for developers, but Amazon’s own operational experience had shown that data stores providing full ACID guarantees tend to have poor availability. Since Amazon’s applications were already built to expect failures and inconsistencies as a normal condition of operating at scale, porting them to a model that surfaces those inconsistencies explicitly turned out to be a relatively small lift; the harder problem was greenfield application design, which required real thought about which conflict-resolution mode fit the use case.

Architecture

Dynamo is what the paper calls a zero-hop DHT: every node maintains enough routing information locally to send a request directly to the nodes that should handle it, without bouncing through intermediate hops the way structured peer-to-peer systems like Chord or Pastry do. That choice is deliberate. Multi-hop routing adds variance to response times, and variance is exactly what you don’t want if your SLAs are measured at the 99.9th percentile, which is how Amazon measures essentially everything.

Partitioning. Keys are hashed with MD5 into a 128-bit space, treated as a ring. Each physical node owns multiple random positions (”virtual nodes”) on that ring, so that when a node joins or leaves, the resulting load shift is spread evenly across many other nodes instead of dumping it all on one neighbor.

Replication. Each key is replicated at N hosts (a per-service configurable, typically 3). The node that owns a key’s position on the ring is the coordinator, and it replicates the key to the N-1 nodes that follow it clockwise. The full list of nodes eligible to serve a key is its preference list, which is intentionally longer than N so there’s slack to absorb failures.

Quorum knobs. Each service tunes two more numbers: R, the minimum number of nodes that must respond to a read, and W, the minimum number that must acknowledge a write. Setting R + W > N gives you quorum-like overlap between reads and writes. Crucially, these are configurable per service, not fixed system-wide, because different services have wildly different tolerance for staleness versus latency. A common production configuration was (N, R, W) = (3, 2, 2): tolerate one node down on either path, without paying for full N-way confirmation.

Notice what’s absent from that write path: no cross-node locking, no blocking on a majority if you don’t want to, no rejection just because the “ideal” replica is briefly unreachable.

Vector Clocks: Encoding “We Don’t Know Which One Is Right”

This is the mechanism I found most interesting in the paper, because it solves a problem most systems just paper over: how do you tell the difference between “this write is newer” and “these two writes genuinely don’t know about each other”?

A plain timestamp can’t do this reliably across machines with unsynchronized clocks. Dynamo instead attaches a vector clock to every version of every object: a list of (node, counter) pairs recording which coordinators have handled writes to that object and how many times. Comparing two vector clocks tells you one of two things: one is a strict ancestor of the other (safe to discard the ancestor), or neither dominates the other, meaning they’re concurrent, conflicting edits that a human or application has to reconcile.

Walking through the paper’s own example makes the mechanic concrete:

D3 and D4 are siblings, not ancestor and descendant. A node that only ever saw D1 or D2 can tell, on receiving D4, that both are now obsolete and can be garbage collected. But a node holding D3 that receives D4 can’t resolve it automatically, the two branches reflect changes that aren’t reflected in each other, so both versions get returned to the next reader, who (or whose application logic) merges them into a new version, D5, which folds the counters from every branch into a single clock.

For a shopping cart, that merge is straightforward: union the item sets from both versions. Nothing gets silently dropped. The tradeoff, as the paper is upfront about, is that this can occasionally resurrect a deleted item, a customer removes something from their cart, a concurrent branch that predates the deletion gets merged back in, and the item reappears. Amazon judged that failure mode (an item you have to remove twice) as dramatically preferable to the alternative (an “add to cart” that silently vanishes).

One practical wrinkle: vector clocks can grow unbounded if many different nodes end up coordinating writes to the same key, which mostly happens during network partitions or failures rather than in normal operation. Dynamo caps this by storing a timestamp alongside each (node, counter) pair and truncating the oldest entry once the clock exceeds a threshold (the paper uses 10). The authors admit this is a blunt instrument that can theoretically corrupt the ancestry chain, but note, with a bit of dry honesty, that it hasn’t actually caused problems in production and so hasn’t been investigated further.

Handling Failures Without Ever Blocking

Sloppy quorums and hinted handoff. A strict quorum system would still make Dynamo unavailable during a partition, if the “correct” N nodes aren’t all reachable, a strict scheme refuses the operation. Dynamo instead relaxes membership: reads and writes go to the first N healthy nodes on the preference list, wherever they happen to sit on the ring. If node A is briefly down, the replica that should have landed on A gets written to the next healthy node instead (say, node D), tagged with a hint saying “this really belongs to A.” Once A recovers, D forwards the hinted replica back and drops its own copy. The write never had to wait for A to come back.

This is also how Dynamo survives losing an entire data center: preference lists are constructed to span multiple data centers, so a full regional outage still leaves enough of a key’s preference list reachable elsewhere to keep serving traffic.

Merkle trees for anti-entropy. Hinted handoff only helps if the outage is short enough that the hint eventually finds its way home. For longer-lived divergence, each node maintains a Merkle tree per key range it hosts, a hash tree where every leaf hashes a key’s data and every parent hashes its children. Two nodes syncing a shared key range just compare root hashes first: if they match, everything underneath is identical and no data has to move. If they don’t, the nodes walk down the tree comparing child hashes until they isolate the specific keys that drifted, transferring only those. It turns “are these two multi-gigabyte replicas identical” from an expensive full comparison into a handful of hash exchanges.

What Breaks First? Load Balance, Not Bandwidth

Once the system is spread across a ring, the next question is whether that spread is actually even. Dynamo went through three different partitioning strategies in production before landing on one that worked well, and the failure modes of the earlier two are instructive.

Strategy 1, random tokens per node with partitioning tied directly to token position, was the original design. It had three separate problems in practice: bootstrapping a new node meant scanning other node’s local stores for the right key ranges, which was resource-intensive and, under heavy holiday load, sometimes took almost a full day to complete; every membership change forced expensive Merkle tree recalculation across many nodes; and there was no clean way to snapshot the whole keyspace for archival, since key ranges had no fixed boundaries.

The fix was to decouple partitioning (how the hash space is divided) from placement (which physical nodes own which partitions). Strategy 3, which Dynamo settled on, divides the space into a fixed number of equal-sized partitions up front and assigns each node a fixed count of them, independent of token randomness. Measured against strategy 1 at equivalent metadata overhead, this reduced the memory needed for membership information by three orders of magnitude while achieving better load-balancing efficiency, and it made both bootstrapping and archival dramatically simpler, because a partition maps to a single file that can be moved or archived as a unit instead of requiring a scan.

The broader lesson buried in that migration: the scheme you ship first because it’s the natural extension of a well-known algorithm (plain consistent hashing) is not necessarily the scheme that survives contact with your actual operational workflows (bootstrapping, archival, background maintenance). Those operational costs didn’t show up in the original design’s complexity analysis at all.

Subscribe now

The Numbers That Made It Real

A few figures from the paper are worth sitting with:

• Applications built on Dynamo received successful, non-timed-out responses for 99.9995% of requests, with no data loss event recorded up to the time of publication.

• Across a 24-hour profiling window of the shopping cart service, 99.94% of reads returned exactly one version of the object. Only 0.00057% saw 2 versions, 0.00047% saw 3, and 0.00009% saw 4. Divergent versions are the exception, not the norm, even in a system explicitly designed to tolerate them.

• Interestingly, the paper notes that the growth in divergent versions correlated with concurrent writers, not failures, and was mostly driven by automated client programs rather than human shopping behavior.

• Moving request coordination logic into the client library instead of routing through a load balancer cut 99.9th-percentile read and write latency from roughly 69ms down to about 30ms, by eliminating an extra network hop and letting clients route directly to the correct preference-list nodes.

• A small in-memory write buffer (as few as a thousand objects) cut 99.9th-percentile write latency during peak traffic by a factor of 5, at the cost of durability if a node crashed before flushing, a tradeoff explicitly offered as an option rather than forced on every service.

• The 99.9th-percentile latencies were consistently an order of magnitude higher than the average latencies, a gap the authors attribute to request-load variability, object size variance, and locality effects, and a big part of why Amazon insists on measuring SLAs at the tail rather than the mean.

Why This Paper Matters Beyond Dynamo

Dynamo’s direct descendants (Cassandra, Riak, Voldemort, and eventually Amazon’s own hosted DynamoDB) borrowed its techniques so thoroughly that it’s easy to forget how counterintuitive the core bet was at the time. A few ideas are worth carrying into systems that have nothing to do with key-value stores:

• Availability and consistency are a dial, not a binary, and the right setting is a business decision, not a technical one. Dynamo doesn’t pick a single tradeoff for every use case; it exposes N, R, and W as knobs so a shopping cart and a session store can land in completely different places on the same infrastructure.

• Pushing complexity to the read path, rather than the write path, is a legitimate design choice when writes are the operation you can least afford to reject. It’s uncomfortable because it means the “happy path” read now has to handle multiple versions, but that discomfort is far more contained than the discomfort of a rejected write.

• The scheme that looks simplest on a whiteboard is not the scheme that survives your actual operational workflows. Strategy 1’s failure wasn’t algorithmic, it was operational: slow bootstrapping, expensive re-hashing, hard-to-archive data. Those costs only show up once you run something in production for a while.

• Tail latency, not average latency, is usually what your customers experience, especially once a request has to satisfy more than one dependency (R or W replicas, in Dynamo’s case). Optimizing for percentiles instead of means shows up repeatedly, from the choice to measure SLAs at p99.9 to the decision to let clients route directly instead of trusting a load balancer.

My Takeaways

What stands out almost twenty years later is how honest the paper is about what it doesn’t solve. Vector clock truncation can theoretically corrupt ancestry and the authors say so plainly. Deleted shopping cart items can resurface and they say that too. Full membership gossip doesn’t scale past a few hundred nodes and they flag it as a known limitation rather than hiding it. That kind of candor about the sharp edges of a production system is rarer in systems papers than it should be, and it’s part of why this one held up.

The other thing that stuck with me: Dynamo isn’t really a paper about a clever algorithm. Consistent hashing, vector clocks, quorum systems, Merkle trees, gossip protocols, none of these were new in 2007. The contribution was combining them around a single, unusually clear design principle, never reject a write, and then following that principle all the way down through partitioning, versioning, and failure handling until the whole system was internally consistent with it. That’s a useful model for any system design problem: pick the one property you genuinely cannot compromise on, and let it discipline every other decision, rather than trying to optimize for everything at once and ending up with a system that’s mediocre on all axes.

Eighteen years later, “eventually consistent, application-resolved, quorum-tunable” is a phrase that describes half the databases powering the modern internet. Dynamo is the paper that made that phrase sound like engineering instead of a compromise.

Everyone is talking about AI agents. Harnesses. Loops. MCP. Planner agents. Multi-agent systems.

Somewhere along the way, two completely different ideas started getting blurred together. People say they’re “building a loop”. In reality, they’re usually building a better harness. Those are not the same thing.

One makes a single agent invocation reliable. The other makes an AI system reliable when nobody is watching. Understanding that distinction changes how you design autonomous systems because the two layers fail in different ways, and no amount of effort spent on one will fix a problem that actually lives in the other.

Think about an airline

The harness is the cockpit. It gives a pilot everything needed to safely fly one flight: instruments, controls, navigation, checklists, safeguards.

The loop is the airline itself. It schedules flights, dispatches crews, performs maintenance, remembers yesterday’s incidents, and decides when the next plane takes off.

You can’t build an airline by making the cockpit better. Likewise, you can’t build an autonomous AI system by continuously improving a single agent invocation. These are two different layers, solving two different problems.

Layer 1: Harness Engineering

A language model is just a probabilistic function. It has no built-in concept of tools, retries, verification, memory, or stopping conditions. A harness is everything wrapped around the model that turns one LLM call into something you can actually trust, which is the system prompt, tool schemas, tool execution, conversation history, retry policies, guardrails, iteration limits, grounding requirements, failure detection.

Its responsibility is simple: make one execution trustworthy.

Every harness runs the same loop

No matter which framework you’re using, be it LangGraph, the OpenAI Agents SDK, Claude Code, Cursor, or your own implementation, the core execution model looks remarkably similar: ask the model what it wants to do, check whether it requested a tool, execute the tool, feed the result back, repeat until it finishes.

def run_agent(user_input: str):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_input},
    ]

    for _ in range(MAX_ITERATIONS):
        assistant = call_model(messages, tools=TOOL_SCHEMAS)
        messages.append(assistant)

        tool_calls = assistant.get("tool_calls")
        if not tool_calls:
            return assistant["content"]  # final answer, loop ends

        for call in tool_calls:
            result = TOOL_IMPLEMENTATIONS[call["name"]](**call["arguments"])
            messages.append({"role": "tool", "content": str(result)})

That’s the execution loop. It is not, on its own, a harness. A harness starts where this code ends, because this loop faithfully executes whatever the model asks, and that’s fine right up until the model starts behaving like a model: answering from memory instead of using a search tool, repeatedly calling the same failing tool, declaring success without checking anything, hallucinating that it already verified a fact.

None of those failures get fixed with a better prompt. They get fixed with code.

A good harness doesn’t ask the model for permission

One of the first failures I hit was with a small local model that had a working search tool available and simply didn’t use it. It answered a factual question straight from memory, confidently, and got it wrong. The system prompt said, in plain language, always search before answering. The model ignored it.

That’s the whole lesson in one sentence: prompts are instructions, and harnesses are enforcement. So instead of trusting the model’s own judgment about whether it had done enough research, the harness rejected the premature answer outright:

if name == "finish_answer" and not used_grounding_tool:
    messages.append({
        "role": "tool",
        "content": "Rejected. You haven't gathered evidence yet.",
    })
    continue

The model isn’t deciding whether it should use tools here. The harness is. That’s the subtle but total shift between prompt engineering and harness engineering. One persuades, the other constrains. A model can ignore a prompt. It cannot ignore your Python.

Guardrails are just software

The other classic failure mode is the infinite tool loop. The model calls the same tool with identical arguments, gets the same result, and calls it again anyway. Rather than hoping it eventually stops on its own, the harness recognizes the pattern and ends it:

call_counts[key] = call_counts.get(key, 0) + 1
if call_counts[key] >= MAX_REPEATS:
    return "stuck", "Model repeated an identical tool call -- stopping."

None of this is AI magic. It’s ordinary software engineering. Iteration limits, timeouts, retry policies, circuit breakers, grounding requirements. None of it makes the model smarter. It makes the system safer, which is a different and, for production purposes, more important property.

A perfect harness still isn’t autonomous

Suppose you’ve built the perfect one. It always uses tools, never loops forever, grounds every answer, detects failures, retries intelligently. Congratulations! You still don’t have an autonomous system, because every morning, you’re still the scheduler. You’re the one deciding when it runs, the one reading every output, the one deciding what happens next. The harness is reliable. The system still depends entirely on a human standing next to it.

That’s where loop engineering begins.

Subscribe now

Layer 2: Loop Engineering

Addy Osmani recently put a name to the layer above the harness: loop engineering. Instead of asking how do I make one agent invocation reliable, it asks a different question entirely: how do I make an AI system operate reliably without me?

That’s not a bigger version of the same problem. Instead of improving one execution, you’re orchestrating many executions over days, weeks, or months, and the unit of design changes. You’re no longer writing prompts. You’re designing behavior over time.

The harness waits. The loop decides when it runs.

A harness doesn’t wake itself up. Something else has to decide it’s time to investigate an incident, review a pull request, or summarize the week’s emails. An event occurs, and the loop triggers the harness:

@app.post("/pagerduty-webhook")
async def webhook(request: Request, background_tasks: BackgroundTasks):
    payload = await request.json()
    background_tasks.add_task(investigate_incident, payload)
    return {"accepted": True}  # ack fast; the harness runs after this returns

Nobody clicked a button. The system decided.

The harness solves. The loop verifies.

Most agent systems stop the moment a model reaches a conclusion. Loop engineering assumes something stricter: every conclusion that’s going to be acted on deserves an independent reviewer; not another prompt, not self-reflection, but a genuinely separate execution with its own instructions.

maker_result = maker.run(incident)
approved, reason = checker.verify(maker_result, raw_tool_trace)

Here’s the detail that makes or breaks this pattern: raw_tool_trace has to actually be the raw trace, not the maker’s summary of it. It’s tempting to just pass the maker’s own evidence bullets to the checker. But that isn’t independent verification, it’s the model reviewing its own paraphrase of its own work. Building the real trail means walking the actual conversation and reconstructing what the tools returned, not what the maker claimed they returned:

def extract_tool_trail(messages: list) -> str:
    """Rebuild the real tool-call/result sequence from the conversation,
    so the checker sees what actually happened, not a summary of it."""
    trail, pending = [], []
    for msg in messages:
        if msg.get("role") == "assistant" and msg.get("tool_calls"):
            pending.extend(
                (c["function"]["name"], c["function"]["arguments"])
                for c in msg["tool_calls"]
            )
        elif msg.get("role") == "tool":
            name, args = pending.pop(0) if pending else ("unknown", {})
            trail.append(f"{name}({args}) -> {msg['content']}")
    return "\n".join(trail)

Skip this and “independent review” is a claim your architecture diagram makes that your code doesn’t actually back up.

The harness forgets. The loop remembers.

Every harness invocation starts from an empty conversation. Without persistent state, today’s investigation disappears the moment it’s done. Loop engineering keeps it around:

incident_store.record(incident)
similar = incident_store.search("database failover")

Now tomorrow’s investigation can learn from yesterday’s. Memory doesn’t belong inside the prompt. It belongs inside the system, outside the lifetime of any single run.

The harness uses tools. The loop connects systems.

A harness knows how to use a tool. A loop decides which tools exist in the first place. This is where something like MCP earns its keep, wiring GitHub, Jira, PagerDuty, Slack, Datadog, or internal APIs into the system once, so any harness invocation can draw on them. The harness consumes those capabilities. The loop is what assembled them into a coherent, autonomous workflow to begin with.

Subscribe now

Where the boundary actually is

The harness is the box in the middle. Everything around it; the trigger, the checker, the memory feeding back in; is the loop.

Which problem are you actually solving?

If your agent answers before gathering evidence, hallucinates tool results, loops forever, or ignores the tools it has; that’s a harness problem, and no amount of scheduling or sub-agents will fix it.

If your agent works perfectly task-by-task, but still needs you to wake it up, review every answer, approve every action, and remember what happened last time; your harness is probably fine. You’re missing the loop.

Subscribe now

Final thought

As AI systems get more autonomous, I think we’ll talk less about prompts and more about architecture, because prompts don’t build reliable systems, software engineering does. The future isn’t a smarter prompt or a bigger model. It’s the software wrapped around them, at both layers. Harnesses make individual executions trustworthy. Loops make entire systems trustworthy. The best AI systems won’t choose between the two; they’ll be built on both, deliberately, as separate design problems rather than one blurred into the other.

Loop Engineering: Addy Osmani frames it as the layer directly above what he separately calls “agent harness engineering”. A single harness runs on a schedule, spawns helper agents, and feeds itself, rather than waiting on a human to invoke it each time.

Imagine you have to start 15,000 fresh containers per second, for a single customer, and each one might be backed by a 10GiB image. Multiply it out and you’d need 150 petabits per second of network bandwidth just to move the bytes, even before a single line of the customer’s code has run.

That’s the wall AWS Lambda hit when it decided to support container images. Lambda had built its reputation on cold starts as fast as 50ms. Customers wanted to bring Docker-sized dependency trees. Those two things, on the surface, don’t coexist.

The paper On-demand Container Loading in AWS Lambda (Brooker, Danilov, Greenwood, Piwonka) is the story of how the Lambda team reconciled them. They did it not by moving data faster, but by figuring out how to avoid moving most of it at all.

What Problem Was AWS Solving?

When Lambda launched in 2015, functions were uploaded as zip files capped at 250MB. Simple: download the archive, unpack it, run the code. It worked because the archives were small and the whole thing could be shipped and unzipped before the MicroVM needed to do anything.

By 2020, customers wanted more. They wanted to bring entire container images (up to 10GiB) built with normal Docker tooling, full of ML libraries, native dependencies, whatever their application needed. That’s a 40x jump in payload size, applied to a system that already had to spin up new capacity thousands of times a second while keeping start times in the tens of milliseconds.

Naively downloading and unpacking a full image per MicroVM doesn’t scale. So the real question the team faced wasn’t “how do we support bigger files”. It was:

How do we let customers ship anything up to 10GiB, without making a single container start ever have to move 10GiB?

Why Brute Force Wasn’t an Option

The obvious approach would be to download the image, unpack it, boot and that is exactly what the original zip-based system did, and it’s exactly what breaks at container scale.

At 15,000 new containers per second, even modest per-container downloads compound into unmanageable network and storage load. And the team wasn’t just fighting bandwidth. They were fighting unnecessary work: most of what’s inside a 10GiB image is never touched during a typical invocation. Prior research the authors cite found that on average only 6.4% of container data is needed at startup.

So the system doesn’t just need to move data faster. It needs to avoid moving data the function will never read.

The Core Insight

The paper’s architecture rests on three empirical properties of real-world Lambda usage, which the authors call out explicitly:

• Cacheability: a small number of images drive most of the scale-up spikes, so the hot set is small relative to the total corpus.

• Commonality: huge numbers of images share the same base layers (official Alpine, Ubuntu, Node images, AWS’s own base layers), so there’s enormous overlap between “different” images.

• Sparsity: most of an image’s bytes are never touched at startup.

Put together, these three properties suggest the same conclusion from three different angles: you don’t need to move a 10GiB image per container start. You need to move the small, shared, frequently-reused slice of it that’s actually requested, and you need to do that at the block level, not the file level.

That’s the whole design in one sentence: flatten the image once (at deploy time, which is rare), chunk it, deduplicate the chunks across the fleet, cache the popular ones aggressively, and load only the blocks the guest kernel actually asks for, on demand, over a virtual block device.

Everything else in the paper, the encryption scheme, the garbage collector, the erasure coding, exists to make that one sentence work safely and fast at Lambda’s scale.

The Tradeoff

Most systems that accelerate container startup work at the filesystem level. That’s what Slacker and Starlight do: they lazily load individual files or layers as they’re needed, which is a natural fit because containers are already built as stacked, overlaid archives.

AWS deliberately didn’t do that.

They chose to keep the boundary between the MicroVM guest and the hypervisor at the raw block device layer (virtio-blk) and push all filesystem logic inside the untrusted guest, rather than exposing filesystem semantics to shared host-side components.

Why give up the natural fit? Security. The authors state plainly that layering multiple filesystems and understanding filesystem semantics on the trusted side of the boundary would unacceptably widen the attack surface of components shared across customers. A block device is a much simpler, better-understood interface to expose across a trust boundary than a filesystem.

The tradeoff:

Less semantic richness at the loading layer, in exchange for a smaller, simpler, more auditable attack surface.

To make block-level loading work without filesystem awareness, they had to do something clever at deploy time: deterministically flatten the whole OCI image, including every stacked tarball layer, into a single ext4 filesystem using a custom deterministic filesystem implementation (no concurrency, no random timestamps). As a result, identical file content always produces identical blocks. That determinism is what later makes chunk-level deduplication possible without any central index.

Architecture

The system splits cleanly along the same fault line as the rest of Lambda: a low-rate, per-function control plane step (image creation), and a high-rate, per-invoke data plane step (loading).

At function creation (whenever a customer pushes new code, rare, maybe a few times a minute across all of CI/CD):

1. The container image is pulled from the registry.

2. It’s deterministically flattened into an ext4 filesystem image.

3. That filesystem is chopped into fixed 512KiB chunks.

4. Each chunk is content-addressed, encrypted, and uploaded to S3 (the “origin tier”) if it doesn’t already exist there.

At invoke time (up to 15,000 times a second, per customer):

1. A new MicroVM boots against a FUSE-backed virtual block device.

2. As the guest kernel and customer code read files, those reads become block requests.

3. A local agent on the worker serves those requests from a per-worker cache if possible.

4. On a local miss, it pulls the chunk from an availability-zone-level distributed cache.

5. On a cache miss there too, it falls back to S3, the origin of last resort.

The result: image creation is expensive-ish but rare, and invocation is cheap because it only ever touches the handful of chunks a given execution actually needs, most of which are already sitting in a cache somewhere because thousands of other customers’ functions share the same base-image chunks.

Deduplication Without Trust

This is the part of the paper I found most interesting, because it solves a problem that sounds contradictory: deduplicate data across mutually-untrusting customers, without ever sharing an encryption key between them.

Encryption is supposed to make ciphertext look random. Deduplication depends on identical plaintext producing identical, comparable data. Those two goals fight each other. The same content encrypted under two different keys produces completely different ciphertext, so you lose your dedup signal the moment you encrypt normally.

AWS’s answer is convergent encryption, an idea that traces back to the Farsite paper from 2002. It derives the encryption key for a chunk from the chunk’s own content (its SHA256 digest), rather than from a customer’s private key. Identical content anywhere in the fleet converges on the identical key, and therefore the identical ciphertext, which is exactly what dedup needs, without any two customers ever having to share a secret.

The per-customer secret still exists, but its job is narrower, it encrypts the manifest (the map of chunk names and keys for a given image), not the chunks themselves. That means a garbage collector can walk the list of chunk names in a manifest without ever having the keys to decrypt their contents, a nice separation of duties.

The payoff, measured in production: approximately 80% of newly uploaded Lambda functions result in zero unique chunks, being re-uploads of previously seen images, largely driven by CI/CD systems, while the remaining 20% that do introduce new content have a mean of only 4.3% unique chunks and a median of 2.5%. That’s a huge amount of redundant data that never has to be stored, moved, or cached twice.

Because widely-shared chunks also mean widely-shared blast radius (corruption or a bad cache node affecting one popular chunk could ripple across many customers), they deliberately weaken deduplication on purpose sometimes, mixing a rotating salt into the key derivation so that otherwise-identical chunks can be forced to diverge across time or availability zones, trading some dedup efficiency for smaller failure domains.

Failure Handling: Garbage Collection as the Scary Part

The authors are refreshingly candid that the riskiest part of this whole system isn’t loading, it’s deleting. Get garbage collection wrong in a deduplicated store and you can silently destroy another customer’s data.

Imagine two customers upload different container images:

Image A
Manifest A
 ├── Chunk A
 ├── Chunk B
 └── Chunk C

Image B
Manifest B
 ├── Chunk B
 ├── Chunk D
 └── Chunk E

Notice that Chunk B is shared. Thanks to deduplication, Lambda stores only one copy of it.

Now suppose Customer A deletes Image A.

Can AWS delete Chunk B?

No, because Image B still references it.

At Lambda’s scale, keeping an exact reference count for billions of chunks would mean constantly updating metadata whenever images are created, deleted, or rewritten. Those updates would become one of the busiest and most failure-prone parts of the storage system. Worse, a single incorrect reference count could cause live customer data to disappear.

Rather than maintaining a central reference-counted index of every chunk (which they consider both complex and uniquely dangerous, given how quickly a live reference graph changes), they use a generational, root-based scheme.

Initially, all converted images are written into an active root:

Root R1 (Active)

Manifest A ──► Chunk A
             ├► Chunk B
             └► Chunk C

Manifest B ──► Chunk B
             ├► Chunk D
             └► Chunk E

After some time, AWS creates a new active root.

Root R1 (Retired, Read-Only)

Manifest A
Manifest B

-------------------------

Root R2 (Active)

(new images go here)

From this point onward, no new writes ever go into R1.

Now imagine Customer B still actively uses Image B.

Instead of keeping Manifest B in the old root forever, Lambda copies both the manifest and every chunk it references into the new root.

Root R2

Manifest B
 ├── Chunk B
 ├── Chunk D
 └── Chunk E

Once every still-live image has been migrated, nothing in R1 is needed anymore.

Instead of asking:

Can I delete Chunk A?

Can I delete Chunk B?

Can I delete Chunk C?

AWS simply asks:

Can I delete Root R1?

If the answer is yes, millions of chunks disappear in one operation.

This turns garbage collection from tracking billions of individual references into deleting entire storage generations.

One subtle detail makes this work. Once R2 becomes active, newly uploaded chunks are not allowed to deduplicate against chunks in R1. AWS includes the active root’s identifier in the deduplication salt, ensuring that identical uploads create new chunks inside R2 rather than reusing chunks from older roots.

Without this, a new image in R2 could accidentally reference data stored in R1:

Root R2
Manifest X
    │
    ▼
Chunk B (stored in R1)

Now R1 can never be deleted because the newest generation still depends on it. By restricting deduplication to the active root, every root remains self-contained and can eventually be reclaimed.

The safety mechanism I liked most comes at the very end of the lifecycle. Instead of deleting a retired root immediately, AWS first marks it expired. An expired root is still readable, but any read immediately raises an alarm and automatically halts further deletion. It’s essentially a tripwire.

The protection isn’t perfect, a read could arrive milliseconds before the alarm propagates, but it transforms the failure mode from silently deleting live customer data into an automatically interrupted operation that engineers can investigate before more damage occurs. For a storage system serving millions of customers, that’s exactly the kind of failure mode you want: loud, observable, and recoverable instead of silent and irreversible.

What Breaks First? Tail Latency, Not Bandwidth

Once you accept that most reads will hit a cache, the next question is: what happens when a cache node is merely slow, not down?

The authors argue this is actually the harder problem operationally, since debugging partial slowness is harder than debugging outright failure, and it’s magnified by how many chunks a single container start touches. If a start needs 1,000 chunks, it will experience the cache’s 99.9th-percentile latency on 63% of tasks, simply due to how many independent draws it’s making.

Rather than the standard fix (replicate data, send redundant requests, eat the extra cost), they reach for erasure coding, following a similar approach to EC-Cache. In production they run a 4-of-5 code: any chunk fetch requests slightly more stripes than strictly necessary and reconstructs as soon as enough arrive, at a cost of roughly 25% extra storage and request volume, in exchange for much better tail latency than either a bare single-copy cache or full replication would give at the same cost.

They also explicitly avoid retry-based masking of slow or failed nodes, because retries under load are a known trigger of metastable failure. Erasure coding lets them do a fixed, predictable amount of work per request regardless of whether things are going well or badly, a philosophy they call “constant work.”

The Numbers That Made It Real

A few figures from the paper stood out:

• On-worker cache hit rate: a median of 67%, with a 10th-percentile low of 65% over a week of production traffic.

• AZ-level cache hit rate on top of that: a median of 99.9%, with a 10th-percentile low of 99.4%.

• Overall, across a full week in one large AWS region: a median of 67% of chunks served from the on-worker cache, 32% from the AZ-level cache, and just 0.06% from S3, the backing store.

• Latency difference between a cache hit and an S3 fetch: a median of 550 microseconds versus 36 milliseconds, roughly 65x, with the gap widening at the tail, 3.7ms versus 175ms at p99.9.

• Deduplication’s storage impact: as much as a 23x reduction for functions that do introduce unique content, and another 5x on top of that from the 80% of uploads that are pure re-uploads.

The headline result, though, is less a number and more a design philosophy: the system was built so that an empty cache (from a power event, a bad deploy, whatever) doesn’t cascade into an outage. Because a cache running at 99.8%+ hit rate normally means a cold cache could throw 500x the usual traffic at S3, they deliberately bound blast radius with concurrency limits, and they actually test cold-cache starts at max concurrency rather than hoping it never happens.

Why This Paper Matters Beyond Lambda

A lot of container-loading research (Slacker, Starlight, eStargz, DADI, FaaSNet) attacks the same underlying problem: don’t ship the whole image, ship what’s needed, when it’s needed. What makes this paper worth reading even if you’ll never build a FaaS platform is the pattern, not the specific system:

• Sparsity, commonality, and cacheability are properties you can probably find in your workload too, and each one independently suggests a different lever (don’t-move-what-isn’t-read, dedup-what’s-shared, cache-what’s-hot).

• Security constraints can and should shape your architecture more than performance constraints do. The choice of block-level over filesystem-level loading here is a security decision dressed up as a performance one.

• Deduplication and encryption aren’t actually opposed if you’re willing to derive keys from content instead of holding them centrally, a trick worth remembering any time “encrypt everything” and “don’t store duplicates” show up in the same set of requirements.

• The scariest failure mode in a system like this usually isn’t the hot path. It’s the one place you delete data. Design that path to fail loudly and slowly, not silently and fast.

The team also underlines a broader lesson worth stealing directly: multi-modal latency distributions (local hit, remote hit, cold fetch) are the norm in storage systems, and single summary statistics, even percentiles, tend to hide that shape. If your system has more than one path a request can take, plotting the empirical CDF will usually tell you more than another dashboard full of p50/p99 numbers.

My Takeaways

What stuck with me most is how much of this paper is about not moving data rather than moving it faster. The team had every incentive to just throw more bandwidth or more caching hardware at the problem, and they still needed caching, still needed erasure coding, but the foundational move was architectural: flatten once, chunk small, dedupe hard, load lazily. Every optimization after that is amplifying a decision made long before the first byte crosses a network link.

The other thing worth sitting with: they chose Rust for a brand-new, security-critical, performance-critical component inside a system that already had Java, Go, and C in its invoke path, and their honest accounting of it includes a genuinely strange autovectorization bug that cost them real engineering time. It’s a useful reminder that “we picked the safe, fast language” doesn’t mean the compiler always cooperates; even well-established toolchains have sharp edges that only show up under production-grade optimization pressure.

Twenty-plus years after Venti and Farsite first explored content-addressed deduplication, this paper is a good demonstration that those old ideas aren’t just academic history. They’re still the right tool when you’re facing down a genuinely new scale problem, as long as you’re willing to rebuild the trust model around them for a new environment.

In my experience, writing code was rarely the biggest bottleneck.

The real friction lived everywhere around the code. Rebuilding context. Reading old tickets. Tracing behaviour across services. Understanding past decisions. Checking CI pipelines. Figuring out who needed to review what. Recovering after interruptions.

Individually, none of these tasks were difficult. Together, they consumed a surprising amount of engineering time.

That’s what led me to build an AI-native engineering workspace.

Not another coding assistant. A workspace designed around the reality of engineering work, where context, workflows, approvals, knowledge, and operational tools work together instead of existing in separate silos.

For me, it became much more than a repository. It became a development environment, an operating model, and a workflow layer for my engineering.

Why I Built It

I built this because too much of my day was being spent on coordination overhead.

A typical task rarely started with code. It started with gathering context. Opening tickets, reading historical discussions, tracing requests across services, validating local setup, checking build status, and understanding previous decisions before I could confidently make the next one.

The biggest slowdown was not the complexity of the work itself.

It was the repeated cost of stitching together information spread across systems, tools, and people.

I wanted a single workspace that could:

• Provide a reproducible development environment

• Preserve system knowledge and historical context

• Route work through the right workflow

• Understand team ownership and approval processes

• Connect seamlessly to the tools surrounding the code

That combination ended up changing my day-to-day productivity far more than any individual AI feature ever did.

What The Workspace Looks Like

At a high level, the workspace is built from six layers:

Each layer removes a different kind of friction. Together they make the workspace feel less like a chat tool and more like a working system.

The Devcontainer Is The Foundation

The workspace would not be useful if setup was painful, so the first principle was reproducibility.

The environment is Docker-first. The root compose setup gives me a stable Linux workspace, mounted source, persistent caches, and Docker access from inside the container. On top of that, the devcontainer provides the actual toolchain: Java, Maven, Node, Android SDK support, terminal defaults, editor extensions, and the environment variables needed for the projects in the workspace.

That matters because it removes startup uncertainty. I do not need to remember which version of Java works, whether Node will break, whether Gradle is installed, or whether I am missing some machine-specific setup from months ago. I reopen in container and start from a known baseline.

This is not just convenience. It is productivity insurance.

The Bootstrap Script Turns The Container Into A Real Workspace

The devcontainer gives me the shell. The setup script turns it into an actual engineering environment.

It handles the repetitive work that otherwise steals time at the start of a session:

• validates .env presence for token-based integrations

• clones or updates the source repos used in the workspace

• installs workspace-managed .cursorrules where needed

• initializes submodules

• creates .context-local/runtime/ for per-ticket artifacts

• copies .devcontainer/mcp.json into .cursor/mcp.json

• pre-pulls Testcontainers images

• warms Maven, Gradle, Yarn, and npm dependencies

• checks emulator connectivity for the mobile workflow

This is one of the least glamorous parts of the setup, but it is one of the most valuable. A lot of engineering time disappears into local drift and repeated setup work. Capturing that in code means I stop paying the same tax over and over.

The Knowledge Base Makes The Agent Useful Beyond Code Generation

Without a knowledge base, an agent only knows the current prompt and whatever files it reads on demand. That works for small edits. It does not work well for system-level tasks.

The knowledge base gives the workspace durable memory. It captures architecture, APIs, events, dependencies, flows, and implementation patterns. Just as important, it tracks freshness, so the system can tell when knowledge is stale and should be rebuilt.

That changes the quality of the interaction dramatically.

Instead of asking the agent to search blindly, I can ask it to explain a flow, estimate impact, reason about a change, or investigate a production issue with actual structural context behind the answer.

That is one of the biggest reasons this setup feels productive: I spend less time reconstructing the system from scratch.

The Delegation Rule Is What Makes The Workflows Reliable

One of the smartest parts of the workspace is the dispatcher rule.

I did not want a single vague AI prompt trying to do everything. I wanted explicit workflows. So the workspace routes commands like plan <ticket>, analyze <ticket>, implement <ticket>, review <ticket>, investigate <symptom>, and what's next to different rule files with different behavior.

That seems simple, but it creates a lot of leverage:

• scope is clearer

• outputs are more consistent

• phases do not bleed into each other

• prompts become reusable commands instead of handcrafted requests

In other words, the workspace delegates by intent. When I use a command, I am not just starting a conversation. I am invoking a contract.

This is a major productivity win because it reduces prompt-writing overhead and makes the workspace much easier to trust.

team.yaml Makes The Workspace Aware Of Human Workflow

Most AI systems can generate a decent answer. Fewer systems understand who needs to act on it.

team.yaml solves that by giving the workspace a model of the team: members, roles, JIRA identities, GitHub identities, tagging rules, and approval gates.

That means the workspace can do more than prepare an artifact. It can route that artifact into the right human workflow.

For example, it can:

• resolve who should approve planning or design work

• determine whose approval actually counts

• mention the right JIRA usernames in the correct format

• add watchers intentionally

• stop progress when approval conditions are not met

That last part matters. The review-gate logic checks whether an approval came from an authorized approver, not just whether someone wrote “LGTM.” That is a subtle but important improvement. It keeps the system fast without making it sloppy.

MCP Is What Turned This Into An Operational Workspace

The biggest jump in usefulness came from MCP integration.

Without MCP, the agent can help me think and edit local files. With MCP, it can operate across the systems that actually shape engineering work: tickets, pull requests, CI, internal knowledge, and quality signals.

In this workspace, MCP connects the agent to:

• GitHub

• JIRA

• Buildkite

• Glean

• SonarQube

• GCP-related tooling

• local filesystem access

Some of these integrations run via Docker, others over HTTP. The important point is that they are part of the same loop. The agent can move from code to ticket, from ticket to CI, from CI to a local artifact, and from there to the next recommended action.

That removes a huge amount of manual context translation.

The Best Example: Incident Work Became Sharper And Faster

One of the clearest examples of productivity gain came from incident analysis.

In this workspace, I used the incident flow to produce a detailed RCA for an automatic ticket-creation failure. The output was not just a summary paragraph. It was a structured artifact with impact, timeline, technical analysis, evidence, root cause, resolution steps, and action items.

Why does that matter? Because work like that normally requires bouncing across ticket history, platform details, code, infrastructure clues, and prior assumptions. In this setup, the workflow is designed to gather those pieces, structure them, and save them back into a ticket-scoped artifact.

The value was not “the AI wrote a document for me”. The value was that the workspace reduced the time needed to assemble evidence, structure the investigation, and preserve the result.

Another Example: what's next Reduced Decision Friction

The what's next workflow is smaller, but it has probably saved me more daily friction than almost anything else.

It does not guess. It inspects state:

• knowledge-base freshness

• local git state

• assigned JIRA work

• linked PR state

• local runtime artifacts

• pending approvals

That means when I ask what to do next, I get an answer grounded in the actual condition of the workspace, not a generic suggestion.

This is especially useful after interruptions. Instead of manually reconstructing context, I can ask once and resume quickly.

Why The Runtime Artifact Model Matters

Another high-leverage choice was storing work under .context-local/runtime/ by ticket.

That means research, tech specs, incident reports, reviews, and approval state are not trapped in a long chat thread. They become durable working files tied to the task they belong to.

This improves productivity because:

• I can pause and resume without losing context

• the agent can continue from prior artifacts instead of starting over

• one phase naturally feeds the next

• ticket history becomes inspectable locally

This is one of the reasons the workspace feels cumulative. It does not just help in the current moment. It preserves momentum.

Why This Felt Like A 10x Productivity Improvement

I do not mean 10x in a simplistic benchmark sense. I mean that the dead time in my workflow dropped sharply.

I spend less time:

• rebuilding context

• fixing local setup

• rewriting prompts

• moving between tools manually

• figuring out who needs to approve what

• recovering after interruptions

And I spend more time on the parts of engineering that actually need judgment: design choices, edge cases, trade-offs, risk, and communication.

That is the real multiplier.

Closing Thought

What changed my productivity was not one AI feature. It was combining a devcontainer, a bootstrap script, a knowledge base, delegation rules, team.yaml, MCP integrations, and ticket-scoped artifacts into one coherent setup.

Once those pieces started working together, the workspace stopped feeling like a tool and started feeling like leverage.

Imagine being asked to process a terabyte of data across thousands of machines. Some servers crash halfway through execution. Others become painfully slow. Network links saturate. Disks fail. And despite all of that, the final result still needs to be correct.

Today, we take it for granted that distributed systems platforms handle these problems for us. Frameworks automatically retry failed tasks, rebalance workloads, and recover from machine outages. But in the early 2000s, none of this was standard.

If you wanted to process data at scale, you often had to build a distributed system before you could even begin solving your actual business problem. Engineers had to think about machine placement, scheduling, partitioning, fault tolerance, retries, and data movement long before they wrote the logic they actually cared about.

The first time I read Google’s MapReduce paper, I was surprised by how little of it is actually about computation. Most of the paper is about failure, coordination, and moving data efficiently. More than twenty years later, those are still the same problems that dominate large-scale distributed systems.

Google’s insight was deceptively simple: what if developers didn’t need to think about any of that?

Instead of asking engineers to build distributed systems, MapReduce asked them to write just two functions: map() and reduce(). Everything else like scheduling, fault recovery, parallelization, load balancing, and data movement became the platform’s problem.

That single idea changed how the industry thinks about large-scale data processing.

What Problem Was Google Solving?

In the early 2000s, Google’s growth created a problem that traditional software architectures simply couldn’t handle.

Search indexing, log analysis, web crawling, and machine learning workloads required processing enormous datasets across clusters containing thousands of commodity machines. Running these jobs wasn’t particularly difficult when everything worked perfectly.

The real challenge was that everything rarely worked perfectly.

At Google’s scale, machine failures were routine. Disks failed. Network links slowed down. Some servers disappeared in the middle of execution, while others became unexpectedly slow. The larger the cluster became, the more likely it was that something would go wrong during every job.

Most distributed systems at the time treated failures as exceptional events. Google had reached a scale where failures had become part of normal operation.

This created a much deeper problem than simply processing large amounts of data.

Every team that wanted to analyze data effectively had to become experts in partitioning, scheduling, fault tolerance, retries, load balancing, and data movement. The business logic was often the easy part. Building the distributed system around it was the hard part.

Google’s question wasn’t:

How do we process a terabyte of data?

It was:

How do we make large-scale data processing accessible to normal engineers without forcing them to become distributed systems experts?

MapReduce was Google’s answer.

Why Existing Approaches Were Not Enough

Before MapReduce, every large-scale data processing job effectively became its own distributed system.

Suppose a team wanted to count words across billions of documents or analyze terabytes of logs. The computation itself was often straightforward. The hard part was everything around it.

Engineers had to figure out how to partition data across machines, schedule work efficiently, recover from failures, handle slow workers, balance load, and coordinate execution across the cluster. Application code quickly became tangled together with infrastructure concerns.

The actual business logic often represented only a small fraction of the overall implementation.

As clusters grew larger, the problem became even worse. More machines meant more failures, more network traffic, and more coordination overhead. Teams repeatedly solved the same distributed systems problems for every new data-processing application they built.

What’s interesting is that computation itself was rarely the bottleneck. The real challenge was coordination.

Moving data across the network, recovering from failures, balancing workloads, and ensuring correctness consumed far more engineering effort than the computation being performed.

Google noticed something important: many of these applications followed a similar execution pattern. If that common pattern could be standardized, the distributed systems complexity could be built once and reused everywhere.

Instead of every team building its own distributed system, Google could provide a framework that handled the hard parts automatically.

That realization eventually became MapReduce.

The Core Insight

The breakthrough behind MapReduce wasn’t a new algorithm. It was a new abstraction.

Google realized that many large-scale data processing jobs, despite looking very different on the surface, followed a remarkably similar pattern. Data was read, transformed into intermediate results, grouped by some key, and then aggregated into a final output.

MapReduce turned that pattern into a programming model.

Developers only needed to describe their computation using three stages:

Map → Shuffle → Reduce

At first glance, this seems like an unusually restrictive way to write software. But that restriction is precisely what gives the system its power.

Once the runtime knows that every job follows the same execution pattern, it can take responsibility for everything else. It can automatically parallelize work across thousands of machines, schedule tasks efficiently, move computation closer to data, recover from failures, retry lost work, and rebalance slow workers.

The developer focuses on the problem they are trying to solve.

The platform focuses on running that computation reliably at scale.

What makes this idea so powerful is that it changes the role of the programmer. Instead of building a distributed system for every new application, developers simply describe the computation they want to perform. The infrastructure takes care of the rest.

That separation between business logic and distributed systems complexity became one of the most influential ideas in modern infrastructure.

The Tradeoff

The brilliance of MapReduce comes from what it deliberately refuses to do.

Most distributed systems try to be flexible. They provide developers with powerful primitives and let them decide how computation should be executed across machines.

MapReduce takes the opposite approach.

Instead of supporting every possible distributed computation, it asks a much simpler question:

What if we only support the subset of problems that fit the Map, Shuffle and Reduce model?

At first glance, that sounds limiting.

And it is.

MapReduce gives up many capabilities that developers might want. There is no arbitrary communication between tasks. There is no fine-grained control over execution. Stateful iterative workloads are difficult to express. Complex dependency graphs don’t fit naturally into the model. Low-latency processing isn’t a design goal.

But those limitations are exactly what make the system powerful.

Because the runtime understands how every job behaves, it can automate much of the complexity that normally falls on developers. Scheduling, fault recovery, load balancing, task placement, retry logic, and parallel execution all become infrastructure concerns rather than application concerns.

The tradeoff is simple:

Less flexibility for the programmer. More leverage from the platform.

What MapReduce sacrifices in generality, it gains in simplicity, scalability, fault tolerance, and operational reliability.

What’s remarkable is how often this idea appears in modern systems.

At Wayfair, most engineers don’t think about leader election, partition recovery, retry coordination, or distributed consensus when building services. Platform teams and infrastructure abstractions handle those concerns. Developers focus on business logic.

MapReduce was one of the earliest and most influential examples of this philosophy. Instead of asking every team to become distributed systems experts, it pushed complexity downward into infrastructure where it could be solved once and reused everywhere.

In many ways, that idea became the blueprint for modern platform engineering.

Architecture

One reason MapReduce became so influential is that the architecture mirrors the programming model itself.

Every component exists to support one of three stages: Map, Shuffle, or Reduce. There are surprisingly few moving parts, yet together they solve some of the hardest problems in distributed systems, task scheduling, fault tolerance, parallel execution, and data movement across thousands of machines.

What’s remarkable is how much complexity is hidden behind such a simple interface. Developers see two functions. Underneath, an entire distributed execution engine coordinates work across the cluster.

At a high level, the architecture looks like this:

Input data lives in Google’s distributed file system and is divided into smaller chunks. Map workers process those chunks independently and emit intermediate key-value pairs.

Those intermediate results are then partitioned by key and stored locally. During the shuffle phase, reduce workers fetch the partitions they need from every mapper, sort them, group values belonging to the same key, and execute the reduce function to produce the final output.

Simple in concept.

Remarkably powerful in practice.

In many ways, the architecture embodies the central idea of the paper: developers describe the computation, while the platform takes responsibility for everything required to execute that computation reliably at scale.

Example: Word Count

The classic example is counting word frequency.

Suppose you have billions of documents and want to know how many times each word appears.

Map

Each worker emits:

("chair", 1)
("sofa", 1)
("chair", 1)
("lamp", 1)

Shuffle

The system groups identical keys together:

("chair", [1,1,1,1,...])
("sofa", [1,1,...])
("lamp", [1,1,1,...])

Reduce

Reducers aggregate the values:

("chair", 12849302)
("sofa", 8349201)
("lamp", 4419022)

The programmer writes a few lines of logic.

The platform handles:

• Splitting data

• Scheduling work

• Moving data

• Recovering from failures

• Handling slow machines

• Writing final outputs

That was revolutionary.

MapReduce changed the unit of abstraction from:

Build a distributed pipeline

to

Write two functions.

Failure Handling

One of the reasons the MapReduce paper became so influential is how aggressively it embraces failure.

Many distributed systems are designed around the assumption that failures are exceptional events. MapReduce assumes the opposite. At Google’s scale, machine failures were not edge cases. They were part of normal operation.

The framework is built on a simple idea: failures should not require human intervention.

If a worker machine crashes halfway through a task, the system doesn’t panic. Workers are monitored through heartbeats, and missing workers are treated as failed. Any in-progress tasks are reassigned to healthy machines, lost map outputs are recomputed, and the job continues running.

Network failures are handled in a similarly pragmatic way. From the master’s perspective, an unreachable worker is indistinguishable from a failed worker. Rather than trying to determine the exact cause, the system simply reschedules the work elsewhere.

Even disk failures are treated as recoverable events. Since map outputs can be regenerated and input/output data is protected by the distributed file system, losing a local disk rarely threatens the overall job.

What’s interesting is that MapReduce doesn’t just prepare for machines that fail.

It also prepares for machines that survive.

In large clusters, some workers inevitably become slower than others. A handful of slow machines can delay an entire job, even when everything else is running perfectly.

To address this, MapReduce introduced one of the paper’s most influential ideas: speculative execution.

Near the end of a job, backup copies of slow tasks are launched on other machines. Whichever copy finishes first wins, while the remaining copies are discarded.

It’s a deceptively simple optimization, but it attacks a problem that still exists in modern distributed systems: tail latency. Often, the slowest machine determines when the entire job completes.

The one significant weakness in the design is the master node.

MapReduce relies on a single master to track task assignments, worker status, and job metadata. If that master fails, the job fails with it. The paper suggests checkpointing as a future improvement, but the implementation described does not fully eliminate this single point of failure.

What makes the design remarkable isn’t that it prevents failures.

It’s that it assumes failures are inevitable and builds recovery directly into the execution model.

That’s a philosophy that continues to influence distributed systems more than two decades later.

What Breaks First?

One of the things I appreciate most about the paper is how honest it is about scalability bottlenecks.

When engineers think about scaling a system, the first question is often:

Will we run out of CPU?

For MapReduce, the answer is usually no.

Most large jobs spend far more time moving data than computing on it. The computation itself is often the easy part. Coordinating thousands of machines and shuffling massive amounts of data across the network is where the real cost lies.

Memory can become a bottleneck, particularly on reducers. When intermediate datasets exceed available RAM, reducers spill data to disk and continue processing. Performance suffers, but the system remains functional.

The first major bottleneck is usually the network.

The shuffle phase dominates many workloads because data produced by map tasks must be transferred across the cluster before reducers can begin their work. In many ways, the architecture is optimized around a single goal: minimizing unnecessary network traffic.

Storage is another pressure point.

Large sorting jobs generate enormous amounts of disk I/O. Map workers continuously spill intermediate files to disk, while reducers read, merge, sort, and write large volumes of data. The system survives these workloads, but disk activity often becomes a limiting factor.

Even the control plane has its limits.

The master node tracks metadata for every map and reduce task using an O(M × R) data structure, where M is the number of map tasks and R is the number of reduce tasks. While this worked well at Google’s scale, it hints at a future bottleneck as clusters and workloads continue to grow.

What I find particularly interesting is that none of these bottlenecks are primarily about computation.

The paper repeatedly reinforces a lesson that still applies to modern distributed systems:

The hardest resource at scale is often not compute. It’s coordination.

Network traffic, data movement, scheduling decisions, and tail latency frequently matter far more than raw processing power.

The Numbers That Made People Pay Attention

What made this paper particularly convincing was that it included real production results.

A representative configuration included:

• 2,000 worker machines

• 200,000 map tasks

• 5,000 reduce tasks

Google demonstrated:

• A 1 TB grep workload completed in roughly 150 seconds

• A 1 TB sort completed in roughly 891 seconds

More importantly, they deliberately killed hundreds of machines during execution.

The result?

The sort job slowed by only about 5%.

That number captures the paper’s real thesis:

Failure is not an exception. Failure is a scheduling event.

Why MapReduce Changed the Industry

Many papers introduce useful ideas.

Very few change how an entire industry thinks about a problem.

MapReduce did.

Before MapReduce, large-scale data processing was often treated as an application problem. Every team that wanted to process massive datasets had to solve the same distributed systems challenges: task scheduling, fault tolerance, load balancing, machine failures, and data movement.

MapReduce changed the boundary.

Instead of asking application developers to solve those problems repeatedly, it pushed them into the platform. Developers described the computation they wanted to perform. The infrastructure handled everything else.

That shift turned out to be far more important than the Map and Reduce functions themselves.

Inside Google, teams could process enormous datasets without reinventing distributed infrastructure for every new application. Outside Google, Hadoop brought the same programming model to the broader industry and made large-scale data processing accessible to organizations that didn’t have Google’s engineering resources.

The influence of the paper extended far beyond Hadoop.

It established data locality as a first-class design principle. It popularized the shuffle phase as a fundamental building block of distributed computation. More importantly, it demonstrated that infrastructure could absorb complexity that applications previously managed themselves.

Even the systems that eventually replaced MapReduce were shaped by it.

Spark, for example, can be viewed partly as a response to MapReduce’s limitations. Excessive disk materialization, high latency, and weak support for iterative workloads motivated a new generation of data processing frameworks.

But Spark only makes sense in a world that MapReduce created.

The conversation shifted from:

How do we build distributed data processing systems?

to:

How do we build a better version of MapReduce?

That’s the mark of a truly influential idea.

The strongest evidence that MapReduce succeeded is that most engineers no longer think about the problems it solved.

Today, we expect platforms to recover from failures automatically, scale across clusters, rebalance slow workers, and schedule work intelligently. Those capabilities feel obvious because modern infrastructure provides them by default.

In 2004, they weren’t infrastructure problems.

They were application problems.

MapReduce helped change that.

More than twenty years later, most distributed data platforms are still following the blueprint it introduced: hide the complexity of distributed systems behind a simple programming model and let the platform do the heavy lifting.

My Learnings

The most valuable systems ideas often look like restrictions, not freedoms.

A narrow programming model can create extraordinary leverage if it allows the platform to absorb complexity.

After working on inventory systems that process hundreds of thousands of events per second, I’ve come to appreciate how true this observation is. CPU is usually easy to buy. Coordination isn’t. Network traffic, retries, partition ownership, and tail latency create far more production incidents than raw compute shortages.Network movement, scheduling decisions, data placement, and tail latency frequently matter more than raw processing power.

Most importantly, MapReduce shows that great infrastructure doesn’t necessarily make systems more powerful.

Sometimes it makes them simpler.

Google took one of the hardest problems in software engineering and reduced it to two functions.

More than twenty years later, most distributed data platforms are still following that blueprint.

The interesting question isn’t how Google built MapReduce in 2004. It’s how we would build it today. In the next post, I’ll walk through the architecture I’d choose in 2026, the components I’d replace, and the tradeoffs I’d make differently. Subscribe if you’d like to follow along.

Counting sounds easy, right?
You add one number to another, maybe you keep a counter in a database or increment a value in memory. Simple.

But at Netflix’s scale, even something as basic as “counting” becomes a serious engineering challenge.

When you have millions of users streaming content every minute across thousands of servers distributed around the world, the act of “adding one more view” to a counter quickly turns into a distributed systems problem.

Let’s dive into what the problem was, why traditional methods fail, and what we can learn from how Netflix approached it.

The Problem: Counting at Netflix Scale

Imagine you are tracking how many people are watching Stranger Things right now.

Every time someone presses play, one counter somewhere needs to go up by one.
Now imagine this happening across hundreds of data centers, each handling its own share of users.

That’s millions of increments per second.

If you had just one database where all those servers sent an “increment” request (INCR), you would have:

• A massive bottleneck with too many requests hitting the same database

• Network delays and contention

• Possible data loss if that central database crashes

And worst of all, your “total view count” could become inconsistent or outdated.

Netflix engineers needed a way to count accurately, fast, and reliably, even when data was spread across thousands of machines.

Follow me on Linkedin

Why Traditional Counting Doesn’t Work

Let’s take the simple approach most of us use in smaller systems:

count = count + 1

It works beautifully on a single machine. But the moment you distribute this logic across hundreds of servers, problems begin.

1. Concurrency Issues: Two servers might try to update the same counter at the same time, causing incorrect totals

2. Latency: The more servers that try to reach the central counter, the slower it gets

3. Failure Handling: What happens if one server crashes mid-update

4. Scalability: A single counter database cannot handle billions of updates per second

Netflix realized that instead of making one machine handle all counts, they needed each machine to count locally, and then combine those counts efficiently.

Netflix’s Solution: Decentralized, Event-Driven Counting

Here’s the key insight:
Do not count everything in one place. Let every server count independently and merge those results later.

This is called decentralized counting.

Each server does its own small counting for a short time window. Then, it sends those local totals, called deltas, to a central pipeline like Kafka.

Think of it like this:

• Every store branch counts how many customers visited today

• At the end of the day, each branch sends their total number to the head office

• The head office adds all those branch numbers together to get the company-wide total

Netflix does exactly this, but continuously and in real time.

How It Works Step by Step

1. Local Counting
   Each Netflix microservice instance keeps a small in-memory counter. It increments locally whenever a new event happens, like a stream start or a like

2. Batching
   Instead of sending every single increment, each instance batches its counts for a short period, for example every few seconds

3. Publishing
   These batched counts are published to Kafka, Netflix’s distributed event streaming platform

4. Aggregation
   A specialized service consumes these delta events and aggregates them by keys, for example show ID, region, or time window

5. Storage
   The final aggregated counts are stored in a scalable database like Cassandra or Redis, which can handle huge read and write volumes

Handling Delays, Duplicates, and Failures

In distributed systems, you cannot assume everything arrives perfectly on time or only once.

Some deltas might get delayed due to network lag, and some might even get sent twice during retries.

Netflix solved this by designing their system to be idempotent, meaning even if the same update is applied twice, the result remains correct.

Each delta has:

• A unique ID to detect duplicates

• A timestamp window to know when the counts happened

• The increment value for how much to add

Even if Kafka replays an event or a message is retried, the aggregator can safely ignore duplicates and apply counts correctly.

This ensures that the final global count is accurate, even if updates come late or in random order.

In-Depth Architecture

For those who’ve stayed with us this far, let’s dive deeper into the architecture. After evaluating multiple approaches, Netflix ultimately adopted a hybrid, event-driven solution for distributed counting. This final design strikes a careful balance between low-latency reads, high throughput, and strong durability.

1. Event Logging in TimeSeries Store

Every counter increment is treated as an immutable event and ingested into Netflix’s TimeSeries abstraction, which acts as the event store. Each event record includes:

• event_time: Timestamp of the increment

• event_id: Unique identifier for idempotency

• counter_name and namespace: To identify which counter it belongs to

• delta: The increment value (positive or negative)

The TimeSeries abstraction, typically backed by Cassandra, ensures high availability, partition tolerance, and fast writes. Events are partitioned using time_bucket and event_bucket columns to prevent wide partitions that could degrade read performance.

Benefits:

• Immutable events prevent accidental data loss

• Built-in retention policies automatically purge old events

• Supports safe retries due to idempotency keys

Handling Wide Partitions: The time_bucket and event_bucket columns play a crucial role in breaking up a wide partition, preventing high-throughput counter events from overwhelming a given partition. For more information regarding this, refer to this blog.

2. Background Aggregation (Rollups)

Reading individual events for each request is expensive. To solve this, Netflix performs continuous aggregation of events in the background.

Key points:

• Rollup windows: Aggregation occurs within a defined time window to ensure consistency

• lastRollupTs: This represents the most recent time when the counter value was last aggregated. For a counter being operated for the first time, this timestamp defaults to a reasonable time in the past.

• Immutable window: Aggregation can only occur safely within an immutable window that is no longer receiving counter events. The “acceptLimit” parameter of the TimeSeries Abstraction plays a crucial role here, as it rejects incoming events with timestamps beyond this limit. During aggregations, this window is pushed slightly further back to account for clock skews.

This results in a rollup count, which represents the sum of all events in the aggregation window.

3. Rollup Storage

Aggregated counts are stored in a persistent Rollup store, again often Cassandra, with one table per dataset.

LastWriteTs: Every time a given counter receives a write, they also log a last-write-timestamp as a columnar update in this table. This is done using Cassandra’s USING TIMESTAMP feature to predictably apply the Last-Write-Win (LWW) semantics. This timestamp is the same as the event_time for the event.

• Each row contains: counter_name, lastRollupCount, lastRollupTs

• LastWriteTs is recorded for every write to track whether the counter has new events not yet aggregated

  

• Subsequent aggregations continue from the last checkpoint

Benefit: Efficient reads with only incremental aggregation needed

4. Caching Layer (EVCache)

To provide low-latency reads, the latest aggregated counts are cached in EVCache.

• Cached value = {lastRollupCount, lastRollupTs}

  

• Reads return the cached value, with a slight acceptable lag

• Triggers a rollup in the background if needed to catch up on events

This allows single-digit millisecond read latencies even at massive scale.

5. Event-Driven Rollup Pipeline

Rollups are triggered using a lightweight rollup event per counter.

rollupEvent: {
  “namespace”: “my_dataset”,
  “counter”: “counter123”
}

This event doesn’t contain the actual number that was added to the counter. Instead, it just tells the Rollup server that this counter was updated and needs to be processed. By sending these lightweight “update signals,” the system knows exactly which counters to aggregate, so it doesn’t have to go through all the events in the database every time.

In-Memory Rollup Queues: Each Rollup server keeps several queues in memory to collect rollup events and process multiple counters at the same time. Using in-memory queues makes the system simpler, cheaper, and easier to adjust if they need more or fewer queues. The trade-off is that if a server crashes, some rollup events in memory might be lost.

Minimize Duplicate Effort: Netflix uses a fast non-cryptographic hash like XXHash to ensure that the same set of counters end up on the same queue. Further, Netflix tries to minimize the amount of duplicate aggregation work by having a separate rollup stack that chooses to run fewer beefier instances.

Dynamic Batching: The Rollup server dynamically adjusts the number of time partitions that need to be scanned based on cardinality of counters in order to prevent overwhelming the underlying store with many parallel read requests.

Adaptive Back-Pressure: Each consumer waits for one batch of counters to finish processing before starting the next batch. If the previous batch took longer, the system waits a bit more before starting the next one. This helps prevent the underlying TimeSeries database from getting overloaded with too many requests at once.

Key Considerations:

• Aggregation occurs without distributed locks, using immutable windows to ensure correctness

• Counters are re-queued if new writes occur, ensuring convergence

• Stale counts self-remediate during the next read-triggered rollup

6. Idempotency and Safety

Each increment event includes event_id + event_time as a unique key. This ensures:

• Safe retries without over-counting

• Accurate aggregation across distributed consumers

• Reliable reset operations without race conditions

7. Experimental “Accurate” Counter

For use cases requiring near real-time accurate counts:

• The delta between the last rollup and current events is computed in real-time during read

• currentAccurateCount = lastRollupCount + delta

• Batched processing is still applied to avoid overwhelming the underlying TimeSeries store

This allows clients to see almost real-time counts, while the system maintains high throughput for millions of counters globally.

Putting It All Together

The final architecture combines:

1. Event logging: Every increment is stored as an immutable event in TimeSeries

2. Background aggregation: Continuous rollups ensure efficient reads

3. Rollup storage: Persistent storage for aggregated counts

4. Caching: EVCache for near-instant reads

5. Event-driven pipeline: Efficient, parallel, and idempotent rollups

6. Optional real-time delta: For “accurate” counter reads

This combination of event logging, aggregation, caching, and idempotency is what allows Netflix to:

• Handle millions of simultaneous increments

• Provide near real-time counts

• Avoid overloading any single server

• Maintain global durability and reliability

Key Design Takeaways

Netflix’s distributed counting system is a perfect example of how simple ideas need solid architecture at scale.

1. Push computation closer to data
   Let each node do part of the work instead of relying on one big system

2. Batch operations
   Do not send every single event. Aggregate small chunks to reduce load

3. Design for eventual consistency
   It is okay if data takes a few seconds to settle as long as it converges correctly

4. Use idempotent events
   Avoid double-counting when messages are retried

5. Stream, don’t store
   Use event-driven architecture for scalability and resilience

Reference: Netflix’s Distributed Counter Abstraction

Final Thoughts

I love how Netflix engineers turn “simple problems” into lessons in elegant system design.

Distributed counting teaches us that even a basic count++ can become complex when scaled across thousands of machines. With the right architecture, batching, and event streaming, it becomes not just manageable but efficient.

It is a reminder that great engineering is about scaling simplicity, not complexity.

Next time you design a system, ask yourself:

What happens when this “simple” feature needs to scale to a billion users?

That is where system design truly begins.

Today, we’re diving into a fascinating story from Netflix’s engineering team, a story that shows how even world-class systems can hit scaling walls when the use case evolves, and how a smart architectural rethink can unlock massive performance gains. In Netflix’s case, the result was a 100× improvement.

In this post, we’ll cover:

• The challenge Netflix faced with its Maestro workflow engine, and why tasks that used to run fine started slowing down

• The architecture behind the slowdown, including stateless workers, polling queues, and database overhead

• The redesign that supercharged Maestro, introducing stateful actors, in-memory workflows, task grouping, and event-driven execution

• The results and takeaways, showing how a well-thought-out system design can make even massive workflows snappy

To set the stage, imagine if every task in your system took an extra 5 seconds to start. Not a big deal, right?

Now multiply that by millions of executions a day. That’s the hidden performance tax Netflix uncovered in Maestro. And their solution is a brilliant example of architecture beating brute force.

Reference: Netflix Tech Blog — 100× Faster: How We Supercharged Netflix Maestro’s Workflow Engine

The Problem Netflix Faced

Netflix has a system called Maestro. Think of Maestro as the “task manager” that controls when and how Netflix runs thousands of background jobs — like moving data, training ML models, generating recommendations, and more.

It’s like a conductor of an orchestra, telling each musician (task) when to start, stop, or wait.

For years, Maestro worked perfectly because most jobs ran only once a day or once every few hours. A few seconds of delay didn’t matter.

But then Netflix started using Maestro for real-time use cases — like ads, games, or quick data pipelines that run every few minutes or even seconds. Suddenly, they noticed:

Every task took an extra 5–10 seconds just to start.

That was overhead inside Maestro itself — before the actual work even began. This made fast workflows feel slow and frustrated developers who had to wait on every change.

Why It Was Slow

Originally, Maestro’s design worked like this:

• A central system put tasks into a queue (like a to-do list).

• Stateless workers (servers with no memory of previous tasks) kept polling the queue:

“Do you have something for me?”

• When they got something, they loaded task details from a database, did the work, then wrote results back.

This worked great at scale, but:

• Polling every few seconds caused delays.

• Repeatedly loading and saving state slowed things further.

• Using multiple systems (queues, databases, workers) added more lag.

It’s like every time you wanted to start a task, you had to:

1. Call your manager

2. Check your email

3. Unlock your computer

4. Then finally start working.

This overhead is fine once a day, but crippling if you’re doing it every minute.

What Netflix Did to Fix It

Netflix didn’t just tweak Maestro, they redesigned its brain for real-time responsiveness.

Here’s how they did it:

1. Stateful Actors: Giving Each Task a “Memory”

Instead of stateless workers constantly asking “what next?”, each workflow or task got its own actor, a lightweight in-memory brain that:

• Lives in memory (JVM)

• Knows its next steps

• Reacts instantly to events

• Avoids repetitive database reads

Result: Tasks now start in milliseconds, not seconds.

2. Keep Data in Memory, Sync in Background

They used the database only as a backup, not as the main source for every step.
The real-time info lives in memory, super fast.

Think of it like working on a Google Doc in real-time, but everything is saved in the background. You don’t hit “Save” every time.

3. Group Tasks Logically

They split workflows into “groups” and assigned each group to a specific server.
This way, related tasks stay on the same machine, less shuffling around, faster execution.

And if a server goes down, another can take over the group.

Think of it like assigning each teacher a set of students. If one teacher is sick, another takes over their class. But students don’t keep running between classrooms unnecessarily.

4. Removed the Middleman Queue

Instead of using an external queue where workers kept asking for tasks, they:

• Used in-memory queues for instant communication inside the system, and

• A small database-backed mechanism to make sure tasks don’t get lost.

Like passing a note directly to a colleague instead of sending an email and waiting for them to check their inbox.

5. Gradual Migration with Real Testing

They built a testing framework to run real production workflows in parallel on the new system to catch bugs.
Then they moved users over gradually.

Like test-driving a new engine before swapping it into the airplane mid-flight.

The Results

The startup delay per task went from 5 seconds → 50 milliseconds.
That’s a 100× speedup.

• Developers no longer have to wait between iterations.

• Real-time workflows run smoothly.

• Maestro remains scalable but is now fast too.

Why This Matters (Even If You’re Not Netflix)

Many engineering teams optimize for scale, but ignore latency inside the system.

Polling, stateless workers, and external queues are simple to build, but add hidden costs.

By shifting to:

• Event-driven designs

• In-memory state

• Actor models

…you can unlock massive speed gains without simply throwing more servers at the problem.

Netflix’s journey is a reminder: sometimes, true performance comes from rethinking architecture, not just optimizing code.

Conclusion

Netflix’s Maestro evolution is more than just a speed hack. It’s a masterclass in system design.

When workloads change, the architecture that once worked well can become the bottleneck. By moving from a centralized, polling-based design to a decentralized, actor-driven model, Netflix made Maestro real-time ready.

For the rest of us, the takeaway is clear:
✅ Don’t underestimate internal latency.
✅ Event-driven + in-memory approaches can deliver big wins.
✅ Architectural redesign can unlock performance leaps that brute force never will.

I was deep into building a tricky feature when a junior engineer popped by. “Hey, can you help me debug this?” I thought, five minutes won’t hurt. Fifteen minutes later, I finally got back to my code, only to be dragged into an ad-hoc meeting about some random issue. While I was on the call, Slack exploded with messages from other team members. By the time I returned to my code, I couldn’t even remember what I was working on. My focus was gone.

And that’s when it hit me: I wasn’t multitasking. I was constantly switching contexts, and it was silently sabotaging my productivity.

The Real Problem: Context Switching

The human brain isn’t wired to handle two demanding tasks at the same time. Every time we switch from coding to answering questions, to attending a meeting, to replying to Slack, our focus fractures. Research suggests it can take around 25 minutes to regain full attention after an interruption.

The cost isn’t just time. Constant context switching leads to mental fatigue, slower problem-solving, and less creativity. And in a fast-paced environment, with ad-hoc meetings, junior engineers needing guidance, and endless pings, this can quietly eat away at your entire day.

How I Fixed It

Once I realized context switching was the culprit, I started experimenting with ways to protect my attention. Here’s what worked for me:

1. Prioritize Ruthlessly

Each morning, I pick 1–2 tasks that truly matter and treat them like non-negotiable appointments. Emails, bug fixes, and minor questions come later. Saying “not now” felt uncomfortable at first, but finishing meaningful work made it worth it.

2. Batch Interruptions

Instead of answering Slack messages and emails the moment they arrive, I created specific time blocks for them. Junior engineers learned I had “collaboration meetings” for questions and problem solving. I could focus without guilt, and the team still got the help they needed.

3. Time-Block Deep Work

I reserved chunks of time purely for coding or writing and blocked them on my calendar as “Focus mode“. Notifications off, door closed, deep work mode on. For the first time, I could actually get into flow and stay there.

4. Minimize Distractions

Phone on silent, unnecessary tabs closed, notifications off. These small boundaries prevented tiny interruptions from breaking my focus and stealing 15–20 minutes at a time.

5. Single-Task, Fully

I stopped juggling multiple tasks. One task, full attention. Ironically, I finished things faster and with better quality. Letting go of the “always busy” mindset was surprisingly liberating.

6. Build a Buffer for Chaos

Unexpected things still happen, urgent meetings, production bugs, random questions. But leaving small gaps between tasks absorbed the chaos instead of letting it derail the day. My stress levels dropped, and I stayed in control.

Conclusion

Multitasking might feel like a superpower, but it’s a trap. Context switching is the silent productivity killer that drains energy, focus, and creativity. Once I started protecting my attention with these habits, I finished tasks faster, felt more energized, and finally regained control over my workday.

Focus on one thing at a time, protect your flow, and let real productivity happen.

A few years ago, I found myself in a situation at work that taught me one of the most valuable lessons of my career.

I was leading a project that had tight deadlines. A colleague kept missing updates I needed from them, which started delaying my work. At first, I stayed silent—I didn’t want to come across as pushy. But the silence only made things worse. My manager asked why timelines were slipping, and I realized I was taking the hit for something I hadn’t spoken up about.

That was my turning point. I had to learn how to express my needs clearly without sounding rude. Instead of saying, “You never give me updates,” I started saying, “I need the report by Wednesday so I can finalize the deck by Friday.” That shift—from vague frustration to clear, respectful communication—changed everything. My colleague responded better, deadlines stopped slipping, and I felt a sense of control return to my work.

That’s the power of assertiveness. It’s not aggression, and it’s not silence. It’s about communicating in a way that values both your perspective and the other person’s.

In today’s fast-paced work environments, technical skills alone aren’t enough to thrive. Whether you’re presenting ideas to stakeholders, negotiating deadlines, or collaborating with peers, how you communicate makes a world of difference.

1. Understand Assertiveness (and How It Differs from Aggression)

• Aggressive communication pushes your perspective at the cost of others.

• Passive communication avoids conflict but often leaves you unheard.

• Assertive communication strikes the balance: it’s direct, clear, respectful, and solution-focused.

For example, instead of writing:
“You never respond to my emails.”
Say:
“I had sent you an email last week regarding the project estimation. Could you share the details by today so I can finalize the report?”

This way, you’re stating facts, setting expectations, and staying professional

2. Use the Pyramid Principle for Clarity

When making your point, avoid burying the key message under unnecessary details. Instead, follow a top-down structure:

• Answer (Assertion): State the most important idea upfront.

• Argument: Support it with logical reasoning.

• Evidence: Back it up with data.

For example, instead of:
“We should consider using the new tool because it might help some processes, though I know there’s resistance.”
Say:
“The new tool can save employees 10 hours a week (Answer). It automates repetitive workflows (Argument), and three other teams have already reported efficiency gains (Evidence).”

3. Master Assertiveness Tools

The second deck introduces practical conversational techniques you can use when facing resistance.

• Broken Record: Calmly repeat your request without getting sidetracked.

• Fogging: Acknowledge the valid part of criticism without absorbing exaggerations.

• Negative Inquiry: Ask questions to clarify vague criticism (e.g., “Could you tell me what you’d like improved in my emails?”).

• Positive No (Yes-No-Yes): Say no respectfully by affirming the other person’s needs, stating your refusal, and ending with a constructive alternative.

Example:
“I understand this feature is important to you (Yes). However, we can’t deliver all ten changes in this release (No). What we can do is prioritize the top three and revisit the rest next sprint (Yes).”

4. Be Specific and Sensitive

Generalizations like “You’re always late” or “This timeline is impossible” spark defensiveness. Assertiveness relies on specific facts and empathy:

• Instead of “He is impatient”, say “He asked for updates five times in two hours yesterday, which disrupted the flow.”

• Instead of “The client is demanding”, say “The client requested five design changes within two days.”

5. Say “No” Without Burning Bridges

Many professionals struggle with saying no because they fear being seen as uncooperative. But over-committing leads to stress and poor delivery.

The Yes-No-Yes approach helps:

• Yes: Show understanding (“I know this project is important to you”).

• No: State your boundary firmly (“We can’t add these changes to this release”).

• Yes: Offer a constructive way forward (“We can include them in the next sprint”).

6. Match Your Communication to Context

Assertiveness also means being mindful of when and how you communicate:

• Avoid sending emotional or confrontational emails—opt for a call or face-to-face chat.

• Don’t schedule heavy discussions at times when people are mentally checked out (like late Friday evenings).

• Keep your language human and jargon-free to ensure clarity

7. Build Influence Without Authority

Being assertive isn’t just about voicing your needs—it’s about influencing others while respecting their perspectives. Use the six principles of persuasion to strengthen your stance:

• Reciprocity: Offer help before asking for it.

• Authority: Share expertise to earn credibility.

• Social Proof: Highlight how others have adopted your idea.

• Scarcity: Emphasize opportunities that may not last.

• Consistency: Link your request to past commitments.

• Liking: Build rapport so people want to say yes

Final Thoughts

Assertiveness is not about being the loudest voice in the room. It’s about being clear, confident, and considerate. When you communicate assertively, you:

• Earn respect without creating resentment

• Manage conflicts productively

• Influence decisions even without formal authority

• Create stronger, more collaborative workplaces

Start small—whether it’s rewriting one email to be more assertive, practicing a Positive No in your next meeting, or using the pyramid principle in your presentations. Over time, assertiveness becomes less of a skill you practice and more of a professional habit that elevates your impact.

A few years back, I was working on a system that sent welcome emails to new users during sign-up. The design was simple: a user performed an action, an event was published to a queue, a consumer picked it up and called the SendGrid API to deliver the email.

One day, we hit an unexpected snag. Our consumer was calling the SendGrid API just fine, but due to a small bug, it never marked the event as processed (acknowledged). The queue assumed the event wasn’t handled, so after the ack deadline expired, it redelivered the same event. The consumer retried the call, SendGrid happily sent the email again, and this loop kept repeating.

To make matters worse, we only had a single consumer instance. That one stuck event essentially blocked the entire queue. What looked like a harmless bug turned into dozens of duplicate emails and a completely clogged pipeline.

This kind of issue is known as the “poison pill” problem: one bad or unacknowledged message keeps coming back, poisoning the queue, and preventing healthy messages from being processed.

From Poison Pills to Dead Letter Queues

In this post, we’ll see how to stop a “poison pill” from poisoning our system. But before diving into the technical details, let’s imagine a simple real-world queue.

Imagine you’re at a bank where people line up to deposit checks. Each person hands their check to the cashier, who processes it one by one.

Now, if someone hands in a torn or illegible check, the cashier can’t process it. If they just keep holding on to that bad check, the entire line behind them comes to a halt.

A smarter system would be: put that bad check aside in a separate tray, let the rest of the line move forward, and later have a manager review the bad check to see what went wrong.

That’s exactly what a Dead Letter Queue (DLQ) does:

• If a message in the queue can’t be processed (like the invalid check), it gets moved aside into a special holding area (the DLQ).

• The rest of the messages keep flowing smoothly.

• Later, engineers can look at the failed messages in the DLQ, fix them, and reprocess them if needed.

Designing the Dead Letter Queue

Now that we know what a Dead Letter Queue (DLQ) is, the real question is: how do we actually implement one?

Before jumping into code, it’s important to step back and design the mechanics carefully. A DLQ isn’t just a “dumping ground” for failed messages — it’s a deliberate safety net that requires us to make a few key decisions:

1. Identify failure modes

   • What kinds of errors can occur in your system?

   • Are they transient (like a temporary network blip or API timeout) or permanent (like malformed data or a missing required field)?

2. Define failure handling strategy

   • Which types of errors should trigger a retry?

   • Which should immediately route the message to the DLQ?

3. Set retry policies

   • How many times should a message be retried before we officially “give up”?

   • Should retries follow exponential backoff, fixed intervals, or custom logic?

4. Decide on metadata for DLQ messages

   • What extra information should we attach when moving a message to the DLQ? (e.g., error type, stack trace, retry count, correlation ID).

   • This metadata is crucial for debugging and reprocessing later.

5. Establish a “poison message” threshold

   • At what point do we stop retrying and mark a message as poisonous?

   • Once it’s in the DLQ, we don’t want it re-entering the main processing loop automatically and causing the same failure cycle.

Key Failure Modes and DLQ Strategies

Once we’ve defined what a DLQ should capture, the next step is to map out the different failure modes that can occur in a Pub/Sub-based system. Each type of failure needs its own handling strategy so that messages don’t get stuck in endless loops.

1. Consumer Failure Modes

When a Pub/Sub consumer cannot process a message:

• Ack deadline: If the consumer doesn’t acknowledge within the ack deadline (default 10s, extendable to 600s), Pub/Sub assumes failure and redelivers the message.

• Retries: Pub/Sub keeps retrying until the message is acknowledged or until it reaches the maximum delivery attempts (if a DLQ is configured).

• Poison messages: Without a DLQ, one bad message can cycle forever, blocking or delaying other messages.

Strategies

• Use idempotent consumers to make retries safe and avoid duplicate side effects (e.g., sending duplicate emails).

• Implement ack deadline extensions for long-running jobs.

• Configure a DLQ with max delivery attempts to prevent poison loops.

2. API Call Failures Inside Consumer

A common failure scenario is when your consumer depends on an external API (e.g., SendGrid, payment gateway) and that API either times out or returns a 5xx error.

Best practices

• Use exponential backoff with jitter to retry API calls locally before giving up.

• Apply circuit breakers to stop hammering unstable dependencies.

• Ensure all external calls use idempotency keys so retries don’t cause duplicates.

• If retries exceed the configured limit, route the message to DLQ.

3. Non-Recoverable Application Errors (e.g., NPE)

Not all failures are retryable. Some represent logic or data issues that will never succeed no matter how many times you retry:

• Schema mismatch (e.g., invalid JSON).

• Missing required fields in the payload.

• Null pointer exceptions or bugs in business logic.

In such cases:

• A DLQ helps capture the bad payload for later analysis.

• But fixing the code (or correcting the data source) is mandatory before replaying.

• Use a DB error log table to persist stack traces, consumer version, and raw payloads for debugging.

4. Pub/Sub DLQ vs. Error Log Table in a Database

When deciding how to capture failed messages, teams usually debate between using:

1. Pub/Sub Dead Letter Queues (DLQs)

2. A database-backed error log table

Both approaches have strengths and trade-offs.

Pub/Sub DLQ (Dead Letter Queue)

Pros

• Automatically supported by Pub/Sub — easy to configure.

• Handles high-throughput failures without manual scaling.

• Allows attaching message attributes as metadata.

• Messages can be re-subscribed to and reprocessed later.

Cons

• Limited querying/filtering — replaying or analyzing requires additional tooling.

• Costs grow with message volume and retention.

Error Log Table (e.g., Cloud SQL / PostgreSQL)

Pros

• Full control: you can store payload + metadata + stack trace + retry count + consumer version.

• Easy to query/filter (e.g., “show me all errors for consumer v2.1 in the last 24h”).

• Can enrich with business metadata (user ID, category, severity).

Cons

• Requires schema design, storage management, and indexing.

• Not built for massive throughput unless carefully optimized.

• Reprocessing requires custom scripts/tools.

Rule of thumb: Use Pub/Sub DLQ for operational resilience and quick retries, and use an error log DB for observability and analytics.

5. Republishing / Re-consuming Events

A DLQ is only useful if you can reprocess messages safely. Here’s how it works in both approaches:

From Pub/Sub Dead Letter Topic

• Messages sit in a separate subscription.

• Use gcloud pubsub subscriptions pull or a custom consumer to fetch them.

• After analysis/fix, republish back to the main topic.

• Use idempotency keys to avoid re-triggering duplicates.

From Error Log Table

• Query failed rows (e.g., based on error category or timestamp).

• Batch re-publish messages via a script or tool.

• Ensure the re-publish process:

  • Updates a reprocessed_at timestamp.

  • Prevents double-sending if the same row is picked up again.

6. Hybrid Approach

In practice, many production systems use a hybrid strategy:

• Pub/Sub DLQ handles high-throughput retries and keeps the main pipeline healthy.

• Error Log Table provides deep visibility into why failures occurred, enriched with stack traces, error categories, and metadata for debugging.

Recommended flow:

Explaining the System Flow

This flowchart represents the end-to-end lifecycle of a message in a Google Pub/Sub–based system with Dead Letter Queues (DLQs) and a Parked Letter Queue (PLQ) or the error table.

Let’s walk through each step:

1. Message Entry – Pub/Sub Topic

• Everything begins with the Pub/Sub Topic.

• Producers publish events here (e.g., “user signup,” “order created”).

• Messages are then delivered to one or more Consumers for processing.

2. Consumer Processing

• The Consumer is the application logic that processes each message.

• Example: validating the payload, calling an external API, saving to DB, etc.

• Two outcomes are possible:

  • Success: Message is acknowledged and marked as Processing Complete.

  • Failure after retries: Message is handed off to the Dead Letter Topic (DLQ).

3. Dead Letter Topic (DLQ) – Handling Transient Errors

• The DLQ captures messages that couldn’t be processed even after retries.

• These are typically transient issues, such as:

  • External API outages

  • Temporary DB lock

  • Network timeouts

• To avoid losing data, we don’t discard these messages immediately. Instead:

  • A Reprocessor Service picks them up later and republishes them to the main topic for another attempt.

  • If successful, they continue as normal through the pipeline.

4. Parked Letter Queue (PLQ) – Handling Persistent Failures

• If a DLQ message still fails even after reprocessing, it moves to the Parked Letter Queue (PLQ) or in this case, an error table.

• The PLQ is the “quarantine area” for persistent or poison messages that won’t succeed automatically.

• We can enhance the DLQ reprocessor to attach a custom attribute indicating how many times a message has landed in the DLQ. This way, the consumer application can use that metadata to decide when to route the message to the error log table.

• Example causes:

  • Schema mismatch (e.g., missing required field)

  • Invalid payload (malformed JSON, bad encoding)

  • Application bug (NullPointerException)

5. Persistence and Analytics

• Every message that enters the PLQ is persisted in an Error Log Table.

• This table contains:

  • Raw payload

  • Error category

  • Stack trace or exception details

  • Retry count and consumer version

• From here, the data flows into BigQuery for analytics and dashboards.

  • Teams can analyze trends (e.g., “What % of errors are API failures vs. schema mismatches?”).

  • Helps prioritize fixes and monitor long-term stability.

6. Manual Intervention

• Engineers can manually inspect PLQ messages.

• Based on investigation, they have two options:

  • Reprocess: Publish back into the Pub/Sub Topic once the root cause is fixed.

  • Discard: If the message is invalid and cannot be salvaged.

• This ensures no data is lost silently and every failure path has a resolution.

Implementation: Building a True DLQ Pattern with Google Pub/Sub

Overview

This basic implementation would demonstrate a true Dead Letter Queue pattern using Google Pub/Sub, where:

• ALL failures go to DLQ first (not directly to PLQ)

• DLQ Reprocessor adds retry count metadata

• Main Consumer makes PLQ decisions based on retry count

• Manual intervention available via web dashboard

Key Principle

Message → Consumer → Fail → DLQ → Reprocessor → Add Retry Count → Main Topic → Consumer → Check Retry Count → If ≥3 → PLQ (Database)

Prefer to skip the explanation and go straight to the implementation? The code is available here.

1. Producer Service

Location: producer-service/app.py

The producer publishes messages to the main topic:

@app.route('/publish', methods=['POST'])
def publish_message():
    """Publish a single message to the main topic"""
    try:
        data = request.get_json()
        
        # Add metadata
        message_data = {
            **data,
            "timestamp": datetime.now().isoformat(),
            "producer_version": "1.0"
        }
        
        # Publish to main topic
        message_id = pubsub_manager.publish_message(MAIN_TOPIC, message_data)
        
        return jsonify({
            "message_id": message_id,
            "status": "published", 
            "timestamp": datetime.now().isoformat(),
            "topic": MAIN_TOPIC
        })

Key Features:

• REST API for message publishing

• Automatic timestamp metadata

• Sample message generation for testing

2. Consumer Service (Core DLQ Logic)

Location: consumer-service/app.py

This is the heart of the DLQ pattern implementation:

def process_message(self, message_data: Dict[Any, Any], message_id: str, 
                   correlation_id: str = None, dlq_retry_count: int = 0) -> bool:
    """Process message with DLQ retry count awareness"""
    try:
        logger.info(f"Processing message {message_id}, type: {message_data.get('type')}, "
                   f"dlq_retry_count: {dlq_retry_count}")
        
        # KEY LOGIC: Check if message exceeded DLQ retry limit
        if dlq_retry_count >= 3:  # Max DLQ retries
            logger.info(f"Message {message_id} exceeded DLQ retry limit, sending to PLQ")
            self._log_to_plq(
                message_data, message_id, 'dlq_max_retries_exceeded', 
                f'Message failed after {dlq_retry_count} DLQ retries',
                correlation_id=correlation_id, dlq_retry_count=dlq_retry_count
            )
            return True  # Acknowledge the message (handled via PLQ)
        
        # Check for simulated failures
        failure_type = simulate_failure(message_data)
        if failure_type:
            logger.warning(f"Processing failed ({failure_type}), will go to DLQ")
            # ALL failures go to DLQ (not directly to PLQ)
            return False
            
        # Normal processing...
        return True
        
    except Exception as e:
        # Handle unexpected errors
        if dlq_retry_count >= 3:
            self._log_to_plq(message_data, message_id, error_type, error_message)
            return True
        return False  # Let it go to DLQ for retry

Message Callback extracts retry count from attributes:

def callback(self, message):
    """Handle incoming Pub/Sub message"""
    try:
        message_data = json.loads(message.data.decode('utf-8'))
        message_id = message.message_id
        correlation_id = message.attributes.get('correlation_id')
        
        # Extract DLQ retry count from message attributes
        dlq_retry_count = int(message.attributes.get('dlq_retry_count', 0))
        
        # Process with retry count awareness
        success = self.process_message(
            message_data, message_id, correlation_id, dlq_retry_count
        )
        
        if success:
            message.ack()  # Successfully processed
            logger.info(f"Message processed successfully: {message_id}")
        else:
            message.nack()  # Send to DLQ
            logger.warning(f"Message nacked, will go to DLQ: {message_id}")

3. DLQ Reprocessor (Retry Count Manager)

Location: dlq-reprocessor/app.py

The reprocessor adds retry count metadata and republishes:

def reprocess_message(self, message_data: Dict[Any, Any], message_id: str, 
                     correlation_id: str = None, dlq_retry_count: int = 0) -> bool:
    """Reprocess a message from DLQ with retry count tracking"""
    try:
        logger.info(f"Reprocessing message {message_id} (DLQ retry: {dlq_retry_count})")
        
        # Check if should reprocess
        should_reprocess, reason = self.should_reprocess(message_data, dlq_retry_count)
        if not should_reprocess:
            logger.info(f"Message not eligible: {reason}")
            return True
        
        # Enrich message with metadata
        enriched_message = self.enrich_message_for_reprocess(message_data, dlq_retry_count)
        
        # Apply exponential backoff delay
        delay = self.reprocess_delay * (2 ** dlq_retry_count)
        time.sleep(min(delay, 60))  # Cap at 60 seconds
        
        # KEY: Add DLQ retry count to message attributes
        attributes = {
            'correlation_id': correlation_id or f"dlq_reprocess_{message_id}",
            'dlq_retry_count': str(dlq_retry_count + 1),  # Increment retry count!
            'reprocess_timestamp': datetime.now().isoformat(),
            'from_dlq': 'true'
        }
        
        # Republish to main topic with retry count
        new_message_id = self.pubsub_manager.publish_message(
            self.main_topic, 
            enriched_message,
            attributes  # This adds the retry count header!
        )
        
        logger.info(f"Message {message_id} republished as {new_message_id} "
                   f"with dlq_retry_count={dlq_retry_count + 1}")
        return True

4. Error Storage (PLQ Implementation)

Location: support/database.py

Failed messages are stored in PostgreSQL for manual intervention:

def log_error(self, message_id: str, message_data: dict, error_type: str, 
              error_message: str, stack_trace: str = None, 
              correlation_id: str = None, dlq_retry_count: int = 0):
    """Log error to PLQ database"""
    try:
        query = """
        INSERT INTO error_log 
        (message_id, message_data, error_type, error_message, stack_trace, 
         correlation_id, dlq_retry_count, created_at)
        VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
        """
        
        with self.get_connection() as conn:
            with conn.cursor() as cursor:
                cursor.execute(query, (
                    message_id,
                    json.dumps(message_data),
                    error_type,
                    error_message, 
                    stack_trace,
                    correlation_id,
                    dlq_retry_count,  # Track retry count in database
                    datetime.now()
                ))
                conn.commit()

5. Failure Simulation

Location: support/common.py

Realistic failure scenarios for testing:

def simulate_failure(message_data: Dict[Any, Any]) -> Optional[str]:
    """Simulate various failure scenarios for DLQ testing"""
    
    # Check for explicit failure simulation
    fail_simulation = message_data.get('fail_simulation')
    if fail_simulation:
        return fail_simulation
    
    # Simulate random failures based on message content
    email = message_data.get('email', '')
    if email and not '@' in email:
        return 'validation_error'
    
    message_type = message_data.get('type')
    if message_type == 'malformed_message':
        return 'schema_error'
    
    # Random API timeout (10% chance)
    if random.random() < 0.1:
        return 'api_timeout'
    
    return None  # No failure

Testing Guide

Step 1: Start the System

# Start all services
./start-system.sh

# Verify services are running
docker-compose ps

Expected Output:

NAME                                STATUS                      PORTS                                
dlq-test-consumer-service-1   Up X minutes                                                     
dlq-test-dlq-reprocessor-1    Up X minutes                                                     
dlq-test-postgres-1           Up X minutes (healthy)     0.0.0.0:5432->5432/tcp               
dlq-test-producer-service-1   Up X minutes               0.0.0.0:8080->8080/tcp               
dlq-test-pubsub-emulator-1    Up X minutes (healthy)     0.0.0.0:8085->8085/tcp               
dlq-test-web-dashboard-1      Up X minutes               0.0.0.0:3000->3000/tcp

Step 2: Test Message Publishing

Test 1: Valid Message (Should Succeed)

curl -X POST http://localhost:8080/publish \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "user_signup",
    "email": "test@example.com", 
    "name": "Test User"
  }'

Expected Response:

{
  "message_id": "1",
  "status": "published",
  "timestamp": "2025-09-11T18:42:31.069694",
  "topic": "user-events"
}

Test 2: Invalid Message (Should Fail → DLQ)

curl -X POST http://localhost:8080/publish \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "user_signup",
    "email": "invalid-email",
    "name": "Invalid User",
    "fail_simulation": "validation_error"
  }'

Test 3: Batch Test Messages

curl -X POST http://localhost:8080/publish/samples

Expected Response:

{
  "description": "Published sample messages including some that will fail for DLQ testing",
  "messages": [
    {"message_id": "3", "type": "user_signup", "will_fail": false},
    {"message_id": "4", "type": "user_signup", "will_fail": true},
    {"message_id": "5", "type": "user_signup", "will_fail": true}
  ],
  "published_count": 5
}

Step 3: Monitor DLQ Flow

Check Consumer Logs

docker-compose logs -f consumer-service

Expected Output:

consumer-service-1  | INFO - Processing message 1, type: user_signup, dlq_retry_count: 0
consumer-service-1  | WARNING - Processing failed (validation_error), will go to DLQ, message_id: 2
consumer-service-1  | WARNING - Message nacked, will go to DLQ: 2

Check DLQ Reprocessor Logs

docker-compose logs -f dlq-reprocessor

Expected Output:

dlq-reprocessor-1  | INFO - Reprocessing message 2 (DLQ retry: 0)
dlq-reprocessor-1  | INFO - Message 2 republished as 8 with dlq_retry_count=1

Check Consumer Processing Retry

docker-compose logs -f consumer-service | grep "dlq_retry_count"

Expected Output:

consumer-service-1  | INFO - Processing message 8, type: user_signup, dlq_retry_count: 1
consumer-service-1  | INFO - Processing message 9, type: user_signup, dlq_retry_count: 2
consumer-service-1  | INFO - Message exceeded DLQ retry limit, sending to PLQ

Step 4: Verify PLQ Storage

docker-compose exec postgres psql -U dlq_user -d dlq_system

-- Check error logs
SELECT id, message_id, error_type, dlq_retry_count, created_at 
FROM error_log 
ORDER BY created_at DESC 
LIMIT 10;

-- Check PLQ messages (retry count >= 3)
SELECT message_id, error_type, dlq_retry_count 
FROM error_log 
WHERE dlq_retry_count >= 3;

Step 5: Web Dashboard Testing

1. Open Dashboard: http://localhost:3000

2. View Error Statistics: Check failed message counts

3. Analytics Page: http://localhost:3000/analytics

4. Manual Intervention: Click on failed messages for details

Step 6: Comprehensive Test Script

# Run all test scenarios
./test-dlq-scenarios.sh

Access the code here: Github

Summary

The poison pill problem shows how a single unprocessed message can bring down an entire pipeline. Dead Letter Queues (DLQs) provide a safety net by isolating failed messages so the rest of the system continues running smoothly.

In this post, we covered:

• What DLQs are and how they solve the poison pill problem.

• Design decisions around retries, failure modes, metadata, and thresholds.

• Strategies for handling transient vs. permanent failures.

• Trade-offs between Pub/Sub DLQs and database error tables.

• Hybrid approach for combining operational resilience with deep visibility.

• Implementation details with a true DLQ pattern in Google Pub/Sub, complete with reprocessing, retry counts, and error logging.

By designing DLQs carefully, you ensure that your system is:

• Resilient: no single message can clog the pipeline.

• Visible: errors are captured and analyzed.

• Flexible: transient errors can be retried, permanent ones quarantined.

• Controllable: engineers always have the final say through manual intervention.

Dead Letter Queues aren’t just about “dumping” bad messages, they’re about building robust, fault-tolerant pipelines that can gracefully handle the unexpected.

It was 2:00 AM when one of our downstream services pinged us:

“Our system says a lane exists, but your APIs disagree. Which one’s right?”

Slack was on fire. Kafka showed a brand-new lane between two facilities, but when someone queried our service, the database claimed nothing had changed. Downstream systems were already processing this phantom lane.

That was the night we learned the hard way about dual transactions.

In this post, I’ll walk you through:

• The problem with dual transactions: why Postgres and Kafka can easily fall out of sync.

• How the Outbox Pattern works: the schema, publisher design, and flow.

• Real-world debugging challenges we faced: stuck rows, duplicates, backlogs, and how we fixed them.

• The role of BigQuery: how we used it for observability, debugging, and analytics.

• Trade-offs and best practices: where Outbox shines, where it doesn’t, and a checklist if you want to adopt it.

• Business impact: how we went from weekly 2 AM escalations to zero.

The World We Lived In

Our service was the source of truth for all facilities: warehouses, carrier hubs, depots, cross docks, suppliers. We also managed lanes, the entities connecting facilities.

The rules were simple but strict:

• If a facility’s schedule changed, downstream needed to know.

• If a lane opened or closed, downstream needed to know.

We used Postgres as the database and Kafka as the event bus. Updates had to land in both. And that’s where the nightmare began.

Dual transactions, Dual Headaches

Two failure modes haunted us:

1. DB succeeds, Kafka fails

   • Data saved in Postgres.

   • Event never published.

   • Downstream blind to reality.

2. Kafka succeeds, DB rolls back

   • Event goes out.

   • DB rejects transaction.

   • Downstream believes a lie.

Both cases left us with corrupt, unsynced worlds. We had dashboards in red, and multiple “let’s sync the facility and lanes data” firefights.

At our peak, this caused 4-5 escalations per week. After fixing it, escalations dropped to zero.

Enter the Outbox Pattern

Instead of trying to write to two systems atomically (which distributed systems hate), we decided to only trust one write path: the database.

Here’s how:

Write Once

• Every DB transaction that modifies business data also inserts an event into an outbox table.

• If the DB commit succeeds, both data + outbox entry are durable.

CREATE TABLE facility_outbox (
    id BIGSERIAL PRIMARY KEY,
    aggregate_type VARCHAR(50),
    aggregate_id UUID,
    event_type VARCHAR(50),
    payload JSONB,
    created_at TIMESTAMP DEFAULT now(),
    processed BOOLEAN DEFAULT false
);

Query-based CDC

• A lightweight publisher job polls this outbox table.

• It picks unprocessed rows, marks them, then pushes them to Kafka.

SELECT * FROM facility_outbox
WHERE processed = false
FOR UPDATE SKIP LOCKED
LIMIT 100;

Mark as Processed

• After a successful publish, mark rows as processed = true.

• If publish fails, retry safely.

Now the DB was the single source of truth, and Kafka simply reflected it. No more split-brain.

Architecture at a Glance

Here’s what the flow looked like after Outbox adoption:

Debugging Nightmares We Solved

This wasn’t just plug-and-play. We hit plenty of edge cases:

1. Stuck Rows

• Sometimes publishers crashed mid-batch.

• Rows got locked forever.

• Fix: SKIP LOCKED ensured other workers could move on.

2. Duplicate Events

• Retries led to re-publishing.

• Solution: consumers became idempotent, keyed by event_id.

3. Slow Publishers

• Outbox grew faster than we could drain it.

• Fix: batch publishes + horizontal scaling of publishers.

4. Backfill Hell

• New systems wanted all historical events.

• Answer: replay directly from the outbox table → no custom scripts.

Why BigQuery Sink?

We weren’t just worried about delivery—we also needed observability.

• Every Kafka topic had a BigQuery sink.

• This let us:

  • Run ad-hoc queries: “Show all lane updates in the last 6h.”

  • Debug mismatches: compare DB state vs. published events.

  • Prove SLAs: how fast events left the outbox.

It was our black box recorder for event history.

Trade-offs We Faced

The Outbox pattern isn’t a silver bullet. We had to weigh:

• Extra DB Load: every change means an extra insert.

• Latency: events are near-real-time, not instant (polling adds ~200–500ms).

• Storage Growth: outbox table can bloat if not purged/archived.

• Operational Complexity: need monitoring, retries, and cleanup policies.

But for us, consistency > absolute latency. We’d rather downstream be slightly late than totally wrong.

When to (and NOT to) Use Outbox

Use Outbox if…

• Your service is a source of truth and must publish events reliably.

• You’re dealing with critical entities (orders, payments, facilities, lanes).

• Dual writes would cause corruption.

Don’t bother if…

• Events are non-critical logs/metrics.

• Latency requirements are microseconds.

• You already have a rock-solid streaming CDC pipeline like Debezium.

Checklist for Adopting Outbox

1. Design outbox schema (include metadata + payload).

2. Use FOR UPDATE SKIP LOCKED to avoid stuck workers.

3. Make consumers idempotent.

4. Add monitoring: queue size, lag, failed publishes.

5. Decide retention: archive or delete processed rows.

6. (Optional) Sink to BigQuery or warehouse for debugging.

Final Thought

After moving to Outbox, our 2 AM “facility lane mismatch” alerts vanished.
No more phantom lanes. No more split-brain facilities.

The system wasn’t perfect—there was extra DB load, and events weren’t instant. But consistency beat chaos.

And honestly? With Outbox in place, the only thing waking us up at 3 AM was the baby, not broken events. 👶✨

I still remember the moment clearly. I was in the middle of presenting a design for a new event-driven system. I had my architecture diagram up, services, topics, retries, and yes, a neat little Dead Letter Queue (DLQ) sitting off to the side.

One of the staff engineers raised a hand and asked me a simple question:

“Once a message is sent to the DLQ, then what? How do you actually know what the issue is?”

For a second, I froze. I had accounted for retries. I had thought about scaling. I had made sure failures wouldn’t break the main pipeline. But I realized I hadn’t thought deeply about what happens after something lands in the DLQ.

That question stuck with me. Because the truth is, in many system designs, a DLQ is treated like a checkbox. We draw it in the diagram, it gives a sense of safety, and we move on. But if you’ve ever actually operated such a system, you know that a DLQ is not the end of the story—it’s the beginning of one.

The Day We Lost Data Quietly

Not long after that presentation, I had my wake-up call.

One of our downstream services had made a schema change. The main queue retried the failing messages, but the consumers could not process them anymore. Eventually, those messages were moved into the DLQ.

And then nothing happened.

There was no alert. There was no dashboard entry calling attention to it. The DLQ sat quietly in the background, collecting failed messages that nobody was watching.

Two weeks later, those messages expired because in AWS SQS, DLQs automatically purge messages after fourteen days. By the time someone noticed missing data in a downstream report, it was gone forever. We had no way of knowing what the payloads were, no trace IDs, no metadata. Just silence.

That incident drove home a painful lesson: DLQs do not save you. They only delay the inevitable. Unless you monitor them and act, they become silent data graves.

So, What Is a DLQ?

Think of a DLQ like the “Lost and Found” counter at an airport. When baggage gets misplaced, it doesn’t vanish, it shows up at the counter. But if no one ever checks the counter, or if the bag doesn’t have a proper tag, it’s almost as good as lost.

That’s what happens in many systems. Messages that can’t be processed get pushed into the DLQ. They sit there quietly, like unclaimed baggage, until someone eventually notices there’s a growing pile of them. By then, the real issue, bad data, a downstream failure, or a misconfiguration, may have already caused damage.

So the real question isn’t do you have a DLQ? It’s what do you do with it?

The Questions We Rarely Ask

Looking back, I realized that my original design doc never asked the right questions:

• What happens when a message actually lands in the DLQ?

• Do we consume and log it anywhere permanent?

• Should metadata, like error type, trace ID, and timestamp be attached so engineers can diagnose the cause?

• Who owns the DLQ, and how quickly should they act when it fills up?

• How do we prevent silent expiration?

Without answers, the DLQ is just a checkbox. It exists, but it does not help.

Poison Pills and Infinite Loops

Later, I ran into another problem: the poison pill message.

This is the message that can never succeed. Perhaps the payload is malformed beyond repair, or the downstream service rejects it permanently.

At one point, we had such a poison pill bouncing endlessly between the main queue and the DLQ because of a naive “automatic redrive” process. Every time it was replayed, it failed in the same way.

Instead of healing the system, our DLQ had turned into a loop of repeated failure. That was the moment I realized: redrives must never be blind. You need context. You need human eyes or smart triage before you send a message back into the system.

Turning DLQs Into Debugging Tools

The breakthrough came when we stopped treating DLQs as storage and started treating them as debugging pipelines.

Instead of dumping raw payloads, we enriched every DLQ message with structured metadata:

• Error category (validation error, downstream failure, business rule violation)

• Originating service

• Retry count

• Trace or correlation ID

• Timestamp

Now, when a message appeared in the DLQ, engineers could immediately understand why it was there. We had transformed the DLQ from a black box into a window into system health.

From DLQ to PLQ: A Permanent Record

Even with metadata, DLQs came with another limitation: expiration.

We solved this by creating what we called a Parked Letter Queue (PLQ), a database-backed error log that stored all failed messages, enriched with context.

The way I like to describe it is that the DLQ is like the emergency room: temporary, chaotic, and focused on urgent triage. The PLQ is the hospital’s archive: a permanent record of every case.

With the PLQ, we finally had:

• A searchable history of failures

• An audit trail for debugging and compliance

• The ability to identify recurring failure patterns over time

That single addition meant we no longer woke up to missing data with no explanation.

A Simple Analogy

Imagine a conveyor belt in a factory. Most items pass through fine, but occasionally one is defective. Instead of halting the belt, the item is pushed into a reject bin, that is your DLQ.

Now imagine if no one ever checks the reject bin. Products keep disappearing, and nobody knows why.

But if every defective item is labeled with why it failed, which machine broke, and at what time, and those labels are logged in a permanent book, you start to see patterns. Maybe Machine A jams every Monday. Maybe Supplier X always ships faulty parts. Suddenly, the reject bin is not just waste; it is a source of insight.

That is how DLQs should be treated.

Designing DLQs That Actually Help

From these experiences, I realized that a DLQ isn’t just a “safety bucket”, it’s a tool for learning, accountability, and resilience. But that only works if it’s designed with intention. Here are some principles that have stuck with me:

1. Monitor them like you mean it.
A DLQ that no one watches is just a black hole. Track not only how many messages land there (depth), but also how long they’ve been sitting (age) and how fast new ones are arriving (throughput). A sudden spike could mean a broken deployment. A slow accumulation might indicate a silent data quality issue. Alerts on these metrics often become your first signal that the system is quietly burning.

2. Enrich messages with metadata.
A raw payload alone is often meaningless at 3 AM when you are debugging. Include trace IDs, service names, timestamps, error codes, or even a short reason for failure. Think of it as leaving breadcrumbs for your future self (or the unlucky teammate on call). The easier it is to reconstruct the story of “what went wrong,” the faster you can fix it.

3. Redrive with discipline, not desperation.
The temptation is to just shovel DLQ messages back into the main queue once the pipeline looks healthy again. That’s risky. If the root cause hasn’t been fixed, you’re just replaying the same movie. Instead, validate fixes, test carefully, and only then reintroduce messages. Redriving should be a controlled process, not a panic button.

4. Consider a PLQ or permanent error log.
DLQs often have retention limits. If you don’t act in time, your evidence just expires. A Parked Letter Queue (PLQ) or dedicated error store solves this by keeping failed messages along with all the metadata, forever if needed. It becomes the system’s diary of failures, a reliable record you can revisit for audits, debugging, or even training better validation rules.

Final Reflection

That simple question during my design presentation, “Once a message is sent to the DLQ, then what?” was one of the most valuable lessons of my career.

DLQs are not the end of the story. They are the beginning of an investigation. They are not insurance policies; they are mirrors reflecting where your system is fragile.

The real measure of resilience is not whether you have a DLQ. It is whether you know what is inside it, and whether you are doing something about it.

If you would like me to share a guide on how to implement DLQs effectively, whether on AWS SQS, Google Pub/Sub, or Kafka, drop a comment below.

Let’s be honest, most of us have faced that moment of frustration when technology doesn’t behave the way we expect. Maybe you clicked “Pay” on a checkout page, the internet blinked, and you weren’t sure whether you should hit the button again. Or maybe you submitted a form twice because the first time seemed to fail, only to realize later that your action got recorded multiple times.

These little glitches don’t just annoy us as users; they can create big headaches for the systems and teams behind the scenes. Duplicate payments, repeated orders, multiple job submissions, these are not just bugs, they’re trust-breakers. When users lose confidence that your system will “do the right thing”, they start doubting whether they should use it at all.

This is where idempotent keys, a concept Stripe brought into the spotlight, come in. And trust me, it’s one of those rare engineering ideas that’s both simple and deeply impactful.

What Stripe Did with Idempotent Keys

Stripe, being a payments company, had a problem to solve:

• Customers might refresh or retry during checkout.

• Network requests might time out and be retried by clients automatically.

• Without safeguards, this could result in someone being charged twice for the same transaction.

Their solution was beautifully straightforward:

Every request sent to Stripe can include a unique idempotent key, essentially a unique identifier (like a UUID).

Here’s how it works:

• The first time Stripe sees that key, it processes the request and stores the result.

• If the same key shows up again, Stripe doesn’t re-run the process, it simply returns the same result it gave the first time.

The magic? To the customer, the action always feels safe and predictable. No matter how many times they click “Pay”, they’ll only ever be charged once.

Why This Matters Beyond Payments

At first glance, idempotent keys sound like a “payments thing”. But if you take a closer look, the underlying principle applies everywhere:

• APIs & Microservices
  When your services call each other, retries are inevitable. Idempotency ensures retries don’t create duplicate records or trigger duplicate jobs.

• Event-Driven Systems
  Events often get delivered more than once. Without idempotency, you risk creating duplicate entities or processing the same task twice.

• Batch Jobs & Uploads
  Imagine uploading a large file, only for the system to fail midway. With idempotency, rerunning the job won’t duplicate data, it just resumes gracefully.

• User Actions
  From submitting feedback forms to booking tickets, idempotency makes sure each “intention” from the user gets captured once, not multiple times.

In other words, anywhere retries or duplicate requests can happen, idempotency helps you stay safe.

Benefits of Idempotent Keys

1. Reliability in the face of retries
   Your system can handle the messy realities of networks and user actions without breaking trust.

2. Better user experience
   Users feel reassured knowing they won’t be double-billed, double-registered, or double-subscribed.

3. Operational safety
   Engineers and support teams spend less time cleaning up duplicate data or refunding accidental charges.

4. Simplicity in design
   Instead of inventing complex duplicate-handling logic, you get a clean, repeatable pattern.

5. Traceability
   Each request is tied to a key, making it easier to debug, audit, and reason about the system’s behaviour.

How to Implement Idempotent Keys in Spring Boot

Let’s make this concrete with a Java Spring Boot example. We’ll design a reusable solution using annotations so that any endpoint can become idempotent just by adding @Idempotent.

Don’t feel like reading the full breakdown? Explore the code directly here.

Step 1. Create the Annotation

We’ll mark methods that should support idempotency.

@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface Idempotent {
    // Optional: custom expiry time in seconds (default = 24 hours)
    long expirySeconds() default 86400; 
}

Step 2. Create the Aspect

This Aspect will:

• Check for the Idempotency-Key header.

• If the key exists in Redis, return cached response.

• If not, process request, save response, and return it.

@Aspect
@Component
public class IdempotencyAspect {

    private final RedisTemplate<String, String> redisTemplate;
    private final ObjectMapper objectMapper;

    public IdempotencyAspect(RedisTemplate<String, String> redisTemplate, ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }

    @Around("@annotation(idempotent)")
    public Object handleIdempotency(ProceedingJoinPoint joinPoint, Idempotent idempotent) throws Throwable {
        
        ServletRequestAttributes requestAttributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
        if (requestAttributes == null) {
            // Not in a web request context, proceed normally
            return joinPoint.proceed();
        }

        HttpServletRequest request = requestAttributes.getRequest();
        HttpServletResponse response = requestAttributes.getResponse();

        String idempotencyKey = request.getHeader("Idempotency-Key");
        if (idempotencyKey == null || idempotencyKey.isEmpty()) {
            if (response != null) {
                response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Missing Idempotency-Key header");
            }
            return null;
        }

        // Add the idempotency key to response headers for visibility
        if (response != null) {
            response.setHeader("X-Idempotency-Key", idempotencyKey);
        }

        // Check if we've seen this key
        String cachedResponse = redisTemplate.opsForValue().get(idempotencyKey);
        if (cachedResponse != null) {
            // Cache HIT - return cached response
            if (response != null) {
                response.setHeader("X-Idempotency-Cache-Status", "HIT");
            }
            
            // For cached responses, we always return a ResponseEntity with the cached body
            // since our controller methods return ResponseEntity
            try {
                Object responseBody = objectMapper.readValue(cachedResponse, Object.class);
                return ResponseEntity.ok(responseBody);
            } catch (Exception e) {
                // If deserialization fails, proceed with normal execution
                // This ensures the system remains robust
                System.err.println("Failed to deserialize cached response: " + e.getMessage());
            }
        }

        // Cache MISS - process the request and cache the response
        if (response != null) {
            response.setHeader("X-Idempotency-Cache-Status", "MISS");
        }

        Object result = joinPoint.proceed();

        // Cache the response if it's successful
        if (result != null) {
            try {
                String responseBody;
                if (result instanceof ResponseEntity<?> responseEntity) {
                    // Extract the body from ResponseEntity and serialize it
                    responseBody = objectMapper.writeValueAsString(responseEntity.getBody());
                } else {
                    responseBody = objectMapper.writeValueAsString(result);
                }
                
                long expirySeconds = idempotent.expirySeconds();
                redisTemplate.opsForValue().set(idempotencyKey, responseBody, expirySeconds, TimeUnit.SECONDS);
            } catch (Exception e) {
                // Log the exception but don't fail the request
                System.err.println("Failed to cache idempotent response: " + e.getMessage());
            }
        }

        return result;
    }
}

Step 3. Use It in a Controller

Now, simply annotate your endpoint:

@RestController
@RequestMapping("/api")
public class SampleController {
    @PostMapping("/orders")
    @Idempotent(expirySeconds = 3600) // 1 hour custom expiry
    public ResponseEntity<Map<String, Object>> createOrder(@RequestBody Map<String, Object> orderRequest) {
        // Simulate order creation
        Map<String, Object> response = new HashMap<>();
        response.put("orderId", UUID.randomUUID().toString());
        response.put("status", "created");
        response.put("timestamp", LocalDateTime.now().toString());
        response.put("customerData", orderRequest);
        
        return ResponseEntity.ok(response);
    }
}

Step 4. Client Example

Clients must send the header:

First request: processes and creates an order.

curl -v -X POST http://localhost:8080/api/orders \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: fixed-test-123" \
  -d '{"customerId": "customer-1", "items": [{"id": "item-1", "quantity": 2}]}'

> Content-Type: application/json
> Idempotency-Key: fixed-test-123
> Content-Length: 72
> 
* upload completely sent off: 72 bytes
< HTTP/1.1 200 
< X-Idempotency-Key: fixed-test-123
< X-Idempotency-Cache-Status: MISS
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Tue, 26 Aug 2025 16:53:22 GMT
< 
* Connection #0 to host localhost left intact
{"orderId":"c9431b5d-2a01-48b4-b788-442a82f5827c","customerData":{"customerId":"customer-1","items":[{"id":"
item-1","quantity":2}]},"status":"created","timestamp":"2025-08-26T16:53:22.806485375"}

Second request with same key: returns the exact same response.

curl -v -X POST http://localhost:8080/api/orders \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: fixed-test-123" \
  -d '{"customerId": "customer-1", "items": [{"id": "item-1", "quantity": 2}]}'

> Content-Type: application/json
> Idempotency-Key: fixed-test-123
> Content-Length: 72
> 
* upload completely sent off: 72 bytes
< HTTP/1.1 200 
< X-Idempotency-Key: fixed-test-123
< X-Idempotency-Cache-Status: HIT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Tue, 26 Aug 2025 16:53:30 GMT
< 
* Connection #0 to host localhost left intact
{"orderId":"c9431b5d-2a01-48b4-b788-442a82f5827c","customerData":{"customerId":"customer-1","items":[{"id":"
item-1","quantity":2}]},"status":"created","timestamp":"2025-08-26T16:53:22.806485375"}

Dive into the full code example on Github.

Things to Keep in Mind

• Expiration
  Don’t keep keys forever. Set a time limit that makes sense (e.g., a day for payments, longer for data uploads).

• Scope
  Define what the key represents. Is it for a whole order? A single file upload? A single API call?

• Atomicity
  Make sure storing and checking the key happen atomically to avoid race conditions.

• Consistency
  Be clear about which parts of the request are idempotent. For example, in payments, the amount and currency should remain constant for the same key.

Closing Thoughts

Idempotent keys are one of those rare engineering practices that quietly do wonders. Stripe made them famous by solving a very human problem: the fear of being charged twice. But the principle is universal, make actions safe, predictable, and trustworthy.

Whether you’re running a payment system, a booking platform, or just an API that retries requests, adopting idempotent keys means you’re building resilience into your foundation. And resilience, at the end of the day, is what separates systems people merely use from systems people deeply trust.

In this post, I will talk about:

• Why software engineers usually get fired?

• Kunal’s story: how not surfacing a design risk led to a failure in production.

• The difference between invisible work and visible communication.

• A simple challenge to make your thinking louder in everyday updates.

Most engineers don’t lose their jobs because they wrote bad code. They lose them because they stayed quiet.

I used to work with an engineer named Kunal.

Kunal wasn’t the loudest voice in the room. In fact, he was the kind of engineer every manager loves: reliable, never missed deadlines, never complained. But that reliability came with silence.

When requirements were vague, he just assumed and started coding.
When timelines were unrealistic, he pulled late nights instead of pushing back.
When he noticed flaws in the design that might cause scaling issues down the line, he kept it to himself.

Each time, silence felt easier than debate. Until it wasn’t.

Three months later, exactly what Kunal had feared came true. The API couldn’t handle the surge in traffic. Latency spiked. Upstream services started failing. Customers complained. Leadership was in midnight war-room calls. And the hardest part? Kunal had seen it coming all along. He just never said it out loud.

That’s when I realized something important:
You can ship bugs and still keep your job. But if you don’t communicate risk, you become the risk.

Invisible work doesn’t get you promoted.
Invisible problems get you replaced.

And the truth is, many of us have a little bit of Kunal in us. We notice risks but tell ourselves, “I’ll just work harder to cover it up.” We think silence will protect us, but it only isolates us.

The best engineers I’ve seen don’t just write great code. They make their thinking visible. They clarify the scope. They surface risks early. They translate technical trade-offs into business clarity. And because of that, people trust them.

If you want to become indispensable, don’t just improve your coding skills. Improve how loudly and clearly you share your thinking.

So here’s a small challenge for you this week: in your next standup or update, don’t just share what you did. Share one risk you see, one trade-off you’re making, and one thing you need clarity on.

Small, consistent signals build big trust.

Most of us will eventually become managers. But all of us, at some point, will have managers.

And here is the tricky part: very few of us actually manage our managers well.

I have worked across startups and large organizations, and one pattern I have noticed is this: talented engineers burn out not because of the work itself, but because their relationship with their manager is misaligned. They either see their manager as an obstacle or as a saviour. Both are flawed mental models.

So let us talk about how to make this relationship work, not by waiting for your manager to magically “get better,” but by learning how to manage up effectively.

Warmly,
~Rahul

In this post

We will talk about:

• What your manager is not

• When your relationship with your manager feels tough

• What a great manager–report relationship looks like

• How to think about performance reviews

• What you should expect from your manager

• How to give feedback to your manager

• How I manage sync-ups and alignment with my manager

What your manager is not

When we are new to the workforce, we often put managers into extreme roles.

• The executioner.
  This is the belief that “If I make a mistake, my manager will punish me, maybe even fire me.” Every interaction feels like walking on eggshells.

• The saviour.
  This is the belief that “My manager will fight for me, protect me, and guarantee my promotion.” Here, we rely on them to make everything better for us.

Both of these mental models are misleading. Why? Because both give away your power.

Yes, your manager has influence. They can promote you, give you challenging opportunities, or if things go badly, let you go. But that doesn’t mean they control your entire career. You do.

Think of it like this: your career is a book, and you are the author. Your manager? At best, they are a key character, sometimes even a co-author of a few chapters. But they are not the hero, and they are not the one holding the pen.

The shift begins when you stop asking, “What will my manager do for me?” and start asking, “What can I do to make this relationship better for both of us?”

When your relationship with your manager feels tough

Maybe your manager doesn’t give you feedback. Maybe they never seem to have time for you. Or maybe you feel like they don’t understand what you are capable of. It happens a lot.

But here is the good news: you can take steps to improve things.

1. Know what good looks like.
   Before fixing to do anything, define what a good end product means to you and the manager. Do you want more context about the work or the strategy? Are the goals clear? Make sure to have an honest feedback loop? Write down everything.

2. Talk about it openly.
   Use your 1:1s. Say something like: “Here is the kind of working relationship I would love us to have. For example, I would really appreciate more feedback on my projects. Does that make sense to you?”

3. Be clear about your needs.
   Vague complaints don’t help. Instead of “I feel unsupported,” say: “It would help me if I got feedback after the demo, so I know how to improve next time.”

4. Give your manager feedback.
   Once a month, tell them what is working. “I appreciated that you gave me context before that meeting, it made me more confident.” Then add what you would like more of.

5. Ask for feedback.
   Don’t wait for annual reviews. Simple questions work: “What is one thing I could do differently that would make your job easier?”

6. Escalate if ignored.
   If you have given clear, specific feedback several times and nothing changes, it is fair to bring it up with their manager. Do it respectfully and factually.

7. Leave if trust is gone.
   If you have tried everything and still don’t trust your manager, it may be time to move on. Don’t let one person is limitations block your career.

What a great manager–report relationship looks like

The best relationships between managers and their team members are built on two things: trust and alignment.

Ask yourself:

• Do I trust my manager to do their job well?

• Does my manager trust me to do my job well?

If either answer is “not really,” there is probably misalignment.

For example:

• You might think your manager is role is to back your ideas.

• But your manager might think their role is to challenge your ideas and push for better solutions.

Neither is wrong, but if you don’t talk about it, you will constantly misunderstand each other.

So, ask questions:

• “What should I do differently to get more visibility?”

• How to give feedback to your manager

Once both of you have laid out your definitions of success, you can co-create a shared vision. This clarity transforms the relationship from transactional (“Do this task”) to collaborative (“Let is make each other successful”).

How to think about performance reviews

Performance reviews often get a bad reputation as judgment days. But if you approach them differently, they can become powerful growth moments.

• Celebrate strengths. If your manager praises something, don’t dismiss it. Lean into it. Strengths are often where your biggest opportunities lie.

• Over-correct on patterns. If you keep hearing “speak up more in meetings,” don’t just nudge slightly. Set yourself a goal: “I’ll contribute at least three points in every meeting.” It might feel unnatural, but that is often what is needed to shift perception.

• Shape your next review. Instead of passively waiting, imagine the review you want six months from now. Do you want to be known as a strong mentor? A clear communicator? Tell your manager: “By the next review, I would love to be recognized for X. Can you help me get there?”

What you should expect from your manager

This isn’t one-sided. Managers owe you certain things. At a minimum, expect them to:

• Give you clarity on priorities and goals.

• Provide regular, constructive feedback.

• Create opportunities for you to grow.

• Remove blockers you can’t clear alone.

• Advocate for your work when recognition or promotions are on the line.

If they are not doing these things, speak up. It is not disrespectful, it is necessary.

How to give feedback to your manager

This is where most people freeze. But here is the truth: managers need feedback too.

• Start with curiosity. Instead of “You don’t support me,” try: “I noticed we don’t often do feedback sessions. Is that intentional? How do you prefer to share feedback?”

• Frame it around impact. “When I don’t get feedback after presentations or product demos, I am not sure if I met expectations. It would help me improve faster if you shared your thoughts.”

• Be specific. Concrete requests are easier to act on.

• Follow up. If things don’t improve, ask again: “I wanted to check if the changes we discussed are working for you as well.”

• Escalate respectfully. If nothing changes, involve their manager with facts, not emotions.

Giving feedback to your manager is not rebellion — it is leadership.

How I manage sync-ups and alignment with my manager

A strong manager–report relationship doesn’t just happen in performance reviews. It is built in the small, consistent ways you communicate. Here is what works for me:

1. Biweekly 1:1s with structure
I keep a simple structure for my one-on-ones so that both of us walk away with clarity:

• 20%: What I have accomplished

• 20%: What I can do better

• 20%: Where I fell short

• 40%: Action items

  • 50% on me

  • 50% on my manager

This balance makes the conversation honest, forward-looking, and accountable on both sides. It is not just me reporting status. It is us aligning and committing to actions together

2. Dedicated Slack channels for visibility
I use Slack heavily to reduce back-and-forth and increase transparency. For each project, I create dedicated channels that include my manager and my team. We use them for:

• Design discussions

• Deliverables and sign-offs

• Roadblocks and clarifications

This ensures my manager always has visibility into the work, without needing to chase me for updates. One principle I live by: your manager should already know the state of your work before they ask.

This combination of structured sync-ups and transparent async updates creates trust, alignment, and accountability.

In a recent project I worked on, I was puzzled by what seemed like an inconsistent indexing situation. This inconsistency bothered me, so I decided to dig deeper. What I discovered taught me that "wrong" isn’t always wrong. It is often about trade-offs I hadn’t considered.

The Starting Point: A Confusing Primary Key

The first thing that caught my attention was the primary key design:

CREATE TABLE locations (
    location_id TEXT NOT NULL,      -- e.g., 'A01020304' (hierarchical location code)
    warehouse_id TEXT NOT NULL,     -- e.g., '25' 
    -- ... other columns
    CONSTRAINT pk_locations_location_id_warehouse_id 
        PRIMARY KEY (location_id, warehouse_id)
);

This struck me as backwards. In most multi-tenant systems I've worked with, you'd cluster by warehouse_id first for tenant isolation. But here, location_id comes first. I wondered if this was a mistake or if there was reasoning I wasn't seeing.

Understanding the Location ID Structure

As I dug deeper, I discovered that these location IDs weren't just random strings. They followed a specific hierarchical structure that was key to understanding the whole system. Each warehouse had thousands (sometimes millions) of storage locations, and every location was encoded into a structured string ID.

For example, a location ID like A01020304 breaks down as:

• Aisle: A01 (an uppercase letter + 2 digits)

• Bay: 02 (2 digits)

• Level: 03 (2 digits)

• Position: 04 (2 digits)

When someone searches for A01%, they're looking for all locations in aisle A01 across any warehouse. When they search for H1%, they want all locations in aisles starting with H1. This spatial organization is crucial for warehouse operations. workers need to find adjacent locations, analyze storage density by aisle, or audit entire sections of the warehouse.

My First Breakthrough: The Migration History

Curious about the primary key design, I dug through the database migration history. What I found surprised me:

Original Design (October 2023)

PRIMARY KEY (warehouse_id, location_id)  -- Warehouse first

Changed to (February 2025)

PRIMARY KEY (location_id, warehouse_id)  -- Location first

So it wasn't a mistake! The previous team had deliberately switched the primary key ordering. This made me even more curious. What drove that decision? And more importantly, I started noticing something concerning in my testing: PostgreSQL kept avoiding this index, even when it seemed like it would be faster.

My Testing Adventure: When the Planner Surprised Me

This is where my investigation got really interesting. I decided to systematically test different query patterns to understand when PostgreSQL was making good choices versus poor ones. What I found challenged everything I thought I knew about query optimization.

Test 1: Location Pattern Query

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE location_id LIKE 'H1%';

Result: 477ms with parallel sequential scan for 92,689 rows (~4% of table)

PostgreSQL chose sequential scan, which makes sense for large result sets.

Test 2: Warehouse-Specific Query

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE warehouse_id = '25';

Natural planner choice: 567ms (sequential scan) Forced index usage: 282ms (50% faster!)

SET enable_seqscan = false;
-- Same query runs in 282ms with bitmap index scan
RESET enable_seqscan;

Test 3: Combined Query

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE warehouse_id = '25' AND location_id LIKE 'H1000%';

Natural planner choice: 501ms (sequential scan) Forced index usage: 354ms (29% faster)

Test 4: LIMIT with OFFSET Queries

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE location_id LIKE 'H1%' AND warehouse_id = '09' 
LIMIT 100 OFFSET 100;

Natural planner choice: 436ms (sequential scan) Forced index usage: 204ms (53% faster)

-- Natural planner result (sequential scan)
Limit  (cost=6560.89..12121.78 rows=100 width=12) (actual time=412.845..436.060 rows=0 loops=1)
  ->  Gather
        ->  Parallel Seq Scan on locations
              Filter: (((location_id)::text ~~ 'H1%'::text) AND ((warehouse_id)::text = '09'::text))
              Rows Removed by Filter: 746758

-- Forced index result  
Limit  (cost=9743.66..19486.90 rows=100 width=12) (actual time=204.298..204.300 rows=0 loops=1)
  ->  Index Only Scan using pk_locations_location_id_warehouse_id on locations
        Index Cond: (warehouse_id = '09'::text)
        Filter: ((location_id)::text ~~ 'H1%'::text)
        Rows Removed by Filter: 30265

Key insight: Even with LIMIT + OFFSET, the index scan won significantly. The index could filter by warehouse_id first (30k rows examined) vs. sequential scan examining 746k rows across workers.

Test 5: Simple LIMIT Queries (When the Planner Gets It Right)

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE location_id LIKE 'H1%' 
LIMIT 100;

Natural planner choice: 0.579ms (sequential scan) Forced index usage: 321ms (550x slower!)

-- Sequential scan result
Limit  (cost=0.00..91.49 rows=100 width=12) (actual time=0.035..0.553 rows=100 loops=1)
  ->  Seq Scan on locations
        Filter: ((location_id)::text ~~ 'H1%'::text)
        Rows Removed by Filter: 3060
        
-- Forced index scan result  
Limit  (cost=0.43..224.78 rows=100 width=12) (actual time=321.386..321.501 rows=100 loops=1)
  ->  Index Only Scan using pk_locations_location_id_warehouse_id on locations
        Filter: ((location_id)::text ~~ 'H1%'::text)
        Rows Removed by Filter: 433527

Here, the planner was absolutely correct! Sequential scan found 100 results after examining only 3,160 rows, while the index scan had to examine 433,627 rows to find the same 100 results.

Why LIMIT changes everything:

• Sequential scan can stop as soon as it finds enough rows

• H1% patterns are distributed throughout the data

• Index scan must traverse scattered index pages to find matches

• For small result sets, "first 100 found" beats "optimally ordered access"

Summary: When Indexes Help vs. Hurt

The Pattern:

• Large scans: Sequential wins or is reasonable

• Medium selectivity with filtering: Index wins significantly (even with LIMIT + OFFSET)

• Simple LIMIT queries: Sequential wins dramatically (can stop early)

Understanding Why PostgreSQL Made These Choices

The more I tested, the more I realized there was a pattern to the planner's behavior. PostgreSQL's cost-based optimizer sometimes chose suboptimal plans, but sometimes it was exactly right. I needed to understand the logic behind these decisions.

1. The Cost Model Mystery

I discovered that PostgreSQL uses abstract "cost units" to compare execution strategies. Looking at my system's configuration, I found these parameters:

-- My system's cost parameters (defaults)
SHOW random_page_cost;        -- 4.0 (random reads cost 4x sequential!)
SHOW effective_cache_size;    -- 1502984kB (~1.5GB in my case)
seq_page_cost = 1.0          -- Cost to read a page sequentially  
cpu_tuple_cost = 0.01        -- Cost to process one row

For my warehouse query (warehouse_id = '25' with 6.5% selectivity), the planner calculated:

• Sequential scan cost: ~67,000 units (read entire table sequentially, low per-page cost)

• Index scan cost: ~130,000 units (random index + heap access, 4x page cost penalty)

This explained why PostgreSQL kept choosing sequential scans! The planner assumed random I/O was 4x more expensive than sequential I/O. But when I forced the index usage, it was actually 50% faster. The cost model seemed to be wrong about modern storage characteristics.

2. The Index Scattering Problem

I realized that with the (location_id, warehouse_id) primary key structure I inherited:

• All warehouse '25' locations were scattered throughout the index

• To find them, the index scan had to jump between many different location_id ranges (A01..., B01..., H1...)

• This created the "random" access pattern that the cost model was heavily penalizing

3. Statistics vs. Reality

Interestingly, PostgreSQL's row count estimates were quite accurate (144k estimated vs 145k actual rows). The problem wasn't with statistics - it was with cost estimation. The planner didn't account for:

• How efficiently modern SSDs handle the "random" access patterns

• How much of the data was already cached in the 1.5GB buffer pool

• How well bitmap heap scans actually perform on this hardware

My Analysis: Was This Bad Design?

The big question that kept bugging me: When the planner consistently chooses slower execution plans, does that mean the database design is wrong?

My Conclusion: It's All About Trade-offs

As I analyzed the system deeper, I realized the primary key design created a fundamental trade-off:

1. Location-first ordering (location_id, warehouse_id) optimized for:

   • Global operations spanning all warehouses.

   • Spatial analysis across the entire location hierarchy

2. But it penalized warehouse-specific queries because:

   • Warehouse data became scattered throughout the index

   • This forced "random" access patterns that the cost model heavily penalized

Understanding the Previous Team's Logic

Looking back at the migration history, I realized this was a completely deliberate decision:

October 2023: PRIMARY KEY (warehouse_id, location_id) (conventional multi-tenant design)

February 2025: PRIMARY KEY (location_id, warehouse_id) (location-first optimization)

The team had switched from the "obvious" warehouse-first design to location-first. This told me:

• Cross-warehouse location queries were more critical to the business than warehouse isolation

• They had probably measured the performance impact and decided the trade-off was worth it

• The 50% penalty for warehouse queries was acceptable given their workload priorities

Issue with the query planner: Outdated cost assumptions

The more I studied this, the more I realized the problem wasn't the index design - it was PostgreSQL's cost model being outdated for modern hardware:

• The cost model assumed random I/O was 4x more expensive than sequential

• Modern SSDs and the 1.5GB buffer pool made this assumption much less relevant

• Bitmap heap scans performed better than the planner estimated on this hardware

What I Learned and How I'd Optimize This System

1. Query Planner "Mistakes" Are Often Normal

My investigation taught me that query planners making suboptimal choices is more common than I expected, especially:

• On modern hardware (SSDs, large RAM)

• With index designs that optimize for specific business patterns

• When cost models haven't kept up with hardware evolution

2. The Power of Testing Assumptions

I developed a methodical approach to test when the planner might be wrong, but learned to be careful with LIMIT queries:

-- My testing approach for the warehouse queries
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM locations WHERE warehouse_id = '25';

-- Force index usage to compare
SET enable_seqscan = false;
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM locations WHERE warehouse_id = '25';
RESET enable_seqscan;

-- But I learned LIMIT queries often favor sequential scans
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM locations WHERE location_id LIKE 'H1%' LIMIT 100;

3. How I'd Optimize: Supplemental Indexes

Based on my analysis, if I were optimizing this system, I'd add a supporting index to get the best of both worlds:

-- Add this to optimize warehouse-specific queries
CREATE INDEX CONCURRENTLY idx_locations_warehouse_location 
ON locations (warehouse_id, location_id);

This would preserve the benefits of the location-first primary key while eliminating the 50% performance penalty for warehouse queries.

4. Cost Model Tuning Experiments I'd Try

I realized this system would benefit from tuning PostgreSQL's cost model to reflect modern hardware realities:

-- Current settings that explain the planner behavior
SHOW random_page_cost;        -- 4.0 (assumes spinning disks!)
SHOW effective_cache_size;    -- 1502984kB (~1.5GB in our case)

-- Experiments I'd try:
SET random_page_cost = 1.1;   -- Reflect SSD performance characteristics
SET effective_cache_size = '4GB';  -- If more memory is available

Lowering random_page_cost would make index scans more attractive to the planner, since it would reduce the penalty for the "random" I/O operations that modern SSDs handle efficiently.

References supporting these tuning approaches:

• PostgreSQL Wiki: Tuning Your PostgreSQL Server - Recommends random_page_cost = 1.1 for SSDs

• PostgreSQL Documentation: Runtime Configuration - Official guidance on cost parameters

• PostgreSQL Performance Tuning Guide - SSD optimization recommendations

My Key Discoveries

1. Query planners aren't perfect, but they're not always wrong - PostgreSQL made poor choices for most of my tests but excellent choices for simple LIMIT queries

2. Context matters enormously - Simple LIMIT queries favoured sequential scans (0.5ms vs 321ms), but LIMIT + OFFSET + filtering favoured indexes (204ms vs 436ms)

3. Index design creates trade-offs - the (location_id, warehouse_id) design optimized cross-warehouse patterns but penalized warehouse isolation

4. Cost models lag hardware - PostgreSQL's assumptions about I/O costs didn't match modern SSD and RAM characteristics in my testing

5. Testing assumptions is crucial - when queries seemed slow, testing alternative execution strategies with SET enable_seqscan = false revealed significant performance gaps

6. LIMIT queries favour sequential scans - when I only needed the first N results, sequential access often won dramatically

My Bottom Line: This inherited system wasn't "wrong" - PostgreSQL's query planner was being conservative based on outdated I/O cost assumptions. The primary key design reflected deliberate business optimization choices, and the performance gaps I observed could be addressed with supplemental indexes if needed.

Testing performed on PostgreSQL 15+ with ~2.24M rows. Results may vary based on hardware, configuration, and data distribution.