Counting sounds easy, right?
You add one number to another, maybe you keep a counter in a database or increment a value in memory. Simple.

But at Netflix’s scale, even something as basic as “counting” becomes a serious engineering challenge.

When you have millions of users streaming content every minute across thousands of servers distributed around the world, the act of “adding one more view” to a counter quickly turns into a distributed systems problem.

Let’s dive into what the problem was, why traditional methods fail, and what we can learn from how Netflix approached it.

The Problem: Counting at Netflix Scale

Imagine you are tracking how many people are watching Stranger Things right now.

Every time someone presses play, one counter somewhere needs to go up by one.
Now imagine this happening across hundreds of data centers, each handling its own share of users.

That’s millions of increments per second.

If you had just one database where all those servers sent an “increment” request (INCR), you would have:

• A massive bottleneck with too many requests hitting the same database

• Network delays and contention

• Possible data loss if that central database crashes

And worst of all, your “total view count” could become inconsistent or outdated.

Netflix engineers needed a way to count accurately, fast, and reliably, even when data was spread across thousands of machines.

Follow me on Linkedin

Why Traditional Counting Doesn’t Work

Let’s take the simple approach most of us use in smaller systems:

count = count + 1

It works beautifully on a single machine. But the moment you distribute this logic across hundreds of servers, problems begin.

1. Concurrency Issues: Two servers might try to update the same counter at the same time, causing incorrect totals

2. Latency: The more servers that try to reach the central counter, the slower it gets

3. Failure Handling: What happens if one server crashes mid-update

4. Scalability: A single counter database cannot handle billions of updates per second

Netflix realized that instead of making one machine handle all counts, they needed each machine to count locally, and then combine those counts efficiently.

Netflix’s Solution: Decentralized, Event-Driven Counting

Here’s the key insight:
Do not count everything in one place. Let every server count independently and merge those results later.

This is called decentralized counting.

Each server does its own small counting for a short time window. Then, it sends those local totals, called deltas, to a central pipeline like Kafka.

Think of it like this:

• Every store branch counts how many customers visited today

• At the end of the day, each branch sends their total number to the head office

• The head office adds all those branch numbers together to get the company-wide total

Netflix does exactly this, but continuously and in real time.

How It Works Step by Step

1. Local Counting
   Each Netflix microservice instance keeps a small in-memory counter. It increments locally whenever a new event happens, like a stream start or a like

2. Batching
   Instead of sending every single increment, each instance batches its counts for a short period, for example every few seconds

3. Publishing
   These batched counts are published to Kafka, Netflix’s distributed event streaming platform

4. Aggregation
   A specialized service consumes these delta events and aggregates them by keys, for example show ID, region, or time window

5. Storage
   The final aggregated counts are stored in a scalable database like Cassandra or Redis, which can handle huge read and write volumes

Handling Delays, Duplicates, and Failures

In distributed systems, you cannot assume everything arrives perfectly on time or only once.

Some deltas might get delayed due to network lag, and some might even get sent twice during retries.

Netflix solved this by designing their system to be idempotent, meaning even if the same update is applied twice, the result remains correct.

Each delta has:

• A unique ID to detect duplicates

• A timestamp window to know when the counts happened

• The increment value for how much to add

Even if Kafka replays an event or a message is retried, the aggregator can safely ignore duplicates and apply counts correctly.

This ensures that the final global count is accurate, even if updates come late or in random order.

In-Depth Architecture

For those who’ve stayed with us this far, let’s dive deeper into the architecture. After evaluating multiple approaches, Netflix ultimately adopted a hybrid, event-driven solution for distributed counting. This final design strikes a careful balance between low-latency reads, high throughput, and strong durability.

1. Event Logging in TimeSeries Store

Every counter increment is treated as an immutable event and ingested into Netflix’s TimeSeries abstraction, which acts as the event store. Each event record includes:

• event_time: Timestamp of the increment

• event_id: Unique identifier for idempotency

• counter_name and namespace: To identify which counter it belongs to

• delta: The increment value (positive or negative)

The TimeSeries abstraction, typically backed by Cassandra, ensures high availability, partition tolerance, and fast writes. Events are partitioned using time_bucket and event_bucket columns to prevent wide partitions that could degrade read performance.

Benefits:

• Immutable events prevent accidental data loss

• Built-in retention policies automatically purge old events

• Supports safe retries due to idempotency keys

Handling Wide Partitions: The time_bucket and event_bucket columns play a crucial role in breaking up a wide partition, preventing high-throughput counter events from overwhelming a given partition. For more information regarding this, refer to this blog.

2. Background Aggregation (Rollups)

Reading individual events for each request is expensive. To solve this, Netflix performs continuous aggregation of events in the background.

Key points:

• Rollup windows: Aggregation occurs within a defined time window to ensure consistency

• lastRollupTs: This represents the most recent time when the counter value was last aggregated. For a counter being operated for the first time, this timestamp defaults to a reasonable time in the past.

• Immutable window: Aggregation can only occur safely within an immutable window that is no longer receiving counter events. The “acceptLimit” parameter of the TimeSeries Abstraction plays a crucial role here, as it rejects incoming events with timestamps beyond this limit. During aggregations, this window is pushed slightly further back to account for clock skews.

This results in a rollup count, which represents the sum of all events in the aggregation window.

3. Rollup Storage

Aggregated counts are stored in a persistent Rollup store, again often Cassandra, with one table per dataset.

LastWriteTs: Every time a given counter receives a write, they also log a last-write-timestamp as a columnar update in this table. This is done using Cassandra’s USING TIMESTAMP feature to predictably apply the Last-Write-Win (LWW) semantics. This timestamp is the same as the event_time for the event.

• Each row contains: counter_name, lastRollupCount, lastRollupTs

• LastWriteTs is recorded for every write to track whether the counter has new events not yet aggregated

  

• Subsequent aggregations continue from the last checkpoint

Benefit: Efficient reads with only incremental aggregation needed

4. Caching Layer (EVCache)

To provide low-latency reads, the latest aggregated counts are cached in EVCache.

• Cached value = {lastRollupCount, lastRollupTs}

  

• Reads return the cached value, with a slight acceptable lag

• Triggers a rollup in the background if needed to catch up on events

This allows single-digit millisecond read latencies even at massive scale.

5. Event-Driven Rollup Pipeline

Rollups are triggered using a lightweight rollup event per counter.

rollupEvent: {
  “namespace”: “my_dataset”,
  “counter”: “counter123”
}

This event doesn’t contain the actual number that was added to the counter. Instead, it just tells the Rollup server that this counter was updated and needs to be processed. By sending these lightweight “update signals,” the system knows exactly which counters to aggregate, so it doesn’t have to go through all the events in the database every time.

In-Memory Rollup Queues: Each Rollup server keeps several queues in memory to collect rollup events and process multiple counters at the same time. Using in-memory queues makes the system simpler, cheaper, and easier to adjust if they need more or fewer queues. The trade-off is that if a server crashes, some rollup events in memory might be lost.

Minimize Duplicate Effort: Netflix uses a fast non-cryptographic hash like XXHash to ensure that the same set of counters end up on the same queue. Further, Netflix tries to minimize the amount of duplicate aggregation work by having a separate rollup stack that chooses to run fewer beefier instances.

Dynamic Batching: The Rollup server dynamically adjusts the number of time partitions that need to be scanned based on cardinality of counters in order to prevent overwhelming the underlying store with many parallel read requests.

Adaptive Back-Pressure: Each consumer waits for one batch of counters to finish processing before starting the next batch. If the previous batch took longer, the system waits a bit more before starting the next one. This helps prevent the underlying TimeSeries database from getting overloaded with too many requests at once.

Key Considerations:

• Aggregation occurs without distributed locks, using immutable windows to ensure correctness

• Counters are re-queued if new writes occur, ensuring convergence

• Stale counts self-remediate during the next read-triggered rollup

6. Idempotency and Safety

Each increment event includes event_id + event_time as a unique key. This ensures:

• Safe retries without over-counting

• Accurate aggregation across distributed consumers

• Reliable reset operations without race conditions

7. Experimental “Accurate” Counter

For use cases requiring near real-time accurate counts:

• The delta between the last rollup and current events is computed in real-time during read

• currentAccurateCount = lastRollupCount + delta

• Batched processing is still applied to avoid overwhelming the underlying TimeSeries store

This allows clients to see almost real-time counts, while the system maintains high throughput for millions of counters globally.

Putting It All Together

The final architecture combines:

1. Event logging: Every increment is stored as an immutable event in TimeSeries

2. Background aggregation: Continuous rollups ensure efficient reads

3. Rollup storage: Persistent storage for aggregated counts

4. Caching: EVCache for near-instant reads

5. Event-driven pipeline: Efficient, parallel, and idempotent rollups

6. Optional real-time delta: For “accurate” counter reads

This combination of event logging, aggregation, caching, and idempotency is what allows Netflix to:

• Handle millions of simultaneous increments

• Provide near real-time counts

• Avoid overloading any single server

• Maintain global durability and reliability

Key Design Takeaways

Netflix’s distributed counting system is a perfect example of how simple ideas need solid architecture at scale.

1. Push computation closer to data
   Let each node do part of the work instead of relying on one big system

2. Batch operations
   Do not send every single event. Aggregate small chunks to reduce load

3. Design for eventual consistency
   It is okay if data takes a few seconds to settle as long as it converges correctly

4. Use idempotent events
   Avoid double-counting when messages are retried

5. Stream, don’t store
   Use event-driven architecture for scalability and resilience

Reference: Netflix’s Distributed Counter Abstraction

Final Thoughts

I love how Netflix engineers turn “simple problems” into lessons in elegant system design.

Distributed counting teaches us that even a basic count++ can become complex when scaled across thousands of machines. With the right architecture, batching, and event streaming, it becomes not just manageable but efficient.

It is a reminder that great engineering is about scaling simplicity, not complexity.

Next time you design a system, ask yourself:

What happens when this “simple” feature needs to scale to a billion users?

That is where system design truly begins.

Today, we’re diving into a fascinating story from Netflix’s engineering team, a story that shows how even world-class systems can hit scaling walls when the use case evolves, and how a smart architectural rethink can unlock massive performance gains. In Netflix’s case, the result was a 100× improvement.

In this post, we’ll cover:

• The challenge Netflix faced with its Maestro workflow engine, and why tasks that used to run fine started slowing down

• The architecture behind the slowdown, including stateless workers, polling queues, and database overhead

• The redesign that supercharged Maestro, introducing stateful actors, in-memory workflows, task grouping, and event-driven execution

• The results and takeaways, showing how a well-thought-out system design can make even massive workflows snappy

To set the stage, imagine if every task in your system took an extra 5 seconds to start. Not a big deal, right?

Now multiply that by millions of executions a day. That’s the hidden performance tax Netflix uncovered in Maestro. And their solution is a brilliant example of architecture beating brute force.

Reference: Netflix Tech Blog — 100× Faster: How We Supercharged Netflix Maestro’s Workflow Engine

The Problem Netflix Faced

Netflix has a system called Maestro. Think of Maestro as the “task manager” that controls when and how Netflix runs thousands of background jobs — like moving data, training ML models, generating recommendations, and more.

It’s like a conductor of an orchestra, telling each musician (task) when to start, stop, or wait.

For years, Maestro worked perfectly because most jobs ran only once a day or once every few hours. A few seconds of delay didn’t matter.

But then Netflix started using Maestro for real-time use cases — like ads, games, or quick data pipelines that run every few minutes or even seconds. Suddenly, they noticed:

Every task took an extra 5–10 seconds just to start.

That was overhead inside Maestro itself — before the actual work even began. This made fast workflows feel slow and frustrated developers who had to wait on every change.

Why It Was Slow

Originally, Maestro’s design worked like this:

• A central system put tasks into a queue (like a to-do list).

• Stateless workers (servers with no memory of previous tasks) kept polling the queue:

“Do you have something for me?”

• When they got something, they loaded task details from a database, did the work, then wrote results back.

This worked great at scale, but:

• Polling every few seconds caused delays.

• Repeatedly loading and saving state slowed things further.

• Using multiple systems (queues, databases, workers) added more lag.

It’s like every time you wanted to start a task, you had to:

1. Call your manager

2. Check your email

3. Unlock your computer

4. Then finally start working.

This overhead is fine once a day, but crippling if you’re doing it every minute.

What Netflix Did to Fix It

Netflix didn’t just tweak Maestro, they redesigned its brain for real-time responsiveness.

Here’s how they did it:

1. Stateful Actors: Giving Each Task a “Memory”

Instead of stateless workers constantly asking “what next?”, each workflow or task got its own actor, a lightweight in-memory brain that:

• Lives in memory (JVM)

• Knows its next steps

• Reacts instantly to events

• Avoids repetitive database reads

Result: Tasks now start in milliseconds, not seconds.

2. Keep Data in Memory, Sync in Background

They used the database only as a backup, not as the main source for every step.
The real-time info lives in memory, super fast.

Think of it like working on a Google Doc in real-time, but everything is saved in the background. You don’t hit “Save” every time.

3. Group Tasks Logically

They split workflows into “groups” and assigned each group to a specific server.
This way, related tasks stay on the same machine, less shuffling around, faster execution.

And if a server goes down, another can take over the group.

Think of it like assigning each teacher a set of students. If one teacher is sick, another takes over their class. But students don’t keep running between classrooms unnecessarily.

4. Removed the Middleman Queue

Instead of using an external queue where workers kept asking for tasks, they:

• Used in-memory queues for instant communication inside the system, and

• A small database-backed mechanism to make sure tasks don’t get lost.

Like passing a note directly to a colleague instead of sending an email and waiting for them to check their inbox.

5. Gradual Migration with Real Testing

They built a testing framework to run real production workflows in parallel on the new system to catch bugs.
Then they moved users over gradually.

Like test-driving a new engine before swapping it into the airplane mid-flight.

The Results

The startup delay per task went from 5 seconds → 50 milliseconds.
That’s a 100× speedup.

• Developers no longer have to wait between iterations.

• Real-time workflows run smoothly.

• Maestro remains scalable but is now fast too.

Why This Matters (Even If You’re Not Netflix)

Many engineering teams optimize for scale, but ignore latency inside the system.

Polling, stateless workers, and external queues are simple to build, but add hidden costs.

By shifting to:

• Event-driven designs

• In-memory state

• Actor models

…you can unlock massive speed gains without simply throwing more servers at the problem.

Netflix’s journey is a reminder: sometimes, true performance comes from rethinking architecture, not just optimizing code.

Conclusion

Netflix’s Maestro evolution is more than just a speed hack. It’s a masterclass in system design.

When workloads change, the architecture that once worked well can become the bottleneck. By moving from a centralized, polling-based design to a decentralized, actor-driven model, Netflix made Maestro real-time ready.

For the rest of us, the takeaway is clear:
✅ Don’t underestimate internal latency.
✅ Event-driven + in-memory approaches can deliver big wins.
✅ Architectural redesign can unlock performance leaps that brute force never will.

I was deep into building a tricky feature when a junior engineer popped by. “Hey, can you help me debug this?” I thought, five minutes won’t hurt. Fifteen minutes later, I finally got back to my code, only to be dragged into an ad-hoc meeting about some random issue. While I was on the call, Slack exploded with messages from other team members. By the time I returned to my code, I couldn’t even remember what I was working on. My focus was gone.

And that’s when it hit me: I wasn’t multitasking. I was constantly switching contexts, and it was silently sabotaging my productivity.

The Real Problem: Context Switching

The human brain isn’t wired to handle two demanding tasks at the same time. Every time we switch from coding to answering questions, to attending a meeting, to replying to Slack, our focus fractures. Research suggests it can take around 25 minutes to regain full attention after an interruption.

The cost isn’t just time. Constant context switching leads to mental fatigue, slower problem-solving, and less creativity. And in a fast-paced environment, with ad-hoc meetings, junior engineers needing guidance, and endless pings, this can quietly eat away at your entire day.

How I Fixed It

Once I realized context switching was the culprit, I started experimenting with ways to protect my attention. Here’s what worked for me:

1. Prioritize Ruthlessly

Each morning, I pick 1–2 tasks that truly matter and treat them like non-negotiable appointments. Emails, bug fixes, and minor questions come later. Saying “not now” felt uncomfortable at first, but finishing meaningful work made it worth it.

2. Batch Interruptions

Instead of answering Slack messages and emails the moment they arrive, I created specific time blocks for them. Junior engineers learned I had “collaboration meetings” for questions and problem solving. I could focus without guilt, and the team still got the help they needed.

3. Time-Block Deep Work

I reserved chunks of time purely for coding or writing and blocked them on my calendar as “Focus mode“. Notifications off, door closed, deep work mode on. For the first time, I could actually get into flow and stay there.

4. Minimize Distractions

Phone on silent, unnecessary tabs closed, notifications off. These small boundaries prevented tiny interruptions from breaking my focus and stealing 15–20 minutes at a time.

5. Single-Task, Fully

I stopped juggling multiple tasks. One task, full attention. Ironically, I finished things faster and with better quality. Letting go of the “always busy” mindset was surprisingly liberating.

6. Build a Buffer for Chaos

Unexpected things still happen, urgent meetings, production bugs, random questions. But leaving small gaps between tasks absorbed the chaos instead of letting it derail the day. My stress levels dropped, and I stayed in control.

Conclusion

Multitasking might feel like a superpower, but it’s a trap. Context switching is the silent productivity killer that drains energy, focus, and creativity. Once I started protecting my attention with these habits, I finished tasks faster, felt more energized, and finally regained control over my workday.

Focus on one thing at a time, protect your flow, and let real productivity happen.

A few years ago, I found myself in a situation at work that taught me one of the most valuable lessons of my career.

I was leading a project that had tight deadlines. A colleague kept missing updates I needed from them, which started delaying my work. At first, I stayed silent—I didn’t want to come across as pushy. But the silence only made things worse. My manager asked why timelines were slipping, and I realized I was taking the hit for something I hadn’t spoken up about.

That was my turning point. I had to learn how to express my needs clearly without sounding rude. Instead of saying, “You never give me updates,” I started saying, “I need the report by Wednesday so I can finalize the deck by Friday.” That shift—from vague frustration to clear, respectful communication—changed everything. My colleague responded better, deadlines stopped slipping, and I felt a sense of control return to my work.

That’s the power of assertiveness. It’s not aggression, and it’s not silence. It’s about communicating in a way that values both your perspective and the other person’s.

In today’s fast-paced work environments, technical skills alone aren’t enough to thrive. Whether you’re presenting ideas to stakeholders, negotiating deadlines, or collaborating with peers, how you communicate makes a world of difference.

1. Understand Assertiveness (and How It Differs from Aggression)

• Aggressive communication pushes your perspective at the cost of others.

• Passive communication avoids conflict but often leaves you unheard.

• Assertive communication strikes the balance: it’s direct, clear, respectful, and solution-focused.

For example, instead of writing:
“You never respond to my emails.”
Say:
“I had sent you an email last week regarding the project estimation. Could you share the details by today so I can finalize the report?”

This way, you’re stating facts, setting expectations, and staying professional

2. Use the Pyramid Principle for Clarity

When making your point, avoid burying the key message under unnecessary details. Instead, follow a top-down structure:

• Answer (Assertion): State the most important idea upfront.

• Argument: Support it with logical reasoning.

• Evidence: Back it up with data.

For example, instead of:
“We should consider using the new tool because it might help some processes, though I know there’s resistance.”
Say:
“The new tool can save employees 10 hours a week (Answer). It automates repetitive workflows (Argument), and three other teams have already reported efficiency gains (Evidence).”

3. Master Assertiveness Tools

The second deck introduces practical conversational techniques you can use when facing resistance.

• Broken Record: Calmly repeat your request without getting sidetracked.

• Fogging: Acknowledge the valid part of criticism without absorbing exaggerations.

• Negative Inquiry: Ask questions to clarify vague criticism (e.g., “Could you tell me what you’d like improved in my emails?”).

• Positive No (Yes-No-Yes): Say no respectfully by affirming the other person’s needs, stating your refusal, and ending with a constructive alternative.

Example:
“I understand this feature is important to you (Yes). However, we can’t deliver all ten changes in this release (No). What we can do is prioritize the top three and revisit the rest next sprint (Yes).”

4. Be Specific and Sensitive

Generalizations like “You’re always late” or “This timeline is impossible” spark defensiveness. Assertiveness relies on specific facts and empathy:

• Instead of “He is impatient”, say “He asked for updates five times in two hours yesterday, which disrupted the flow.”

• Instead of “The client is demanding”, say “The client requested five design changes within two days.”

5. Say “No” Without Burning Bridges

Many professionals struggle with saying no because they fear being seen as uncooperative. But over-committing leads to stress and poor delivery.

The Yes-No-Yes approach helps:

• Yes: Show understanding (“I know this project is important to you”).

• No: State your boundary firmly (“We can’t add these changes to this release”).

• Yes: Offer a constructive way forward (“We can include them in the next sprint”).

6. Match Your Communication to Context

Assertiveness also means being mindful of when and how you communicate:

• Avoid sending emotional or confrontational emails—opt for a call or face-to-face chat.

• Don’t schedule heavy discussions at times when people are mentally checked out (like late Friday evenings).

• Keep your language human and jargon-free to ensure clarity

7. Build Influence Without Authority

Being assertive isn’t just about voicing your needs—it’s about influencing others while respecting their perspectives. Use the six principles of persuasion to strengthen your stance:

• Reciprocity: Offer help before asking for it.

• Authority: Share expertise to earn credibility.

• Social Proof: Highlight how others have adopted your idea.

• Scarcity: Emphasize opportunities that may not last.

• Consistency: Link your request to past commitments.

• Liking: Build rapport so people want to say yes

Final Thoughts

Assertiveness is not about being the loudest voice in the room. It’s about being clear, confident, and considerate. When you communicate assertively, you:

• Earn respect without creating resentment

• Manage conflicts productively

• Influence decisions even without formal authority

• Create stronger, more collaborative workplaces

Start small—whether it’s rewriting one email to be more assertive, practicing a Positive No in your next meeting, or using the pyramid principle in your presentations. Over time, assertiveness becomes less of a skill you practice and more of a professional habit that elevates your impact.

A few years back, I was working on a system that sent welcome emails to new users during sign-up. The design was simple: a user performed an action, an event was published to a queue, a consumer picked it up and called the SendGrid API to deliver the email.

One day, we hit an unexpected snag. Our consumer was calling the SendGrid API just fine, but due to a small bug, it never marked the event as processed (acknowledged). The queue assumed the event wasn’t handled, so after the ack deadline expired, it redelivered the same event. The consumer retried the call, SendGrid happily sent the email again, and this loop kept repeating.

To make matters worse, we only had a single consumer instance. That one stuck event essentially blocked the entire queue. What looked like a harmless bug turned into dozens of duplicate emails and a completely clogged pipeline.

This kind of issue is known as the “poison pill” problem: one bad or unacknowledged message keeps coming back, poisoning the queue, and preventing healthy messages from being processed.

From Poison Pills to Dead Letter Queues

In this post, we’ll see how to stop a “poison pill” from poisoning our system. But before diving into the technical details, let’s imagine a simple real-world queue.

Imagine you’re at a bank where people line up to deposit checks. Each person hands their check to the cashier, who processes it one by one.

Now, if someone hands in a torn or illegible check, the cashier can’t process it. If they just keep holding on to that bad check, the entire line behind them comes to a halt.

A smarter system would be: put that bad check aside in a separate tray, let the rest of the line move forward, and later have a manager review the bad check to see what went wrong.

That’s exactly what a Dead Letter Queue (DLQ) does:

• If a message in the queue can’t be processed (like the invalid check), it gets moved aside into a special holding area (the DLQ).

• The rest of the messages keep flowing smoothly.

• Later, engineers can look at the failed messages in the DLQ, fix them, and reprocess them if needed.

Designing the Dead Letter Queue

Now that we know what a Dead Letter Queue (DLQ) is, the real question is: how do we actually implement one?

Before jumping into code, it’s important to step back and design the mechanics carefully. A DLQ isn’t just a “dumping ground” for failed messages — it’s a deliberate safety net that requires us to make a few key decisions:

1. Identify failure modes

   • What kinds of errors can occur in your system?

   • Are they transient (like a temporary network blip or API timeout) or permanent (like malformed data or a missing required field)?

2. Define failure handling strategy

   • Which types of errors should trigger a retry?

   • Which should immediately route the message to the DLQ?

3. Set retry policies

   • How many times should a message be retried before we officially “give up”?

   • Should retries follow exponential backoff, fixed intervals, or custom logic?

4. Decide on metadata for DLQ messages

   • What extra information should we attach when moving a message to the DLQ? (e.g., error type, stack trace, retry count, correlation ID).

   • This metadata is crucial for debugging and reprocessing later.

5. Establish a “poison message” threshold

   • At what point do we stop retrying and mark a message as poisonous?

   • Once it’s in the DLQ, we don’t want it re-entering the main processing loop automatically and causing the same failure cycle.

Key Failure Modes and DLQ Strategies

Once we’ve defined what a DLQ should capture, the next step is to map out the different failure modes that can occur in a Pub/Sub-based system. Each type of failure needs its own handling strategy so that messages don’t get stuck in endless loops.

1. Consumer Failure Modes

When a Pub/Sub consumer cannot process a message:

• Ack deadline: If the consumer doesn’t acknowledge within the ack deadline (default 10s, extendable to 600s), Pub/Sub assumes failure and redelivers the message.

• Retries: Pub/Sub keeps retrying until the message is acknowledged or until it reaches the maximum delivery attempts (if a DLQ is configured).

• Poison messages: Without a DLQ, one bad message can cycle forever, blocking or delaying other messages.

Strategies

• Use idempotent consumers to make retries safe and avoid duplicate side effects (e.g., sending duplicate emails).

• Implement ack deadline extensions for long-running jobs.

• Configure a DLQ with max delivery attempts to prevent poison loops.

2. API Call Failures Inside Consumer

A common failure scenario is when your consumer depends on an external API (e.g., SendGrid, payment gateway) and that API either times out or returns a 5xx error.

Best practices

• Use exponential backoff with jitter to retry API calls locally before giving up.

• Apply circuit breakers to stop hammering unstable dependencies.

• Ensure all external calls use idempotency keys so retries don’t cause duplicates.

• If retries exceed the configured limit, route the message to DLQ.

3. Non-Recoverable Application Errors (e.g., NPE)

Not all failures are retryable. Some represent logic or data issues that will never succeed no matter how many times you retry:

• Schema mismatch (e.g., invalid JSON).

• Missing required fields in the payload.

• Null pointer exceptions or bugs in business logic.

In such cases:

• A DLQ helps capture the bad payload for later analysis.

• But fixing the code (or correcting the data source) is mandatory before replaying.

• Use a DB error log table to persist stack traces, consumer version, and raw payloads for debugging.

4. Pub/Sub DLQ vs. Error Log Table in a Database

When deciding how to capture failed messages, teams usually debate between using:

1. Pub/Sub Dead Letter Queues (DLQs)

2. A database-backed error log table

Both approaches have strengths and trade-offs.

Pub/Sub DLQ (Dead Letter Queue)

Pros

• Automatically supported by Pub/Sub — easy to configure.

• Handles high-throughput failures without manual scaling.

• Allows attaching message attributes as metadata.

• Messages can be re-subscribed to and reprocessed later.

Cons

• Limited querying/filtering — replaying or analyzing requires additional tooling.

• Costs grow with message volume and retention.

Error Log Table (e.g., Cloud SQL / PostgreSQL)

Pros

• Full control: you can store payload + metadata + stack trace + retry count + consumer version.

• Easy to query/filter (e.g., “show me all errors for consumer v2.1 in the last 24h”).

• Can enrich with business metadata (user ID, category, severity).

Cons

• Requires schema design, storage management, and indexing.

• Not built for massive throughput unless carefully optimized.

• Reprocessing requires custom scripts/tools.

Rule of thumb: Use Pub/Sub DLQ for operational resilience and quick retries, and use an error log DB for observability and analytics.

5. Republishing / Re-consuming Events

A DLQ is only useful if you can reprocess messages safely. Here’s how it works in both approaches:

From Pub/Sub Dead Letter Topic

• Messages sit in a separate subscription.

• Use gcloud pubsub subscriptions pull or a custom consumer to fetch them.

• After analysis/fix, republish back to the main topic.

• Use idempotency keys to avoid re-triggering duplicates.

From Error Log Table

• Query failed rows (e.g., based on error category or timestamp).

• Batch re-publish messages via a script or tool.

• Ensure the re-publish process:

  • Updates a reprocessed_at timestamp.

  • Prevents double-sending if the same row is picked up again.

6. Hybrid Approach

In practice, many production systems use a hybrid strategy:

• Pub/Sub DLQ handles high-throughput retries and keeps the main pipeline healthy.

• Error Log Table provides deep visibility into why failures occurred, enriched with stack traces, error categories, and metadata for debugging.

Recommended flow:

Explaining the System Flow

This flowchart represents the end-to-end lifecycle of a message in a Google Pub/Sub–based system with Dead Letter Queues (DLQs) and a Parked Letter Queue (PLQ) or the error table.

Let’s walk through each step:

1. Message Entry – Pub/Sub Topic

• Everything begins with the Pub/Sub Topic.

• Producers publish events here (e.g., “user signup,” “order created”).

• Messages are then delivered to one or more Consumers for processing.

2. Consumer Processing

• The Consumer is the application logic that processes each message.

• Example: validating the payload, calling an external API, saving to DB, etc.

• Two outcomes are possible:

  • Success: Message is acknowledged and marked as Processing Complete.

  • Failure after retries: Message is handed off to the Dead Letter Topic (DLQ).

3. Dead Letter Topic (DLQ) – Handling Transient Errors

• The DLQ captures messages that couldn’t be processed even after retries.

• These are typically transient issues, such as:

  • External API outages

  • Temporary DB lock

  • Network timeouts

• To avoid losing data, we don’t discard these messages immediately. Instead:

  • A Reprocessor Service picks them up later and republishes them to the main topic for another attempt.

  • If successful, they continue as normal through the pipeline.

4. Parked Letter Queue (PLQ) – Handling Persistent Failures

• If a DLQ message still fails even after reprocessing, it moves to the Parked Letter Queue (PLQ) or in this case, an error table.

• The PLQ is the “quarantine area” for persistent or poison messages that won’t succeed automatically.

• We can enhance the DLQ reprocessor to attach a custom attribute indicating how many times a message has landed in the DLQ. This way, the consumer application can use that metadata to decide when to route the message to the error log table.

• Example causes:

  • Schema mismatch (e.g., missing required field)

  • Invalid payload (malformed JSON, bad encoding)

  • Application bug (NullPointerException)

5. Persistence and Analytics

• Every message that enters the PLQ is persisted in an Error Log Table.

• This table contains:

  • Raw payload

  • Error category

  • Stack trace or exception details

  • Retry count and consumer version

• From here, the data flows into BigQuery for analytics and dashboards.

  • Teams can analyze trends (e.g., “What % of errors are API failures vs. schema mismatches?”).

  • Helps prioritize fixes and monitor long-term stability.

6. Manual Intervention

• Engineers can manually inspect PLQ messages.

• Based on investigation, they have two options:

  • Reprocess: Publish back into the Pub/Sub Topic once the root cause is fixed.

  • Discard: If the message is invalid and cannot be salvaged.

• This ensures no data is lost silently and every failure path has a resolution.

Implementation: Building a True DLQ Pattern with Google Pub/Sub

Overview

This basic implementation would demonstrate a true Dead Letter Queue pattern using Google Pub/Sub, where:

• ALL failures go to DLQ first (not directly to PLQ)

• DLQ Reprocessor adds retry count metadata

• Main Consumer makes PLQ decisions based on retry count

• Manual intervention available via web dashboard

Key Principle

Message → Consumer → Fail → DLQ → Reprocessor → Add Retry Count → Main Topic → Consumer → Check Retry Count → If ≥3 → PLQ (Database)

Prefer to skip the explanation and go straight to the implementation? The code is available here.

1. Producer Service

Location: producer-service/app.py

The producer publishes messages to the main topic:

@app.route('/publish', methods=['POST'])
def publish_message():
    """Publish a single message to the main topic"""
    try:
        data = request.get_json()
        
        # Add metadata
        message_data = {
            **data,
            "timestamp": datetime.now().isoformat(),
            "producer_version": "1.0"
        }
        
        # Publish to main topic
        message_id = pubsub_manager.publish_message(MAIN_TOPIC, message_data)
        
        return jsonify({
            "message_id": message_id,
            "status": "published", 
            "timestamp": datetime.now().isoformat(),
            "topic": MAIN_TOPIC
        })

Key Features:

• REST API for message publishing

• Automatic timestamp metadata

• Sample message generation for testing

2. Consumer Service (Core DLQ Logic)

Location: consumer-service/app.py

This is the heart of the DLQ pattern implementation:

def process_message(self, message_data: Dict[Any, Any], message_id: str, 
                   correlation_id: str = None, dlq_retry_count: int = 0) -> bool:
    """Process message with DLQ retry count awareness"""
    try:
        logger.info(f"Processing message {message_id}, type: {message_data.get('type')}, "
                   f"dlq_retry_count: {dlq_retry_count}")
        
        # KEY LOGIC: Check if message exceeded DLQ retry limit
        if dlq_retry_count >= 3:  # Max DLQ retries
            logger.info(f"Message {message_id} exceeded DLQ retry limit, sending to PLQ")
            self._log_to_plq(
                message_data, message_id, 'dlq_max_retries_exceeded', 
                f'Message failed after {dlq_retry_count} DLQ retries',
                correlation_id=correlation_id, dlq_retry_count=dlq_retry_count
            )
            return True  # Acknowledge the message (handled via PLQ)
        
        # Check for simulated failures
        failure_type = simulate_failure(message_data)
        if failure_type:
            logger.warning(f"Processing failed ({failure_type}), will go to DLQ")
            # ALL failures go to DLQ (not directly to PLQ)
            return False
            
        # Normal processing...
        return True
        
    except Exception as e:
        # Handle unexpected errors
        if dlq_retry_count >= 3:
            self._log_to_plq(message_data, message_id, error_type, error_message)
            return True
        return False  # Let it go to DLQ for retry

Message Callback extracts retry count from attributes:

def callback(self, message):
    """Handle incoming Pub/Sub message"""
    try:
        message_data = json.loads(message.data.decode('utf-8'))
        message_id = message.message_id
        correlation_id = message.attributes.get('correlation_id')
        
        # Extract DLQ retry count from message attributes
        dlq_retry_count = int(message.attributes.get('dlq_retry_count', 0))
        
        # Process with retry count awareness
        success = self.process_message(
            message_data, message_id, correlation_id, dlq_retry_count
        )
        
        if success:
            message.ack()  # Successfully processed
            logger.info(f"Message processed successfully: {message_id}")
        else:
            message.nack()  # Send to DLQ
            logger.warning(f"Message nacked, will go to DLQ: {message_id}")

3. DLQ Reprocessor (Retry Count Manager)

Location: dlq-reprocessor/app.py

The reprocessor adds retry count metadata and republishes:

def reprocess_message(self, message_data: Dict[Any, Any], message_id: str, 
                     correlation_id: str = None, dlq_retry_count: int = 0) -> bool:
    """Reprocess a message from DLQ with retry count tracking"""
    try:
        logger.info(f"Reprocessing message {message_id} (DLQ retry: {dlq_retry_count})")
        
        # Check if should reprocess
        should_reprocess, reason = self.should_reprocess(message_data, dlq_retry_count)
        if not should_reprocess:
            logger.info(f"Message not eligible: {reason}")
            return True
        
        # Enrich message with metadata
        enriched_message = self.enrich_message_for_reprocess(message_data, dlq_retry_count)
        
        # Apply exponential backoff delay
        delay = self.reprocess_delay * (2 ** dlq_retry_count)
        time.sleep(min(delay, 60))  # Cap at 60 seconds
        
        # KEY: Add DLQ retry count to message attributes
        attributes = {
            'correlation_id': correlation_id or f"dlq_reprocess_{message_id}",
            'dlq_retry_count': str(dlq_retry_count + 1),  # Increment retry count!
            'reprocess_timestamp': datetime.now().isoformat(),
            'from_dlq': 'true'
        }
        
        # Republish to main topic with retry count
        new_message_id = self.pubsub_manager.publish_message(
            self.main_topic, 
            enriched_message,
            attributes  # This adds the retry count header!
        )
        
        logger.info(f"Message {message_id} republished as {new_message_id} "
                   f"with dlq_retry_count={dlq_retry_count + 1}")
        return True

4. Error Storage (PLQ Implementation)

Location: support/database.py

Failed messages are stored in PostgreSQL for manual intervention:

def log_error(self, message_id: str, message_data: dict, error_type: str, 
              error_message: str, stack_trace: str = None, 
              correlation_id: str = None, dlq_retry_count: int = 0):
    """Log error to PLQ database"""
    try:
        query = """
        INSERT INTO error_log 
        (message_id, message_data, error_type, error_message, stack_trace, 
         correlation_id, dlq_retry_count, created_at)
        VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
        """
        
        with self.get_connection() as conn:
            with conn.cursor() as cursor:
                cursor.execute(query, (
                    message_id,
                    json.dumps(message_data),
                    error_type,
                    error_message, 
                    stack_trace,
                    correlation_id,
                    dlq_retry_count,  # Track retry count in database
                    datetime.now()
                ))
                conn.commit()

5. Failure Simulation

Location: support/common.py

Realistic failure scenarios for testing:

def simulate_failure(message_data: Dict[Any, Any]) -> Optional[str]:
    """Simulate various failure scenarios for DLQ testing"""
    
    # Check for explicit failure simulation
    fail_simulation = message_data.get('fail_simulation')
    if fail_simulation:
        return fail_simulation
    
    # Simulate random failures based on message content
    email = message_data.get('email', '')
    if email and not '@' in email:
        return 'validation_error'
    
    message_type = message_data.get('type')
    if message_type == 'malformed_message':
        return 'schema_error'
    
    # Random API timeout (10% chance)
    if random.random() < 0.1:
        return 'api_timeout'
    
    return None  # No failure

Testing Guide

Step 1: Start the System

# Start all services
./start-system.sh

# Verify services are running
docker-compose ps

Expected Output:

NAME                                STATUS                      PORTS                                
dlq-test-consumer-service-1   Up X minutes                                                     
dlq-test-dlq-reprocessor-1    Up X minutes                                                     
dlq-test-postgres-1           Up X minutes (healthy)     0.0.0.0:5432->5432/tcp               
dlq-test-producer-service-1   Up X minutes               0.0.0.0:8080->8080/tcp               
dlq-test-pubsub-emulator-1    Up X minutes (healthy)     0.0.0.0:8085->8085/tcp               
dlq-test-web-dashboard-1      Up X minutes               0.0.0.0:3000->3000/tcp

Step 2: Test Message Publishing

Test 1: Valid Message (Should Succeed)

curl -X POST http://localhost:8080/publish \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "user_signup",
    "email": "test@example.com", 
    "name": "Test User"
  }'

Expected Response:

{
  "message_id": "1",
  "status": "published",
  "timestamp": "2025-09-11T18:42:31.069694",
  "topic": "user-events"
}

Test 2: Invalid Message (Should Fail → DLQ)

curl -X POST http://localhost:8080/publish \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "user_signup",
    "email": "invalid-email",
    "name": "Invalid User",
    "fail_simulation": "validation_error"
  }'

Test 3: Batch Test Messages

curl -X POST http://localhost:8080/publish/samples

Expected Response:

{
  "description": "Published sample messages including some that will fail for DLQ testing",
  "messages": [
    {"message_id": "3", "type": "user_signup", "will_fail": false},
    {"message_id": "4", "type": "user_signup", "will_fail": true},
    {"message_id": "5", "type": "user_signup", "will_fail": true}
  ],
  "published_count": 5
}

Step 3: Monitor DLQ Flow

Check Consumer Logs

docker-compose logs -f consumer-service

Expected Output:

consumer-service-1  | INFO - Processing message 1, type: user_signup, dlq_retry_count: 0
consumer-service-1  | WARNING - Processing failed (validation_error), will go to DLQ, message_id: 2
consumer-service-1  | WARNING - Message nacked, will go to DLQ: 2

Check DLQ Reprocessor Logs

docker-compose logs -f dlq-reprocessor

Expected Output:

dlq-reprocessor-1  | INFO - Reprocessing message 2 (DLQ retry: 0)
dlq-reprocessor-1  | INFO - Message 2 republished as 8 with dlq_retry_count=1

Check Consumer Processing Retry

docker-compose logs -f consumer-service | grep "dlq_retry_count"

Expected Output:

consumer-service-1  | INFO - Processing message 8, type: user_signup, dlq_retry_count: 1
consumer-service-1  | INFO - Processing message 9, type: user_signup, dlq_retry_count: 2
consumer-service-1  | INFO - Message exceeded DLQ retry limit, sending to PLQ

Step 4: Verify PLQ Storage

docker-compose exec postgres psql -U dlq_user -d dlq_system

-- Check error logs
SELECT id, message_id, error_type, dlq_retry_count, created_at 
FROM error_log 
ORDER BY created_at DESC 
LIMIT 10;

-- Check PLQ messages (retry count >= 3)
SELECT message_id, error_type, dlq_retry_count 
FROM error_log 
WHERE dlq_retry_count >= 3;

Step 5: Web Dashboard Testing

1. Open Dashboard: http://localhost:3000

2. View Error Statistics: Check failed message counts

3. Analytics Page: http://localhost:3000/analytics

4. Manual Intervention: Click on failed messages for details

Step 6: Comprehensive Test Script

# Run all test scenarios
./test-dlq-scenarios.sh

Access the code here: Github

Summary

The poison pill problem shows how a single unprocessed message can bring down an entire pipeline. Dead Letter Queues (DLQs) provide a safety net by isolating failed messages so the rest of the system continues running smoothly.

In this post, we covered:

• What DLQs are and how they solve the poison pill problem.

• Design decisions around retries, failure modes, metadata, and thresholds.

• Strategies for handling transient vs. permanent failures.

• Trade-offs between Pub/Sub DLQs and database error tables.

• Hybrid approach for combining operational resilience with deep visibility.

• Implementation details with a true DLQ pattern in Google Pub/Sub, complete with reprocessing, retry counts, and error logging.

By designing DLQs carefully, you ensure that your system is:

• Resilient: no single message can clog the pipeline.

• Visible: errors are captured and analyzed.

• Flexible: transient errors can be retried, permanent ones quarantined.

• Controllable: engineers always have the final say through manual intervention.

Dead Letter Queues aren’t just about “dumping” bad messages, they’re about building robust, fault-tolerant pipelines that can gracefully handle the unexpected.

It was 2:00 AM when one of our downstream services pinged us:

“Our system says a lane exists, but your APIs disagree. Which one’s right?”

Slack was on fire. Kafka showed a brand-new lane between two facilities, but when someone queried our service, the database claimed nothing had changed. Downstream systems were already processing this phantom lane.

That was the night we learned the hard way about dual transactions.

In this post, I’ll walk you through:

• The problem with dual transactions: why Postgres and Kafka can easily fall out of sync.

• How the Outbox Pattern works: the schema, publisher design, and flow.

• Real-world debugging challenges we faced: stuck rows, duplicates, backlogs, and how we fixed them.

• The role of BigQuery: how we used it for observability, debugging, and analytics.

• Trade-offs and best practices: where Outbox shines, where it doesn’t, and a checklist if you want to adopt it.

• Business impact: how we went from weekly 2 AM escalations to zero.

The World We Lived In

Our service was the source of truth for all facilities: warehouses, carrier hubs, depots, cross docks, suppliers. We also managed lanes, the entities connecting facilities.

The rules were simple but strict:

• If a facility’s schedule changed, downstream needed to know.

• If a lane opened or closed, downstream needed to know.

We used Postgres as the database and Kafka as the event bus. Updates had to land in both. And that’s where the nightmare began.

Dual transactions, Dual Headaches

Two failure modes haunted us:

1. DB succeeds, Kafka fails

   • Data saved in Postgres.

   • Event never published.

   • Downstream blind to reality.

2. Kafka succeeds, DB rolls back

   • Event goes out.

   • DB rejects transaction.

   • Downstream believes a lie.

Both cases left us with corrupt, unsynced worlds. We had dashboards in red, and multiple “let’s sync the facility and lanes data” firefights.

At our peak, this caused 4-5 escalations per week. After fixing it, escalations dropped to zero.

Enter the Outbox Pattern

Instead of trying to write to two systems atomically (which distributed systems hate), we decided to only trust one write path: the database.

Here’s how:

Write Once

• Every DB transaction that modifies business data also inserts an event into an outbox table.

• If the DB commit succeeds, both data + outbox entry are durable.

CREATE TABLE facility_outbox (
    id BIGSERIAL PRIMARY KEY,
    aggregate_type VARCHAR(50),
    aggregate_id UUID,
    event_type VARCHAR(50),
    payload JSONB,
    created_at TIMESTAMP DEFAULT now(),
    processed BOOLEAN DEFAULT false
);

Query-based CDC

• A lightweight publisher job polls this outbox table.

• It picks unprocessed rows, marks them, then pushes them to Kafka.

SELECT * FROM facility_outbox
WHERE processed = false
FOR UPDATE SKIP LOCKED
LIMIT 100;

Mark as Processed

• After a successful publish, mark rows as processed = true.

• If publish fails, retry safely.

Now the DB was the single source of truth, and Kafka simply reflected it. No more split-brain.

Architecture at a Glance

Here’s what the flow looked like after Outbox adoption:

Debugging Nightmares We Solved

This wasn’t just plug-and-play. We hit plenty of edge cases:

1. Stuck Rows

• Sometimes publishers crashed mid-batch.

• Rows got locked forever.

• Fix: SKIP LOCKED ensured other workers could move on.

2. Duplicate Events

• Retries led to re-publishing.

• Solution: consumers became idempotent, keyed by event_id.

3. Slow Publishers

• Outbox grew faster than we could drain it.

• Fix: batch publishes + horizontal scaling of publishers.

4. Backfill Hell

• New systems wanted all historical events.

• Answer: replay directly from the outbox table → no custom scripts.

Why BigQuery Sink?

We weren’t just worried about delivery—we also needed observability.

• Every Kafka topic had a BigQuery sink.

• This let us:

  • Run ad-hoc queries: “Show all lane updates in the last 6h.”

  • Debug mismatches: compare DB state vs. published events.

  • Prove SLAs: how fast events left the outbox.

It was our black box recorder for event history.

Trade-offs We Faced

The Outbox pattern isn’t a silver bullet. We had to weigh:

• Extra DB Load: every change means an extra insert.

• Latency: events are near-real-time, not instant (polling adds ~200–500ms).

• Storage Growth: outbox table can bloat if not purged/archived.

• Operational Complexity: need monitoring, retries, and cleanup policies.

But for us, consistency > absolute latency. We’d rather downstream be slightly late than totally wrong.

When to (and NOT to) Use Outbox

Use Outbox if…

• Your service is a source of truth and must publish events reliably.

• You’re dealing with critical entities (orders, payments, facilities, lanes).

• Dual writes would cause corruption.

Don’t bother if…

• Events are non-critical logs/metrics.

• Latency requirements are microseconds.

• You already have a rock-solid streaming CDC pipeline like Debezium.

Checklist for Adopting Outbox

1. Design outbox schema (include metadata + payload).

2. Use FOR UPDATE SKIP LOCKED to avoid stuck workers.

3. Make consumers idempotent.

4. Add monitoring: queue size, lag, failed publishes.

5. Decide retention: archive or delete processed rows.

6. (Optional) Sink to BigQuery or warehouse for debugging.

Final Thought

After moving to Outbox, our 2 AM “facility lane mismatch” alerts vanished.
No more phantom lanes. No more split-brain facilities.

The system wasn’t perfect—there was extra DB load, and events weren’t instant. But consistency beat chaos.

And honestly? With Outbox in place, the only thing waking us up at 3 AM was the baby, not broken events. 👶✨

I still remember the moment clearly. I was in the middle of presenting a design for a new event-driven system. I had my architecture diagram up, services, topics, retries, and yes, a neat little Dead Letter Queue (DLQ) sitting off to the side.

One of the staff engineers raised a hand and asked me a simple question:

“Once a message is sent to the DLQ, then what? How do you actually know what the issue is?”

For a second, I froze. I had accounted for retries. I had thought about scaling. I had made sure failures wouldn’t break the main pipeline. But I realized I hadn’t thought deeply about what happens after something lands in the DLQ.

That question stuck with me. Because the truth is, in many system designs, a DLQ is treated like a checkbox. We draw it in the diagram, it gives a sense of safety, and we move on. But if you’ve ever actually operated such a system, you know that a DLQ is not the end of the story—it’s the beginning of one.

The Day We Lost Data Quietly

Not long after that presentation, I had my wake-up call.

One of our downstream services had made a schema change. The main queue retried the failing messages, but the consumers could not process them anymore. Eventually, those messages were moved into the DLQ.

And then nothing happened.

There was no alert. There was no dashboard entry calling attention to it. The DLQ sat quietly in the background, collecting failed messages that nobody was watching.

Two weeks later, those messages expired because in AWS SQS, DLQs automatically purge messages after fourteen days. By the time someone noticed missing data in a downstream report, it was gone forever. We had no way of knowing what the payloads were, no trace IDs, no metadata. Just silence.

That incident drove home a painful lesson: DLQs do not save you. They only delay the inevitable. Unless you monitor them and act, they become silent data graves.

So, What Is a DLQ?

Think of a DLQ like the “Lost and Found” counter at an airport. When baggage gets misplaced, it doesn’t vanish, it shows up at the counter. But if no one ever checks the counter, or if the bag doesn’t have a proper tag, it’s almost as good as lost.

That’s what happens in many systems. Messages that can’t be processed get pushed into the DLQ. They sit there quietly, like unclaimed baggage, until someone eventually notices there’s a growing pile of them. By then, the real issue, bad data, a downstream failure, or a misconfiguration, may have already caused damage.

So the real question isn’t do you have a DLQ? It’s what do you do with it?

The Questions We Rarely Ask

Looking back, I realized that my original design doc never asked the right questions:

• What happens when a message actually lands in the DLQ?

• Do we consume and log it anywhere permanent?

• Should metadata, like error type, trace ID, and timestamp be attached so engineers can diagnose the cause?

• Who owns the DLQ, and how quickly should they act when it fills up?

• How do we prevent silent expiration?

Without answers, the DLQ is just a checkbox. It exists, but it does not help.

Poison Pills and Infinite Loops

Later, I ran into another problem: the poison pill message.

This is the message that can never succeed. Perhaps the payload is malformed beyond repair, or the downstream service rejects it permanently.

At one point, we had such a poison pill bouncing endlessly between the main queue and the DLQ because of a naive “automatic redrive” process. Every time it was replayed, it failed in the same way.

Instead of healing the system, our DLQ had turned into a loop of repeated failure. That was the moment I realized: redrives must never be blind. You need context. You need human eyes or smart triage before you send a message back into the system.

Turning DLQs Into Debugging Tools

The breakthrough came when we stopped treating DLQs as storage and started treating them as debugging pipelines.

Instead of dumping raw payloads, we enriched every DLQ message with structured metadata:

• Error category (validation error, downstream failure, business rule violation)

• Originating service

• Retry count

• Trace or correlation ID

• Timestamp

Now, when a message appeared in the DLQ, engineers could immediately understand why it was there. We had transformed the DLQ from a black box into a window into system health.

From DLQ to PLQ: A Permanent Record

Even with metadata, DLQs came with another limitation: expiration.

We solved this by creating what we called a Parked Letter Queue (PLQ), a database-backed error log that stored all failed messages, enriched with context.

The way I like to describe it is that the DLQ is like the emergency room: temporary, chaotic, and focused on urgent triage. The PLQ is the hospital’s archive: a permanent record of every case.

With the PLQ, we finally had:

• A searchable history of failures

• An audit trail for debugging and compliance

• The ability to identify recurring failure patterns over time

That single addition meant we no longer woke up to missing data with no explanation.

A Simple Analogy

Imagine a conveyor belt in a factory. Most items pass through fine, but occasionally one is defective. Instead of halting the belt, the item is pushed into a reject bin, that is your DLQ.

Now imagine if no one ever checks the reject bin. Products keep disappearing, and nobody knows why.

But if every defective item is labeled with why it failed, which machine broke, and at what time, and those labels are logged in a permanent book, you start to see patterns. Maybe Machine A jams every Monday. Maybe Supplier X always ships faulty parts. Suddenly, the reject bin is not just waste; it is a source of insight.

That is how DLQs should be treated.

Designing DLQs That Actually Help

From these experiences, I realized that a DLQ isn’t just a “safety bucket”, it’s a tool for learning, accountability, and resilience. But that only works if it’s designed with intention. Here are some principles that have stuck with me:

1. Monitor them like you mean it.
A DLQ that no one watches is just a black hole. Track not only how many messages land there (depth), but also how long they’ve been sitting (age) and how fast new ones are arriving (throughput). A sudden spike could mean a broken deployment. A slow accumulation might indicate a silent data quality issue. Alerts on these metrics often become your first signal that the system is quietly burning.

2. Enrich messages with metadata.
A raw payload alone is often meaningless at 3 AM when you are debugging. Include trace IDs, service names, timestamps, error codes, or even a short reason for failure. Think of it as leaving breadcrumbs for your future self (or the unlucky teammate on call). The easier it is to reconstruct the story of “what went wrong,” the faster you can fix it.

3. Redrive with discipline, not desperation.
The temptation is to just shovel DLQ messages back into the main queue once the pipeline looks healthy again. That’s risky. If the root cause hasn’t been fixed, you’re just replaying the same movie. Instead, validate fixes, test carefully, and only then reintroduce messages. Redriving should be a controlled process, not a panic button.

4. Consider a PLQ or permanent error log.
DLQs often have retention limits. If you don’t act in time, your evidence just expires. A Parked Letter Queue (PLQ) or dedicated error store solves this by keeping failed messages along with all the metadata, forever if needed. It becomes the system’s diary of failures, a reliable record you can revisit for audits, debugging, or even training better validation rules.

Final Reflection

That simple question during my design presentation, “Once a message is sent to the DLQ, then what?” was one of the most valuable lessons of my career.

DLQs are not the end of the story. They are the beginning of an investigation. They are not insurance policies; they are mirrors reflecting where your system is fragile.

The real measure of resilience is not whether you have a DLQ. It is whether you know what is inside it, and whether you are doing something about it.

If you would like me to share a guide on how to implement DLQs effectively, whether on AWS SQS, Google Pub/Sub, or Kafka, drop a comment below.

Let’s be honest, most of us have faced that moment of frustration when technology doesn’t behave the way we expect. Maybe you clicked “Pay” on a checkout page, the internet blinked, and you weren’t sure whether you should hit the button again. Or maybe you submitted a form twice because the first time seemed to fail, only to realize later that your action got recorded multiple times.

These little glitches don’t just annoy us as users; they can create big headaches for the systems and teams behind the scenes. Duplicate payments, repeated orders, multiple job submissions, these are not just bugs, they’re trust-breakers. When users lose confidence that your system will “do the right thing”, they start doubting whether they should use it at all.

This is where idempotent keys, a concept Stripe brought into the spotlight, come in. And trust me, it’s one of those rare engineering ideas that’s both simple and deeply impactful.

What Stripe Did with Idempotent Keys

Stripe, being a payments company, had a problem to solve:

• Customers might refresh or retry during checkout.

• Network requests might time out and be retried by clients automatically.

• Without safeguards, this could result in someone being charged twice for the same transaction.

Their solution was beautifully straightforward:

Every request sent to Stripe can include a unique idempotent key, essentially a unique identifier (like a UUID).

Here’s how it works:

• The first time Stripe sees that key, it processes the request and stores the result.

• If the same key shows up again, Stripe doesn’t re-run the process, it simply returns the same result it gave the first time.

The magic? To the customer, the action always feels safe and predictable. No matter how many times they click “Pay”, they’ll only ever be charged once.

Why This Matters Beyond Payments

At first glance, idempotent keys sound like a “payments thing”. But if you take a closer look, the underlying principle applies everywhere:

• APIs & Microservices
  When your services call each other, retries are inevitable. Idempotency ensures retries don’t create duplicate records or trigger duplicate jobs.

• Event-Driven Systems
  Events often get delivered more than once. Without idempotency, you risk creating duplicate entities or processing the same task twice.

• Batch Jobs & Uploads
  Imagine uploading a large file, only for the system to fail midway. With idempotency, rerunning the job won’t duplicate data, it just resumes gracefully.

• User Actions
  From submitting feedback forms to booking tickets, idempotency makes sure each “intention” from the user gets captured once, not multiple times.

In other words, anywhere retries or duplicate requests can happen, idempotency helps you stay safe.

Benefits of Idempotent Keys

1. Reliability in the face of retries
   Your system can handle the messy realities of networks and user actions without breaking trust.

2. Better user experience
   Users feel reassured knowing they won’t be double-billed, double-registered, or double-subscribed.

3. Operational safety
   Engineers and support teams spend less time cleaning up duplicate data or refunding accidental charges.

4. Simplicity in design
   Instead of inventing complex duplicate-handling logic, you get a clean, repeatable pattern.

5. Traceability
   Each request is tied to a key, making it easier to debug, audit, and reason about the system’s behaviour.

How to Implement Idempotent Keys in Spring Boot

Let’s make this concrete with a Java Spring Boot example. We’ll design a reusable solution using annotations so that any endpoint can become idempotent just by adding @Idempotent.

Don’t feel like reading the full breakdown? Explore the code directly here.

Step 1. Create the Annotation

We’ll mark methods that should support idempotency.

@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface Idempotent {
    // Optional: custom expiry time in seconds (default = 24 hours)
    long expirySeconds() default 86400; 
}

Step 2. Create the Aspect

This Aspect will:

• Check for the Idempotency-Key header.

• If the key exists in Redis, return cached response.

• If not, process request, save response, and return it.

@Aspect
@Component
public class IdempotencyAspect {

    private final RedisTemplate<String, String> redisTemplate;
    private final ObjectMapper objectMapper;

    public IdempotencyAspect(RedisTemplate<String, String> redisTemplate, ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }

    @Around("@annotation(idempotent)")
    public Object handleIdempotency(ProceedingJoinPoint joinPoint, Idempotent idempotent) throws Throwable {
        
        ServletRequestAttributes requestAttributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
        if (requestAttributes == null) {
            // Not in a web request context, proceed normally
            return joinPoint.proceed();
        }

        HttpServletRequest request = requestAttributes.getRequest();
        HttpServletResponse response = requestAttributes.getResponse();

        String idempotencyKey = request.getHeader("Idempotency-Key");
        if (idempotencyKey == null || idempotencyKey.isEmpty()) {
            if (response != null) {
                response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Missing Idempotency-Key header");
            }
            return null;
        }

        // Add the idempotency key to response headers for visibility
        if (response != null) {
            response.setHeader("X-Idempotency-Key", idempotencyKey);
        }

        // Check if we've seen this key
        String cachedResponse = redisTemplate.opsForValue().get(idempotencyKey);
        if (cachedResponse != null) {
            // Cache HIT - return cached response
            if (response != null) {
                response.setHeader("X-Idempotency-Cache-Status", "HIT");
            }
            
            // For cached responses, we always return a ResponseEntity with the cached body
            // since our controller methods return ResponseEntity
            try {
                Object responseBody = objectMapper.readValue(cachedResponse, Object.class);
                return ResponseEntity.ok(responseBody);
            } catch (Exception e) {
                // If deserialization fails, proceed with normal execution
                // This ensures the system remains robust
                System.err.println("Failed to deserialize cached response: " + e.getMessage());
            }
        }

        // Cache MISS - process the request and cache the response
        if (response != null) {
            response.setHeader("X-Idempotency-Cache-Status", "MISS");
        }

        Object result = joinPoint.proceed();

        // Cache the response if it's successful
        if (result != null) {
            try {
                String responseBody;
                if (result instanceof ResponseEntity<?> responseEntity) {
                    // Extract the body from ResponseEntity and serialize it
                    responseBody = objectMapper.writeValueAsString(responseEntity.getBody());
                } else {
                    responseBody = objectMapper.writeValueAsString(result);
                }
                
                long expirySeconds = idempotent.expirySeconds();
                redisTemplate.opsForValue().set(idempotencyKey, responseBody, expirySeconds, TimeUnit.SECONDS);
            } catch (Exception e) {
                // Log the exception but don't fail the request
                System.err.println("Failed to cache idempotent response: " + e.getMessage());
            }
        }

        return result;
    }
}

Step 3. Use It in a Controller

Now, simply annotate your endpoint:

@RestController
@RequestMapping("/api")
public class SampleController {
    @PostMapping("/orders")
    @Idempotent(expirySeconds = 3600) // 1 hour custom expiry
    public ResponseEntity<Map<String, Object>> createOrder(@RequestBody Map<String, Object> orderRequest) {
        // Simulate order creation
        Map<String, Object> response = new HashMap<>();
        response.put("orderId", UUID.randomUUID().toString());
        response.put("status", "created");
        response.put("timestamp", LocalDateTime.now().toString());
        response.put("customerData", orderRequest);
        
        return ResponseEntity.ok(response);
    }
}

Step 4. Client Example

Clients must send the header:

First request: processes and creates an order.

curl -v -X POST http://localhost:8080/api/orders \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: fixed-test-123" \
  -d '{"customerId": "customer-1", "items": [{"id": "item-1", "quantity": 2}]}'

> Content-Type: application/json
> Idempotency-Key: fixed-test-123
> Content-Length: 72
> 
* upload completely sent off: 72 bytes
< HTTP/1.1 200 
< X-Idempotency-Key: fixed-test-123
< X-Idempotency-Cache-Status: MISS
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Tue, 26 Aug 2025 16:53:22 GMT
< 
* Connection #0 to host localhost left intact
{"orderId":"c9431b5d-2a01-48b4-b788-442a82f5827c","customerData":{"customerId":"customer-1","items":[{"id":"
item-1","quantity":2}]},"status":"created","timestamp":"2025-08-26T16:53:22.806485375"}

Second request with same key: returns the exact same response.

curl -v -X POST http://localhost:8080/api/orders \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: fixed-test-123" \
  -d '{"customerId": "customer-1", "items": [{"id": "item-1", "quantity": 2}]}'

> Content-Type: application/json
> Idempotency-Key: fixed-test-123
> Content-Length: 72
> 
* upload completely sent off: 72 bytes
< HTTP/1.1 200 
< X-Idempotency-Key: fixed-test-123
< X-Idempotency-Cache-Status: HIT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Tue, 26 Aug 2025 16:53:30 GMT
< 
* Connection #0 to host localhost left intact
{"orderId":"c9431b5d-2a01-48b4-b788-442a82f5827c","customerData":{"customerId":"customer-1","items":[{"id":"
item-1","quantity":2}]},"status":"created","timestamp":"2025-08-26T16:53:22.806485375"}

Dive into the full code example on Github.

Things to Keep in Mind

• Expiration
  Don’t keep keys forever. Set a time limit that makes sense (e.g., a day for payments, longer for data uploads).

• Scope
  Define what the key represents. Is it for a whole order? A single file upload? A single API call?

• Atomicity
  Make sure storing and checking the key happen atomically to avoid race conditions.

• Consistency
  Be clear about which parts of the request are idempotent. For example, in payments, the amount and currency should remain constant for the same key.

Closing Thoughts

Idempotent keys are one of those rare engineering practices that quietly do wonders. Stripe made them famous by solving a very human problem: the fear of being charged twice. But the principle is universal, make actions safe, predictable, and trustworthy.

Whether you’re running a payment system, a booking platform, or just an API that retries requests, adopting idempotent keys means you’re building resilience into your foundation. And resilience, at the end of the day, is what separates systems people merely use from systems people deeply trust.

In this post, I will talk about:

• Why software engineers usually get fired?

• Kunal’s story: how not surfacing a design risk led to a failure in production.

• The difference between invisible work and visible communication.

• A simple challenge to make your thinking louder in everyday updates.

Most engineers don’t lose their jobs because they wrote bad code. They lose them because they stayed quiet.

I used to work with an engineer named Kunal.

Kunal wasn’t the loudest voice in the room. In fact, he was the kind of engineer every manager loves: reliable, never missed deadlines, never complained. But that reliability came with silence.

When requirements were vague, he just assumed and started coding.
When timelines were unrealistic, he pulled late nights instead of pushing back.
When he noticed flaws in the design that might cause scaling issues down the line, he kept it to himself.

Each time, silence felt easier than debate. Until it wasn’t.

Three months later, exactly what Kunal had feared came true. The API couldn’t handle the surge in traffic. Latency spiked. Upstream services started failing. Customers complained. Leadership was in midnight war-room calls. And the hardest part? Kunal had seen it coming all along. He just never said it out loud.

That’s when I realized something important:
You can ship bugs and still keep your job. But if you don’t communicate risk, you become the risk.

Invisible work doesn’t get you promoted.
Invisible problems get you replaced.

And the truth is, many of us have a little bit of Kunal in us. We notice risks but tell ourselves, “I’ll just work harder to cover it up.” We think silence will protect us, but it only isolates us.

The best engineers I’ve seen don’t just write great code. They make their thinking visible. They clarify the scope. They surface risks early. They translate technical trade-offs into business clarity. And because of that, people trust them.

If you want to become indispensable, don’t just improve your coding skills. Improve how loudly and clearly you share your thinking.

So here’s a small challenge for you this week: in your next standup or update, don’t just share what you did. Share one risk you see, one trade-off you’re making, and one thing you need clarity on.

Small, consistent signals build big trust.

Most of us will eventually become managers. But all of us, at some point, will have managers.

And here is the tricky part: very few of us actually manage our managers well.

I have worked across startups and large organizations, and one pattern I have noticed is this: talented engineers burn out not because of the work itself, but because their relationship with their manager is misaligned. They either see their manager as an obstacle or as a saviour. Both are flawed mental models.

So let us talk about how to make this relationship work, not by waiting for your manager to magically “get better,” but by learning how to manage up effectively.

Warmly,
~Rahul

In this post

We will talk about:

• What your manager is not

• When your relationship with your manager feels tough

• What a great manager–report relationship looks like

• How to think about performance reviews

• What you should expect from your manager

• How to give feedback to your manager

• How I manage sync-ups and alignment with my manager

What your manager is not

When we are new to the workforce, we often put managers into extreme roles.

• The executioner.
  This is the belief that “If I make a mistake, my manager will punish me, maybe even fire me.” Every interaction feels like walking on eggshells.

• The saviour.
  This is the belief that “My manager will fight for me, protect me, and guarantee my promotion.” Here, we rely on them to make everything better for us.

Both of these mental models are misleading. Why? Because both give away your power.

Yes, your manager has influence. They can promote you, give you challenging opportunities, or if things go badly, let you go. But that doesn’t mean they control your entire career. You do.

Think of it like this: your career is a book, and you are the author. Your manager? At best, they are a key character, sometimes even a co-author of a few chapters. But they are not the hero, and they are not the one holding the pen.

The shift begins when you stop asking, “What will my manager do for me?” and start asking, “What can I do to make this relationship better for both of us?”

When your relationship with your manager feels tough

Maybe your manager doesn’t give you feedback. Maybe they never seem to have time for you. Or maybe you feel like they don’t understand what you are capable of. It happens a lot.

But here is the good news: you can take steps to improve things.

1. Know what good looks like.
   Before fixing to do anything, define what a good end product means to you and the manager. Do you want more context about the work or the strategy? Are the goals clear? Make sure to have an honest feedback loop? Write down everything.

2. Talk about it openly.
   Use your 1:1s. Say something like: “Here is the kind of working relationship I would love us to have. For example, I would really appreciate more feedback on my projects. Does that make sense to you?”

3. Be clear about your needs.
   Vague complaints don’t help. Instead of “I feel unsupported,” say: “It would help me if I got feedback after the demo, so I know how to improve next time.”

4. Give your manager feedback.
   Once a month, tell them what is working. “I appreciated that you gave me context before that meeting, it made me more confident.” Then add what you would like more of.

5. Ask for feedback.
   Don’t wait for annual reviews. Simple questions work: “What is one thing I could do differently that would make your job easier?”

6. Escalate if ignored.
   If you have given clear, specific feedback several times and nothing changes, it is fair to bring it up with their manager. Do it respectfully and factually.

7. Leave if trust is gone.
   If you have tried everything and still don’t trust your manager, it may be time to move on. Don’t let one person is limitations block your career.

What a great manager–report relationship looks like

The best relationships between managers and their team members are built on two things: trust and alignment.

Ask yourself:

• Do I trust my manager to do their job well?

• Does my manager trust me to do my job well?

If either answer is “not really,” there is probably misalignment.

For example:

• You might think your manager is role is to back your ideas.

• But your manager might think their role is to challenge your ideas and push for better solutions.

Neither is wrong, but if you don’t talk about it, you will constantly misunderstand each other.

So, ask questions:

• “What should I do differently to get more visibility?”

• How to give feedback to your manager

Once both of you have laid out your definitions of success, you can co-create a shared vision. This clarity transforms the relationship from transactional (“Do this task”) to collaborative (“Let is make each other successful”).

How to think about performance reviews

Performance reviews often get a bad reputation as judgment days. But if you approach them differently, they can become powerful growth moments.

• Celebrate strengths. If your manager praises something, don’t dismiss it. Lean into it. Strengths are often where your biggest opportunities lie.

• Over-correct on patterns. If you keep hearing “speak up more in meetings,” don’t just nudge slightly. Set yourself a goal: “I’ll contribute at least three points in every meeting.” It might feel unnatural, but that is often what is needed to shift perception.

• Shape your next review. Instead of passively waiting, imagine the review you want six months from now. Do you want to be known as a strong mentor? A clear communicator? Tell your manager: “By the next review, I would love to be recognized for X. Can you help me get there?”

What you should expect from your manager

This isn’t one-sided. Managers owe you certain things. At a minimum, expect them to:

• Give you clarity on priorities and goals.

• Provide regular, constructive feedback.

• Create opportunities for you to grow.

• Remove blockers you can’t clear alone.

• Advocate for your work when recognition or promotions are on the line.

If they are not doing these things, speak up. It is not disrespectful, it is necessary.

How to give feedback to your manager

This is where most people freeze. But here is the truth: managers need feedback too.

• Start with curiosity. Instead of “You don’t support me,” try: “I noticed we don’t often do feedback sessions. Is that intentional? How do you prefer to share feedback?”

• Frame it around impact. “When I don’t get feedback after presentations or product demos, I am not sure if I met expectations. It would help me improve faster if you shared your thoughts.”

• Be specific. Concrete requests are easier to act on.

• Follow up. If things don’t improve, ask again: “I wanted to check if the changes we discussed are working for you as well.”

• Escalate respectfully. If nothing changes, involve their manager with facts, not emotions.

Giving feedback to your manager is not rebellion — it is leadership.

How I manage sync-ups and alignment with my manager

A strong manager–report relationship doesn’t just happen in performance reviews. It is built in the small, consistent ways you communicate. Here is what works for me:

1. Biweekly 1:1s with structure
I keep a simple structure for my one-on-ones so that both of us walk away with clarity:

• 20%: What I have accomplished

• 20%: What I can do better

• 20%: Where I fell short

• 40%: Action items

  • 50% on me

  • 50% on my manager

This balance makes the conversation honest, forward-looking, and accountable on both sides. It is not just me reporting status. It is us aligning and committing to actions together

2. Dedicated Slack channels for visibility
I use Slack heavily to reduce back-and-forth and increase transparency. For each project, I create dedicated channels that include my manager and my team. We use them for:

• Design discussions

• Deliverables and sign-offs

• Roadblocks and clarifications

This ensures my manager always has visibility into the work, without needing to chase me for updates. One principle I live by: your manager should already know the state of your work before they ask.

This combination of structured sync-ups and transparent async updates creates trust, alignment, and accountability.

In a recent project I worked on, I was puzzled by what seemed like an inconsistent indexing situation. This inconsistency bothered me, so I decided to dig deeper. What I discovered taught me that "wrong" isn’t always wrong. It is often about trade-offs I hadn’t considered.

The Starting Point: A Confusing Primary Key

The first thing that caught my attention was the primary key design:

CREATE TABLE locations (
    location_id TEXT NOT NULL,      -- e.g., 'A01020304' (hierarchical location code)
    warehouse_id TEXT NOT NULL,     -- e.g., '25' 
    -- ... other columns
    CONSTRAINT pk_locations_location_id_warehouse_id 
        PRIMARY KEY (location_id, warehouse_id)
);

This struck me as backwards. In most multi-tenant systems I've worked with, you'd cluster by warehouse_id first for tenant isolation. But here, location_id comes first. I wondered if this was a mistake or if there was reasoning I wasn't seeing.

Understanding the Location ID Structure

As I dug deeper, I discovered that these location IDs weren't just random strings. They followed a specific hierarchical structure that was key to understanding the whole system. Each warehouse had thousands (sometimes millions) of storage locations, and every location was encoded into a structured string ID.

For example, a location ID like A01020304 breaks down as:

• Aisle: A01 (an uppercase letter + 2 digits)

• Bay: 02 (2 digits)

• Level: 03 (2 digits)

• Position: 04 (2 digits)

When someone searches for A01%, they're looking for all locations in aisle A01 across any warehouse. When they search for H1%, they want all locations in aisles starting with H1. This spatial organization is crucial for warehouse operations. workers need to find adjacent locations, analyze storage density by aisle, or audit entire sections of the warehouse.

My First Breakthrough: The Migration History

Curious about the primary key design, I dug through the database migration history. What I found surprised me:

Original Design (October 2023)

PRIMARY KEY (warehouse_id, location_id)  -- Warehouse first

Changed to (February 2025)

PRIMARY KEY (location_id, warehouse_id)  -- Location first

So it wasn't a mistake! The previous team had deliberately switched the primary key ordering. This made me even more curious. What drove that decision? And more importantly, I started noticing something concerning in my testing: PostgreSQL kept avoiding this index, even when it seemed like it would be faster.

My Testing Adventure: When the Planner Surprised Me

This is where my investigation got really interesting. I decided to systematically test different query patterns to understand when PostgreSQL was making good choices versus poor ones. What I found challenged everything I thought I knew about query optimization.

Test 1: Location Pattern Query

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE location_id LIKE 'H1%';

Result: 477ms with parallel sequential scan for 92,689 rows (~4% of table)

PostgreSQL chose sequential scan, which makes sense for large result sets.

Test 2: Warehouse-Specific Query

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE warehouse_id = '25';

Natural planner choice: 567ms (sequential scan) Forced index usage: 282ms (50% faster!)

SET enable_seqscan = false;
-- Same query runs in 282ms with bitmap index scan
RESET enable_seqscan;

Test 3: Combined Query

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE warehouse_id = '25' AND location_id LIKE 'H1000%';

Natural planner choice: 501ms (sequential scan) Forced index usage: 354ms (29% faster)

Test 4: LIMIT with OFFSET Queries

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE location_id LIKE 'H1%' AND warehouse_id = '09' 
LIMIT 100 OFFSET 100;

Natural planner choice: 436ms (sequential scan) Forced index usage: 204ms (53% faster)

-- Natural planner result (sequential scan)
Limit  (cost=6560.89..12121.78 rows=100 width=12) (actual time=412.845..436.060 rows=0 loops=1)
  ->  Gather
        ->  Parallel Seq Scan on locations
              Filter: (((location_id)::text ~~ 'H1%'::text) AND ((warehouse_id)::text = '09'::text))
              Rows Removed by Filter: 746758

-- Forced index result  
Limit  (cost=9743.66..19486.90 rows=100 width=12) (actual time=204.298..204.300 rows=0 loops=1)
  ->  Index Only Scan using pk_locations_location_id_warehouse_id on locations
        Index Cond: (warehouse_id = '09'::text)
        Filter: ((location_id)::text ~~ 'H1%'::text)
        Rows Removed by Filter: 30265

Key insight: Even with LIMIT + OFFSET, the index scan won significantly. The index could filter by warehouse_id first (30k rows examined) vs. sequential scan examining 746k rows across workers.

Test 5: Simple LIMIT Queries (When the Planner Gets It Right)

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT location_id, warehouse_id
FROM locations
WHERE location_id LIKE 'H1%' 
LIMIT 100;

Natural planner choice: 0.579ms (sequential scan) Forced index usage: 321ms (550x slower!)

-- Sequential scan result
Limit  (cost=0.00..91.49 rows=100 width=12) (actual time=0.035..0.553 rows=100 loops=1)
  ->  Seq Scan on locations
        Filter: ((location_id)::text ~~ 'H1%'::text)
        Rows Removed by Filter: 3060
        
-- Forced index scan result  
Limit  (cost=0.43..224.78 rows=100 width=12) (actual time=321.386..321.501 rows=100 loops=1)
  ->  Index Only Scan using pk_locations_location_id_warehouse_id on locations
        Filter: ((location_id)::text ~~ 'H1%'::text)
        Rows Removed by Filter: 433527

Here, the planner was absolutely correct! Sequential scan found 100 results after examining only 3,160 rows, while the index scan had to examine 433,627 rows to find the same 100 results.

Why LIMIT changes everything:

• Sequential scan can stop as soon as it finds enough rows

• H1% patterns are distributed throughout the data

• Index scan must traverse scattered index pages to find matches

• For small result sets, "first 100 found" beats "optimally ordered access"

Summary: When Indexes Help vs. Hurt

The Pattern:

• Large scans: Sequential wins or is reasonable

• Medium selectivity with filtering: Index wins significantly (even with LIMIT + OFFSET)

• Simple LIMIT queries: Sequential wins dramatically (can stop early)

Understanding Why PostgreSQL Made These Choices

The more I tested, the more I realized there was a pattern to the planner's behavior. PostgreSQL's cost-based optimizer sometimes chose suboptimal plans, but sometimes it was exactly right. I needed to understand the logic behind these decisions.

1. The Cost Model Mystery

I discovered that PostgreSQL uses abstract "cost units" to compare execution strategies. Looking at my system's configuration, I found these parameters:

-- My system's cost parameters (defaults)
SHOW random_page_cost;        -- 4.0 (random reads cost 4x sequential!)
SHOW effective_cache_size;    -- 1502984kB (~1.5GB in my case)
seq_page_cost = 1.0          -- Cost to read a page sequentially  
cpu_tuple_cost = 0.01        -- Cost to process one row

For my warehouse query (warehouse_id = '25' with 6.5% selectivity), the planner calculated:

• Sequential scan cost: ~67,000 units (read entire table sequentially, low per-page cost)

• Index scan cost: ~130,000 units (random index + heap access, 4x page cost penalty)

This explained why PostgreSQL kept choosing sequential scans! The planner assumed random I/O was 4x more expensive than sequential I/O. But when I forced the index usage, it was actually 50% faster. The cost model seemed to be wrong about modern storage characteristics.

2. The Index Scattering Problem

I realized that with the (location_id, warehouse_id) primary key structure I inherited:

• All warehouse '25' locations were scattered throughout the index

• To find them, the index scan had to jump between many different location_id ranges (A01..., B01..., H1...)

• This created the "random" access pattern that the cost model was heavily penalizing

3. Statistics vs. Reality

Interestingly, PostgreSQL's row count estimates were quite accurate (144k estimated vs 145k actual rows). The problem wasn't with statistics - it was with cost estimation. The planner didn't account for:

• How efficiently modern SSDs handle the "random" access patterns

• How much of the data was already cached in the 1.5GB buffer pool

• How well bitmap heap scans actually perform on this hardware

My Analysis: Was This Bad Design?

The big question that kept bugging me: When the planner consistently chooses slower execution plans, does that mean the database design is wrong?

My Conclusion: It's All About Trade-offs

As I analyzed the system deeper, I realized the primary key design created a fundamental trade-off:

1. Location-first ordering (location_id, warehouse_id) optimized for:

   • Global operations spanning all warehouses.

   • Spatial analysis across the entire location hierarchy

2. But it penalized warehouse-specific queries because:

   • Warehouse data became scattered throughout the index

   • This forced "random" access patterns that the cost model heavily penalized

Understanding the Previous Team's Logic

Looking back at the migration history, I realized this was a completely deliberate decision:

October 2023: PRIMARY KEY (warehouse_id, location_id) (conventional multi-tenant design)

February 2025: PRIMARY KEY (location_id, warehouse_id) (location-first optimization)

The team had switched from the "obvious" warehouse-first design to location-first. This told me:

• Cross-warehouse location queries were more critical to the business than warehouse isolation

• They had probably measured the performance impact and decided the trade-off was worth it

• The 50% penalty for warehouse queries was acceptable given their workload priorities

Issue with the query planner: Outdated cost assumptions

The more I studied this, the more I realized the problem wasn't the index design - it was PostgreSQL's cost model being outdated for modern hardware:

• The cost model assumed random I/O was 4x more expensive than sequential

• Modern SSDs and the 1.5GB buffer pool made this assumption much less relevant

• Bitmap heap scans performed better than the planner estimated on this hardware

What I Learned and How I'd Optimize This System

1. Query Planner "Mistakes" Are Often Normal

My investigation taught me that query planners making suboptimal choices is more common than I expected, especially:

• On modern hardware (SSDs, large RAM)

• With index designs that optimize for specific business patterns

• When cost models haven't kept up with hardware evolution

2. The Power of Testing Assumptions

I developed a methodical approach to test when the planner might be wrong, but learned to be careful with LIMIT queries:

-- My testing approach for the warehouse queries
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM locations WHERE warehouse_id = '25';

-- Force index usage to compare
SET enable_seqscan = false;
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM locations WHERE warehouse_id = '25';
RESET enable_seqscan;

-- But I learned LIMIT queries often favor sequential scans
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM locations WHERE location_id LIKE 'H1%' LIMIT 100;

3. How I'd Optimize: Supplemental Indexes

Based on my analysis, if I were optimizing this system, I'd add a supporting index to get the best of both worlds:

-- Add this to optimize warehouse-specific queries
CREATE INDEX CONCURRENTLY idx_locations_warehouse_location 
ON locations (warehouse_id, location_id);

This would preserve the benefits of the location-first primary key while eliminating the 50% performance penalty for warehouse queries.

4. Cost Model Tuning Experiments I'd Try

I realized this system would benefit from tuning PostgreSQL's cost model to reflect modern hardware realities:

-- Current settings that explain the planner behavior
SHOW random_page_cost;        -- 4.0 (assumes spinning disks!)
SHOW effective_cache_size;    -- 1502984kB (~1.5GB in our case)

-- Experiments I'd try:
SET random_page_cost = 1.1;   -- Reflect SSD performance characteristics
SET effective_cache_size = '4GB';  -- If more memory is available

Lowering random_page_cost would make index scans more attractive to the planner, since it would reduce the penalty for the "random" I/O operations that modern SSDs handle efficiently.

References supporting these tuning approaches:

• PostgreSQL Wiki: Tuning Your PostgreSQL Server - Recommends random_page_cost = 1.1 for SSDs

• PostgreSQL Documentation: Runtime Configuration - Official guidance on cost parameters

• PostgreSQL Performance Tuning Guide - SSD optimization recommendations

My Key Discoveries

1. Query planners aren't perfect, but they're not always wrong - PostgreSQL made poor choices for most of my tests but excellent choices for simple LIMIT queries

2. Context matters enormously - Simple LIMIT queries favoured sequential scans (0.5ms vs 321ms), but LIMIT + OFFSET + filtering favoured indexes (204ms vs 436ms)

3. Index design creates trade-offs - the (location_id, warehouse_id) design optimized cross-warehouse patterns but penalized warehouse isolation

4. Cost models lag hardware - PostgreSQL's assumptions about I/O costs didn't match modern SSD and RAM characteristics in my testing

5. Testing assumptions is crucial - when queries seemed slow, testing alternative execution strategies with SET enable_seqscan = false revealed significant performance gaps

6. LIMIT queries favour sequential scans - when I only needed the first N results, sequential access often won dramatically

My Bottom Line: This inherited system wasn't "wrong" - PostgreSQL's query planner was being conservative based on outdated I/O cost assumptions. The primary key design reflected deliberate business optimization choices, and the performance gaps I observed could be addressed with supplemental indexes if needed.

Testing performed on PostgreSQL 15+ with ~2.24M rows. Results may vary based on hardware, configuration, and data distribution.