.NET Event Pipeline: Horizontal Scaling, Backoff, and Dead Letter Fixes

This review examines three architectural improvements for a .NET event-driven order pipeline, focusing on horizontal scaling, exponential backoff, and a dead letter path, based on a founder's implementation details.

TL;DR

Best for: Indie founders building event-driven .NET applications with PostgreSQL and Kafka, needing robust message processing and fault tolerance. Skip if: Your stack does not include .NET, PostgreSQL, or Kafka, or if you require a fully managed message queue solution with these features built-in. Bottom line: The outlined fixes provide essential resilience and scalability patterns for critical event pipelines, moving beyond basic outbox implementations.

METHODOLOGY

This v0 review draws on the founder's published claims at the provided dev.to URL; independent benchmarks are pending. Update cadence: re-tested when claims diverge from observed behavior.

• Tool name + version + date observed: Architectural patterns for .NET event-driven pipelines, specifically an outbox pattern with PostgreSQL and Kafka, as described in Aftab Khan's dev.to post. The post was ingested on 2026-05-24, detailing improvements made "a few weeks ago" from that date. Specific versions of .NET, PostgreSQL, or Kafka clients are not detailed, but the patterns are generally applicable to modern versions.
• Source signal URL: https://dev.to/aftabkh4n/i-kept-improving-my-net-order-pipeline-after-a-cto-left-feedback-here-is-where-it-ended-up-25ii
• What's covered in this review: This review covers the founder's claims regarding three specific architectural improvements: horizontal scaling for an outbox pattern using PostgreSQL's FOR UPDATE SKIP LOCKED, an exponential backoff mechanism for Kafka publish failures implemented in C#, and the conceptual addition of a dead letter path. SQL and C# code snippets are analyzed as artifacts.
• What's NOT covered: This review does not cover independent performance benchmarks, the long-term workflow impact of these changes, specific edge cases beyond those explicitly addressed, or the full implementation details of the dead letter path, which was only conceptually introduced in the source.

WHAT IT DOES

The original .NET event-driven order pipeline utilized an outbox pattern, writing orders and outbox records within the same PostgreSQL transaction. A background service polled every five seconds, published to Kafka, and marked messages as processed upon success. CTO Andrew Tan identified three key gaps in this initial design, which the author, Aftab Khan, subsequently addressed.

Horizontal scaling with FOR UPDATE SKIP LOCKED

The initial pipeline lacked protection against multiple poller instances processing the same message concurrently. The fix involves a raw SQL query using FOR UPDATE SKIP LOCKED. This PostgreSQL feature locks selected rows for the duration of the transaction. Crucially, SKIP LOCKED instructs other concurrent poller instances to bypass rows already locked, preventing duplicate processing without waiting. This enables horizontal scaling of poller services, ensuring each message is claimed and processed by only one instance.

Exponential backoff for Kafka failures

Originally, the service retried Kafka publishes every five seconds, regardless of broker availability. If Kafka was down, this led to a constant hammering of connection attempts. The improved service now tracks Kafka publish failures and dynamically adjusts the wait interval. A C# implementation logs warnings and delays subsequent attempts using an increasing _currentBackoff value, calculated as Math.Min(_currentBackoff * 2, maxBackoff). This prevents resource exhaustion and allows Kafka to recover without being overloaded by retry attempts.

Dead letter path for persistent failures

The original pipeline had no mechanism to handle messages that repeatedly failed to publish. Such messages would simply remain in the outbox indefinitely. The updated design conceptually introduces a dead letter path. While specific code snippets for this were not provided in the source, the intent is to move messages that exceed a RetryCount threshold (e.g., 3 retries) to a separate location. This prevents poisoned messages from blocking the pipeline and allows for manual inspection or alternative processing.

WHAT'S INTERESTING / WHAT'S NOT

What's interesting about this post is its direct, pragmatic approach to solving common distributed systems challenges with concrete, stack-specific solutions. The problem-solution framing, driven by CTO Andrew Tan's feedback, provides a clear rationale for each improvement. The use of FOR UPDATE SKIP LOCKED in PostgreSQL is a particularly robust and elegant pattern for achieving exactly-once processing semantics (or at least, at-most-once delivery from the outbox) in a horizontally scaled, database-backed outbox. It demonstrates a deep understanding of database concurrency primitives. Similarly, implementing exponential backoff in C# is a fundamental resilience pattern, critical for any system interacting with external, potentially unreliable services like Kafka. These are not theoretical concepts; they are actionable code snippets that address real-world production issues.

What's not as detailed is the implementation of the dead letter path. While the concept is sound and necessary, the source post only mentions it as a goal, without providing the corresponding SQL or C# code. This leaves a gap for readers looking to implement the complete solution. Additionally, the article focuses on specific fixes rather than a holistic architectural overview, which might require readers to infer broader system design choices. It also assumes a certain level of familiarity with .NET, PostgreSQL, and Kafka, which is appropriate for its target audience but means it's not an introductory guide.

PRICING

This review covers architectural patterns and code implementations, not a commercial tool or service. Therefore, there is no direct pricing information to provide. The cost implications are tied to the underlying infrastructure (PostgreSQL, Kafka, .NET hosting) and developer time.

VERDICT

For indie founders and small teams building event-driven systems on the .NET, PostgreSQL, and Kafka stack, the architectural patterns detailed by Aftab Khan are essential. The FOR UPDATE SKIP LOCKED mechanism for horizontal scaling is a best-in-class approach for ensuring message uniqueness in a database outbox, directly addressing a critical concurrency challenge. The exponential backoff implementation is a non-negotiable component for building resilient integrations with external messaging systems like Kafka. While the dead letter path's implementation details are missing, its inclusion as a conceptual fix highlights a complete understanding of robust message processing. These patterns move beyond basic event-driven setups, providing the foundational resilience and scalability required for production-grade applications.

WHAT WE'D TEST NEXT

Our next steps would involve building a dedicated test rig to validate these claims. We would specifically benchmark the performance implications of FOR UPDATE SKIP LOCKED under varying loads and concurrency levels, measuring database contention and throughput. A full implementation of the dead letter path, including its integration with monitoring and alerting systems, would be developed and tested for its effectiveness in isolating and managing poisoned messages. We would also explore the system's behavior when Kafka experiences prolonged outages, verifying the exponential backoff's efficacy and its interaction with the dead letter queue. Finally, we would compare this bespoke solution against managed message queue services that offer similar features out-of-the-box, to quantify the trade-offs between custom implementation and vendor-provided resilience features.