How a fintech startup designed a pluggable analytics pipeline for 10,000 daily events

A fintech team replaced ad-hoc SQL queries with a three-part, event-driven system for performance commissions. The architecture prioritizes decoupling and cost-effectiveness over real-time processing for its analytics needs.

A fintech startup’s commission tracking system was breaking. Management needed to calculate agent performance, but the initial solution, a complex SQL query joining over seven tables, created a fire drill for every ad-hoc report. The core system was designed to handle an estimated 10,000 agent actions per day, but retrieving state from event logs was becoming untenable.

The underlying issue wasn't the SQL. It was that the system was storing events but not state. Retrieving an agent's current performance required replaying all historical events every single time. This prompted a redesign focused on a dedicated, event-driven analytics pipeline.

Defining requirements to avoid over-engineering

The team first established sharp functional and non-functional requirements. The system needed to track user actions, generate scheduled and on-demand reports, and archive old data. Critically, the non-functional requirements dictated the architecture. The pipeline could not slow down the core application and could tolerate delays, defining it as an eventually consistent, batch-oriented system.

This explicitly ruled out low-latency reads or sub-second aggregations. By defining what the system was not, the team avoided the common trap of building for a real-time scenario they didn't have. The design also required the system to be pluggable, capable of being added to any service without re-architecture.

Sizing the problem with back-of-envelope math

Before selecting tools, the team estimated the data load. They projected approximately 10,000 events per day, based on 200 agents performing 50 actions each. With each enriched JSON event estimated at 2KB, this amounted to a manageable 20MB of raw data daily. A monthly report would cover around 300,000 events.

At this scale, the analysis concluded that a distributed streaming platform like Kafka was unnecessary for initial processing. The total raw event data after a year would be around 7GB, making archival to a low-cost object store like AWS S3 a cost-effective strategy. This estimation step grounded the technical choices in concrete operational realities, not hype.

A decoupled architecture for event capture

The proposed architecture splits the pipeline into three loosely coupled parts. The source material details the first stage: event capture and staging. In this model, front-end applications fire events for user actions. The back-end service receives them, enriches them with context like agent IDs and timestamps, and writes the structured event to its existing relational database (RDBMS).

The choice of an RDBMS at this stage was deliberate. Events are written once, are structured predictably due to the enrichment step, and benefit from the atomicity of database transactions. A cron job then periodically reads unprocessed events, copies them to an AWS S3 bucket, and marks them for deletion upon a successful write. A second process handles the final deletion. This two-step pattern provides a basic guarantee against data loss if the S3 transfer fails mid-batch.

WHAT WE'D CHANGE

The playbook is a solid, pragmatic starting point, but the reliance on a cron job introduces a point of failure common in older architectures. Cron jobs are simple but can fail silently, lack robust retry logic, and create processing spikes. A modern alternative is a Change Data Capture (CDC) pipeline using a tool like Debezium. CDC treats the database's transaction log as a stream, capturing every event as it's written and pushing it to a message queue. This eliminates the need for polling via cron.

The manual "mark and delete" process described is a hand-rolled implementation of the transactional outbox pattern. While functional, it's prone to implementation errors. Using a dedicated outbox library or a CDC-based approach provides stronger, more reliable at-least-once delivery guarantees without custom logic.

Finally, performing enrichment in the core application before the initial database write can create a bottleneck under heavy load. For greater resilience, the application could write raw, unenriched events to a dedicated queue like Amazon SQS. A separate, scalable enrichment service could then process events from this queue before they are staged for batch analysis. This decouples the core application from the complexities of the analytics pipeline even further.

LANDING

The architecture demonstrates a disciplined, scale-appropriate approach to building internal systems. By correctly identifying the problem as one of state retrieval, not just complex queries, the team was able to design a focused solution. The decision to use a simple RDBMS and cron job, rather than a more complex streaming platform, was justified by clear estimations of data volume. This pragmatism delivers the required business value without incurring the technical debt or operational overhead of a prematurely scaled system.

The investor read

This playbook signals a mature, anti-hype approach to internal tooling for bootstrapped or capital-efficient startups. Instead of adopting expensive, complex data stacks, the design solves a specific business problem (commission tracking) with commodity cloud services (RDBMS, S3, cron). This indicates high operational discipline. An investable company demonstrates this pragmatism, building only what is necessary while designing for future evolution. The architecture’s plug-and-play nature is key; it allows the company to swap out components, like replacing the cron job with a CDC pipeline, as scale and revenue justify the engineering cost. This is a positive signal for founders who understand how to manage technical debt and infrastructure spend.

Pull quote: “The underlying issue wasn't the SQL. It was that the system was storing events but not state.”