When should you trust a sudden change in your metrics?

Dan Luu's essay on "Suspicious Discontinuities" sparked a discussion on data literacy, urging founders to question abrupt metric shifts instead of taking them at face value as business reality.

Where it happened

The conversation centers on Dan Luu's June 2024 essay, "Suspicious Discontinuities," and the subsequent discussion on Hacker News. The essay itself provides the core argument, drawing on dozens of examples from tech companies, while the HN thread, with over 200 comments, explores the practical implications for founders, engineers, and analysts.

Side A: The face-value interpretation

This position treats dashboards as a direct representation of reality. A chart is a window onto the business, and its movements correspond to real-world events. If signups spike, a marketing campaign succeeded. If latency drops, performance optimizations worked. This view assumes the underlying data collection is reliable and consistent. The primary task is to react to the story the chart tells, celebrating upward trends and correcting downward ones. Action is prioritized over forensic investigation. For many time-constrained founders and executives, this is the default operating mode; the dashboard is a tool for quick decisions, not a subject of deep epistemological inquiry. The numbers are presumed innocent until proven guilty.

Side B: The skeptical analyst

Championed by Dan Luu, this side argues that a chart is an artifact of a measurement system, not reality itself. Sudden, sharp changes in a metric, or "discontinuities," are rarely caused by a sudden, collective shift in human behavior. Instead, they are tell-tale signs of a change in the system. As Luu states, "When you see a discontinuity, your first guess should be that it's a bug or some kind of instrumentation or data processing change." Examples include a browser changing how it reports its version number, a re-architecture altering how metrics are aggregated, or a simple bug fix that inadvertently changes what gets counted. The proper response to a discontinuity is not to ask "What did our users do?" but "What did we do?" This approach demands deep skepticism and a willingness to investigate the plumbing of the data pipeline before drawing any conclusions about the business.

What's underneath

The disagreement is not about the value of data, but about where trust in an organization is placed. The face-value approach trusts the instrumentation and the dashboards it produces. The skeptical approach trusts that instrumentation is fragile and complex, and therefore requires constant verification. This tension reveals a common organizational divide between the consumers of data (executives, marketing, product managers) and the producers of data (engineers, data scientists). The former group needs clean narratives to make decisions, while the latter is acutely aware of the messy reality behind the numbers. A "suspicious discontinuity" is the moment that messy reality breaks through the clean narrative, forcing the organization to confront how it truly knows what it knows.

The investor read

This discussion is a proxy for founder maturity in due diligence. An investor can use the 'suspicious discontinuities' framework to probe pitch deck charts. A founder who can explain every sharp turn in their graphs as a function of either a real-world event or a data artifact demonstrates a high degree of operational control. Conversely, a founder who takes every hockey-stick chart at face value may lack the technical or analytical depth to manage the business effectively. This signals a growing expectation that data integrity is a core competency, not just a back-office function.

Pull quote: “When you see a discontinuity, your first guess should be that it's a bug or some kind of instrumentation or data processing change.”