LWN.net built a custom scraper defense to protect its content and privacy

The technical publisher rejected commercial WAFs, opting for a self-hosted solution it claims cut unwanted traffic by 90%. The playbook reveals a new cost center for independent content businesses.

Scrapers were consuming hundreds of gigabytes of data from LWN.net, a volume that threatened the subscription-based publisher's infrastructure. In response, the site's developers implemented a series of custom defenses that they report cut this unwanted traffic by 90%, offering a playbook for others facing the same threat.

The decision to build, rather than buy a solution from a provider like Cloudflare, was deliberate. It prioritized reader privacy and configuration control over the convenience of a commercial service.

First, simple rate limiting

LWN's initial defense was basic IP-based rate limiting. This is a standard first step for many sites. The system would temporarily block IP addresses that made an excessive number of requests in a short period. While effective against naive bots, this approach proved insufficient. Sophisticated scrapers operate from distributed IP ranges, often using residential proxies or large cloud provider blocks, easily circumventing simple limits. The publisher needed a more nuanced approach.

A custom, multi-factor system

The core of LWN's strategy is a custom system that analyzes multiple signals. It reportedly examines not just the request rate but also the user-agent string, the sequence of pages requested, and other behavioral patterns. The system assigns a "suspicion score" to each visitor. When a score crosses a certain threshold, the system presents a JavaScript challenge. This is designed to be trivial for a standard browser to solve but difficult for simple script-based scrapers. Only if that fails does it escalate to presenting a CAPTCHA.

Rejecting the commercial WAF

LWN explicitly chose not to use a commercial Web Application Firewall (WAF) like Cloudflare. The primary reason cited was reader privacy. Routing all traffic through a third-party service would mean sharing visitor IP addresses and browsing habits, a trade-off the publisher was unwilling to make. A secondary reason was control. A self-hosted solution allows for fine-grained tuning of the detection rules, avoiding the "black box" nature of commercial offerings that can sometimes lead to false positives that block legitimate readers.

What we'd change

The build-it-yourself approach is a viable strategy for a publication like LWN, which employs developers and serves a technically sophisticated audience. It is not a universal solution. For most independent publishers or small SaaS companies, the engineering cost and ongoing maintenance of a custom anti-scraping system are prohibitive. The developer hours required to build, test, and maintain such a system could be better spent on core product development.

A commercial WAF, while imperfect, offers a far lower total cost of ownership for resource-constrained teams. The decision to build rather than buy reflects a deep-seated skepticism of handing reader data to a third party. This is a valid philosophical stance but an expensive operational choice. Furthermore, the anti-scraping landscape is an arms race. Commercial providers have dedicated security teams whose sole job is to track and counter new evasion techniques. A custom, part-time solution risks falling behind quickly.

Landing

LWN's defense is a case study in the escalating costs of publishing on the open web. The rise of large-scale AI model training has turned every piece of public content into a potential training dataset, forcing publishers to become defenders. While LWN's solution prioritizes privacy, it also establishes a new, non-optional line item for any serious content business: a budget for traffic analysis and bot mitigation. This is no longer a niche technical problem. It is a fundamental cost of doing business for anyone creating valuable information online.

The investor read

This signals a new, non-optional cost center for any content-based business: "scraper defense." Investors evaluating publishers or data-as-a-service companies must now include scraper mitigation strategy and its associated costs in their due diligence. The move by LWN highlights a market gap for privacy-preserving, self-hostable, or more transparent WAF solutions that appeal to businesses wary of vendors like Cloudflare. It also reinforces the value of proprietary datasets that are not easily accessible on the public web. A business with a strong defense or a less-scrapeable asset has a more durable moat.

Pull quote: “The decision to build rather than buy reflects a deep-seated skepticism of handing reader data to a third party.”