A modern crawling stack for 500 sites on a $310/month budget

An analysis of a real-world data collection stack using Cloudflare Workers, D1, Vectorize, and Node.js to power AI agents, based on a developer's public breakdown of costs and scaling issues.

The Answer Up Front

This hybrid Cloudflare and Node.js stack is a pragmatic choice for early-stage teams building AI products that need fresh, structured data from a few hundred websites. It's a cost-effective, serverless-first architecture that gets the job done on a budget. Teams anticipating crawling thousands of sites or facing sophisticated anti-bot measures should look elsewhere. The reliance on raw Headless Chrome is its primary scaling bottleneck, a problem the original architect already suspects. For moderate scale, this is a strong, modern reference architecture; for large scale, it's a starting point for a more robust system.

Methodology

This v0 review analyzes a crawling infrastructure stack detailed by user pystar on Reddit on June 17, 2026. The analysis is based entirely on the author's published claims, including the architecture, components, cost breakdown, and reported scaling challenges. We have not independently benchmarked this stack's performance, memory usage, or success rate against different types of websites. This review covers the strategic implications of the architectural choices (e.g., hybrid compute, integrated data storage) and evaluates the reported pain points. It does not verify the cost figures or the specific limits encountered. As a v0 review drawing on a single public source, its purpose is to evaluate a real-world implementation, not to provide a definitive benchmark. Updates will follow if we conduct independent tests.

What It Does

The stack is designed to crawl approximately 500 business websites to extract content for AI agents. Its goal is to keep website data, embeddings, and knowledge of interactive elements current.

Hybrid compute for crawling

The architecture uses a two-tiered approach for crawling. Simple crawls are executed on Cloudflare Workers, providing a low-cost, serverless option for basic sites. For more complex single-page applications (SPAs) that require significant JavaScript rendering, the system falls back to a dedicated Node.js service running a headless Chromium instance. This hybrid model attempts to balance cost and capability, using the cheaper serverless option where possible and reserving the more expensive, stateful compute for difficult targets.

An integrated Cloudflare data pipeline

Data storage is handled entirely within the Cloudflare ecosystem. Raw HTML and extracted text are stored in D1, Cloudflare's serverless SQL database. The extracted content is then used to generate embeddings, which are stored in Cloudflare Vectorize. This creates a tightly integrated data pipeline from crawl to vector storage, minimizing data transfer costs and simplifying infrastructure management by keeping everything with a single vendor.

A tiered re-crawl schedule

The crawling logic is not a simple brute-force re-crawl. It uses a tiered schedule based on assumed page importance. High-priority pages like the homepage and pricing are crawled every six hours. Core business pages (about, services) are checked daily, while all other discovered pages are crawled weekly. This intelligent scheduling aims to maximize data freshness for critical content while minimizing unnecessary compute and bandwidth costs.

What's Interesting / What's Not

The most interesting aspect is the concrete cost breakdown: ~$310 per month for 500 sites. This provides a valuable data point for any team modeling the cost of goods sold (COGS) for an AI product built on public web data. At roughly $0.62 per site per month, it's an efficient implementation. The hybrid Workers-plus-Node.js model is a clever optimization for this scale.

The reported scaling issues are entirely predictable and represent the stack's primary weakness. The author notes that Headless Chrome is memory-heavy, limiting concurrency to about three crawls per instance. This is a well-documented problem. Browser automation is resource-intensive, and a vanilla implementation will always hit a wall. Similarly, getting blocked by sites protected by Cloudflare's own bot detection is a classic cat-and-mouse problem that requires more than just respectful delays to solve. It typically necessitates sophisticated proxy networks and browser fingerprinting management, which are not included in this budget.

The author's final question, comparing Headless Chrome to Playwright, gets to the heart of the matter but may be focused on the wrong variable. While Playwright offers a more ergonomic API, it is built on the same underlying browser engines and is unlikely to solve the fundamental memory consumption or bot detection problems. The real next step is moving away from direct browser automation toward commercial proxy and scraping infrastructure.

Pricing

This is a self-built stack, not a commercial product. The author reported the following monthly operational costs as of June 2026:

• Compute for crawlers: ~$180
• Embedding API calls: ~$90
• Storage (D1 + Vectorize): ~$40
• Total: ~$310

Verdict

This stack is an excellent, publicly documented reference for teams needing to solve data collection for AI at a moderate scale. Its use of a serverless-first, hybrid model is modern and cost-effective. For founders building RAG systems or agents that need to stay current with a few hundred specific websites, this is a viable and affordable architecture. However, the reported issues with memory limits and bot detection are clear signals that this approach does not scale gracefully into the thousands of sites. It's a solid blueprint for an MVP or a small-scale product, but teams should plan to replace the home-grown crawler component with a commercial-grade solution as they grow.

What We'd Test Next

A v2 of this review would require hands-on testing. First, we would directly address the author's question by benchmarking a swap-in replacement of Headless Chrome with Playwright to measure any differences in memory footprint, CPU usage, and crawl success rate on a standardized set of complex sites. Second, we would introduce a commercial proxy service to quantify its impact on successfully crawling Cloudflare-protected sites and calculate the new total cost. Finally, we would compare the total cost and reliability of this entire self-managed stack against a leading scraping API like ScraperAPI or Bright Data's Scraping Browser to determine the break-even point where building your own is no longer economical.

The investor read

This stack is a microcosm of the infrastructure powering a new wave of AI agent and RAG-based companies. The key takeaway is the low, transparent COGS: ~$310/month to maintain data freshness from 500 sources. This signals that the base cost of data acquisition for many AI products is becoming a rounding error, shifting the value to the models and application layer. However, the reported scaling issues highlight the durable market for specialized scraping infrastructure. Companies like Bright Data, Oxylabs, and ScraperAPI are well-positioned to capture spend as these small-scale projects mature and find that building and maintaining robust, large-scale crawling systems is a significant distraction from their core product. This stack is a 'buy' signal for the picks-and-shovels players in the web data space.

Pull quote: “For moderate scale, this is a strong, modern reference architecture; for large scale, it's a starting point for a more robust system.”