Robust Reachability: Beyond Simple DNS for Web Crawlers

A dev.to post details how a simple DNS lookup failed to guarantee host reachability for a web crawler, outlining four technical 'leaks' that necessitated a more robust, multi-stage check.

The post by devto details how a seemingly straightforward DNS check, net.LookupHost, failed to accurately determine host reachability for a web crawler. This initial "reachability gate" frequently misidentified live corporate websites as dead, leading to missed data and wasted resources. The founder outlines four specific technical vulnerabilities that necessitated a more robust, multi-stage check.

The core problem was that a simple net.LookupHost call, intended to quickly filter unreachable hosts, proved unreliable. The founder reports that this naive check frequently returned errors for hosts that were, in fact, live and accessible through other means. This led to a revised "reachability gate" designed to address specific failure modes and prevent wasted resources.

CNAME Chains Caused Timeouts

The first vulnerability involved CNAME chains, common for corporate sites behind CDNs or load balancers. The founder explains that resolvers like LookupHost chase these chains in series under a single deadline. A slow intermediate hop, or an unexpected record type like NXDOMAIN at an intermediate tier, could exhaust the budget. This caused the call to return a timeout, incorrectly marking a host as dead. The post notes this failure mode looks identical to a genuinely dead origin from the caller's perspective, leading to "burned a perfectly good URL" that a human could later access.

Datacenter IPs Were Silently Blocked

A second issue arose from origins explicitly dropping connections from cloud IP ranges. The founder reports that from their datacenter's egress IP, the TCP handshake would simply time out, appearing exactly like a dead origin. However, the same host would fetch cleanly via a residential proxy. The naive DNS check would pass these hosts, leading to the crawler burning its full timeout on connections that were never going to land. This meant the check failed to differentiate between a truly down origin and one merely blocking the datacenter IP.

TLS Version Mismatches Failed Handshakes

The third leak concerned TLS version incompatibility. The crawler's production HTTP clients pinned a minimum TLS 1.2 version. Some long-tail origins, however, still only negotiated TLS 1.0 or 1.1. The DNS check would pass, and the TCP handshake would succeed, but the TLS handshake would then fail with a protocol-version alert. This resulted in spending a residential proxy request only to discover a protocol mismatch, an expense that could have been avoided with a more advanced gate.

Avoiding Unnecessary Proxy Costs

The final problem implicitly detailed by the founder was the financial cost associated with residential proxies. Routing every uncertain host through the proxy "just to be sure" turned the reachability check into a significant expenditure. The initial gate's failures meant that many hosts that could have been quickly identified as problematic (due to TLS, CNAME issues, or datacenter blocks) were instead passed through to the more expensive proxy layer. The improved reachability gate aimed to prevent this by failing URLs earlier in the process, thereby reducing unnecessary spend on proxy requests.

The devto post details a specific technical problem and its solution for a web crawler operating at scale. While the principles of robust network reachability are universal, the direct applicability of this playbook depends on the specific operational context. The detailed analysis of CNAME chain timeouts and datacenter IP blocks is highly relevant for any large-scale data collection or bot operation. However, a smaller-scale SaaS product might not encounter these issues with the same frequency or impact.

The focus on net.LookupHost and Go's MinVersion: TLS 1.2 suggests a specific tech stack. Founders using different languages or HTTP clients might face similar problems but require different implementation details. The core lesson — that a single DNS lookup is insufficient for production-grade reachability — remains valid, but the exact "leaks" and their solutions would need re-evaluation based on the specific client, network environment, and target websites. For example, the cost implications of residential proxies are less relevant for applications not relying on them for evasion.

Building a reliable reachability gate for web crawlers requires moving beyond basic DNS lookups. The devto founder's experience demonstrates that robust infrastructure demands a multi-layered approach, accounting for network topology, IP reputation, and protocol compatibility. This technical depth is critical for maintaining data integrity and optimizing operational spend in high-volume scraping environments.

The investor read

This technical deep dive into web crawler infrastructure highlights the increasing complexity and cost associated with large-scale data acquisition. For investors, it signals that companies relying on web scraping for market intelligence, competitive analysis, or data products face significant operational overhead and technical debt if their reachability checks are not robust. The emphasis on residential proxy costs and datacenter IP blocks underscores the arms race between data collectors and website defenses. Investable solutions in this space would either offer superior, cost-effective reachability infrastructure or provide data products that abstract away this complexity entirely, delivering verified, clean data at scale. This also indicates a potential market for specialized infrastructure tools that address these "leaks" more efficiently.

Pull quote: “Building a reliable reachability gate for web crawlers requires moving beyond basic DNS lookups.”