Web Crawler: Politeness, Scope and Reliable Progress

A web crawler needs a stopping rule

This exercise designs a crawler for a selected collection of public documentation sites. Fetching pages is only one step toward search; parsing, deduplication, indexing, and ranking are separate systems. The design is not a reconstruction of Google's crawler.

Bound the collection

Assume one million HTML fetches per day averaging 200 kB each. That is about 11.6 requests per second and 200 GB of response bodies per day in decimal units, excluding compression, metadata, retries, and assets. A month's retained bodies would be about 6 TB before deduplication.

The aggregate rate looks modest, but all 11.6 requests per second directed at one small site could be excessive. Define an allowed-domain list, byte budget, per-site page budget, fetch timeout, response-size limit, and recrawl policy. Infinite calendars and parameter combinations can defeat a crawl-depth limit without ever leaving a site.

Schedule by origin, not just FIFO order

Maintain a frontier entry with URL, origin, priority, next eligible time, and attempt state. A scheduler chooses among origins whose next fetch is due, then leases a bounded batch to a worker. Keep per-origin concurrency and spacing so adding workers does not multiply pressure on one server.

Origin grouping distinguishes scheme, host, and port. Several origins can share the same physical server; an additional host or IP budget may be necessary. Conversely, one domain can serve many unrelated origins. Treat the grouping as an operational policy rather than a universal definition of a website.

Use a declared crawler user agent with a useful contact or information page. Apply robots rules before fetching paths, and implement caching and failure behavior according to the protocol. Robots rules are cooperative crawl instructions, not access authorization or permission to republish content. Robots Exclusion Protocol, RFC 9309.

A retry must not lose the URL

When a worker leases a URL, mark it in flight with an expiry. After a successful fetch, atomically record the response metadata, durable body reference, and completion state. If the worker crashes after storing the body but before completion, retrying may fetch twice; a content digest and idempotent result record can prevent duplicate stored artifacts.

If the worker marks completion before the body is durable, a crash can permanently lose a page while the frontier reports success. Prefer duplicate work over untraceable loss. Track attempts and stop after a defined budget rather than retrying a permanently broken page forever.

Respect server throttling and relevant Retry-After guidance. Back off on repeated errors. DNS caches should follow authoritative TTLs and revalidate as needed; an arbitrary long TTL can keep contacting an outdated address.

Normalize only what the URI rules justify

Lowercasing the scheme and host is different from lowercasing a path. Removing a fragment from the HTTP fetch target is different from removing a query parameter. A www host and a bare host can contain different content; do not merge them merely because their names resemble each other.

Track redirects and any canonical hints separately from the original discovered URL. A hint may be wrong or point outside the allowed collection. Apply scope and network-address checks again after each redirect, including protection against requests to local or private services when crawling untrusted URLs.

A Bloom filter is not a complete visited set

A Bloom filter can quickly say “definitely unseen” or “possibly seen.” If “possibly seen” causes the crawler to drop the URL permanently, false positives become missing pages. For a collection that requires completeness, consult an authoritative visited store for positives. If omissions are acceptable, quantify that trade-off instead of calling the filter exact deduplication.

Use a cryptographic digest to identify exactly equal stored bodies. Near-duplicate detection is a separate heuristic that may compare normalized text or similarity fingerprints. Keep a representative plus provenance so two sources are not silently treated as the same document when their dates, access rules, or meaning differ.

Decide what to revisit

A daily recrawl of every page wastes work on stable manuals and may still miss frequently changing pages. Store validators such as ETag or Last-Modified where available and send conditional requests. Schedule revisits using observed changes and editorial priority, while maintaining a maximum age for important pages.

Test an endless calendar, a redirect to a disallowed address, robots retrieval failure, a worker crash after upload, and a server returning throttling responses. Measure fresh-page coverage, bytes per useful changed page, oldest unfinished lease, and per-origin error rate. A crawler is healthy when it makes useful, accountable progress within its boundaries, not when its request counter grows fastest.