Pretraining Corpus Collection at Scale
Curation guides tell you what belongs in a pretraining corpus. This guide covers the layer underneath: the fetch infrastructure that has to acquire billions of documents without melting target sites, double-paying for duplicate content, or collapsing into a single geographic viewpoint. (For the filtering and quality side, see LLM training data curation — the two layers are deliberately separate concerns.)
Pipeline Shape
A collection pipeline that scales cleanly has four stages with queues between them:
[URL frontier] -> [fetch tier (proxied workers)] -> [normalize + dedup] -> [object storage]
^ |
+--------------- discovered links <----------------+The fetch tier is the only stage that touches the open internet, so it is the only stage that needs proxies — and the only stage whose throughput you cannot simply buy with more CPU. Its ceiling is set by politeness limits and IP reputation, which is why the proxy strategy is the throughput strategy.
Capacity Planning Before You Write Code
Work the arithmetic first, because it decides your proxy budget:
- Volume: 100M pages at an average 80 KB of HTML is ~8 TB of transfer. At residential pricing of $4.25–$4.75/GB, that is a real line item — which is why the dedup section below pays for itself.
- Rate: 100M pages in 30 days is ~39 pages/second sustained. Spread across 200 async workers, each worker needs ~0.2 requests/second — politeness limits, not compute, will be your constraint.
- Headroom: the Hex network has capacity for 50B+ requests per week, so the frontier-scale ceiling sits far above what a single pretraining run needs; your practical limits are per-domain politeness and budget.
Two bandwidth-saving decisions matter more than any code optimization: fetch HTML only (block media at the client level), and dedup URLs before fetching, not just content after.
The Fetch Tier: An Async Worker Pool
A single process with an asyncio pool comfortably saturates hundreds of concurrent fetches. Rotation happens per request by default on the residential gateway — every fetch gets a fresh IP, which is exactly right for breadth-first corpus collection (this is the opposite of the sticky-session guidance for agents, and the difference is intentional: corpus fetches are stateless one-shots).
import asyncio
import hashlib
import os
from urllib.parse import urlsplit
import httpx
PROXY_URL = (
f"http://{os.environ['HEX_PROXY_USER']}:{os.environ['HEX_PROXY_PASS']}"
"@gate.hexproxies.com:8080"
)
class DomainThrottle:
"""Per-domain politeness: max N in-flight + fixed delay per domain."""
def __init__(self, per_domain_concurrency: int = 2, delay_s: float = 1.5):
self._sems: dict[str, asyncio.Semaphore] = {}
self._concurrency = per_domain_concurrency
self._delay = delay_s
def _sem(self, domain: str) -> asyncio.Semaphore:
if domain not in self._sems:
self._sems[domain] = asyncio.Semaphore(self._concurrency)
return self._sems[domain]
async def fetch(self, client: httpx.AsyncClient, url: str) -> httpx.Response | None:
domain = urlsplit(url).netloc
async with self._sem(domain):
try:
response = await client.get(url)
return response
except httpx.HTTPError:
return None
finally:
await asyncio.sleep(self._delay)
async def run_workers(urls: list[str], concurrency: int = 200) -> dict[str, int]:
throttle = DomainThrottle()
stats = {"fetched": 0, "dedup_skipped": 0, "failed": 0}
seen_hashes: set[str] = set()
queue: asyncio.Queue[str] = asyncio.Queue()
for url in urls:
queue.put_nowait(url)
async with httpx.AsyncClient(
proxy=PROXY_URL, timeout=30.0, follow_redirects=True,
headers={"User-Agent": "research-crawler (contact: data@yourlab.example)"},
) as client:
async def worker() -> None:
while not queue.empty():
url = await queue.get()
response = await throttle.fetch(client, url)
if response is None or response.status_code != 200:
stats["failed"] += 1
continue
digest = hashlib.sha256(response.content).hexdigest()
if digest in seen_hashes:
stats["dedup_skipped"] += 1
continue
seen_hashes.add(digest)
stats["fetched"] += 1
# hand off to the normalize/store queue here
await asyncio.gather(*(worker() for _ in range(concurrency)))
return statsIdentify your crawler honestly in the User-Agent. Pretraining collection is a research activity; pretending to be a browser forfeits the goodwill that keeps research crawling viable.
Per-Domain Rate Limiting Is the Real Scheduler
Global concurrency limits do not protect individual sites — 200 workers that all happen to pull URLs from one domain will hammer it even with rotation. The DomainThrottle above enforces two invariants per domain: bounded in-flight requests and a minimum inter-request delay. Tune both from robots.txt Crawl-delay where present, and treat 429s as a signal to double the domain's delay, not to rotate harder. Rotation hides your infrastructure's identity; it does not make over-crawling acceptable, and aggressive per-domain rates will burn IP reputation that the whole pool shares.
Dedup Instrumentation, Not Just Dedup
Exact-hash dedup (above) is table stakes. What pipelines actually need is instrumentation around it, because dedup rates are an early-warning system:
- URL-level dedup rate before fetching: a rising rate means your frontier is recycling — fix discovery, save bandwidth.
- Content-hash dedup rate after fetching: 5–15% is normal (mirrors, www/non-www, trailing-slash variants). A sudden jump usually means a site is serving an error page with HTTP 200 to suspected bots — sample and inspect rather than silently storing garbage.
- Near-dup rate (MinHash/SimHash in the normalize stage): track it per domain; domains with extreme near-dup rates (boilerplate-heavy forums, calendar pages) deserve frontier-level demotion, which again saves fetch budget.
Every skipped fetch at $4.25–$4.75/GB is money; every duplicate that reaches training is a quality cost. The instrumentation pays twice.
Geo-Distributed Collection
Web content is geo-variant: news sites localize, e-commerce shows regional catalogs, and some publications are only reachable from their home market. Collecting everything from one country bakes a single regional viewpoint into the corpus. Hex residential ports are country-targeted (195+ countries; country selection is port-based), so geo-distribution is a worker-pool parameter rather than an infrastructure project: run regional worker groups, each with its country's port, and partition the frontier by the content's likely home region. Detail on port mapping lives on the residential proxies page.
A practical split for a general web corpus: route each domain's fetches through the country suggested by its ccTLD or hosting region, and default the rest to a balanced mix rather than 100% US egress.
Checkpointing and Resume
A multi-week collection run will be interrupted — spot instances reclaim, deploys roll, a bug surfaces at document 40 million. Design for resume from the start: persist frontier state (fetched, in-flight, pending) to durable storage on an interval, write fetched documents with content-addressed names (the hash you already computed) so re-writes after a crash are idempotent, and make workers stateless so a restarted pool picks up exactly where the frontier says it should. The dedup sets above must survive restarts too; an in-memory set is fine for a prototype, but production runs want a Bloom filter or RocksDB-backed seen-set checkpointed alongside the frontier. The test of a good pipeline is that kill -9 on the whole fleet costs you minutes of progress, not a re-fetch of paid bandwidth.
Cost Controls
Three controls keep the bill proportional to corpus value: (1) hard per-run byte budgets enforced in the worker (kill the run, not the card); (2) HTML-only fetching — never download images, video, or fonts into a text corpus; (3) conditional requests (If-Modified-Since) on recrawl passes so unchanged pages cost a header, not a body.
Compliance and Provenance
Honor robots.txt (including Crawl-delay), respect site terms, and keep per-document provenance records: URL, fetch timestamp, egress country, content hash. Provenance is what lets you honor takedown requests and answer dataset-documentation questions later — retrofitting it onto billions of stored documents is far harder than logging it at fetch time. If your pipeline feeds a RAG system rather than pretraining, the RAG data pipeline post covers the fresher-data variant of these patterns.