Proxies for AI training data collection

Most of the public web you would crawl for a training set already exists as a download. Common Crawl publishes monthly archives of billions of pages, Wikipedia and Wikimedia ship full dumps, and many sources offer official APIs or bulk exports under clear licences. Those cost no proxy traffic, carry provenance, and are what the large models were built on.

Proxies earn their price on what the dumps miss: pages published since the last archive, sites that serve different content by country or language, sources behind geographic restrictions, and the long tail that crawlers never reached. Use residential exits for those, record every fetch, and keep the proxied share of the corpus small and deliberate.

We fetched 50 public pages (news, documentation, product pages, blog posts and forum threads) through the TrueProxies residential gateway on October 1, 2026, extracted the main text with trafilatura and counted it with tiktoken's o200k_base encoding. The median page held 990 tokens, so a million tokens is about 1,010 pages. Multiply by the MB each pipeline moves per page and you have the traffic bill.

A plain HTTP fetch gets the HTML and nothing else: 0.072 MB per page at the median in our test. A full browser render loads scripts, styles, images, fonts, trackers and ads, 2.284 MB per page, or 31.7x more, for the same text. Render only when the text is built by JavaScript, and when you do, abort image, font and media requests: that alone cut the browser figure by 59%.

Check a sample of each source with both methods before the crawl. If the HTML fetch already contains the article body, the browser is wasted traffic and wasted time; our browser loads took several times longer than the fetches through the same exits.

GB-based plans exit from 50M+ residential IPs in 150+ countries, with country, region, city and ASN targeting on the username. That is how a multilingual corpus gets the local edition of a site rather than the version served to foreign datacenters: a German exit for the German pages, a Japanese exit for the Japanese ones, each recorded with the exit country in the provenance log.

Concurrent connections aren't billed separately. Plans allow up to 3,000 connections (Datacenter IPv6: 3,000). A training crawl is the rotating-session case: no session ID on the username, so every request can take a fresh exit and the load spreads across the pool. Keep per-domain rates polite regardless; a proxy does not change a site's rate limits.

For volume, the 100 GB and 1,000 GB Traffic packs bring the price to $0.97/GB and $0.81/GB, and the ladder continues to $0.60/GB at 5,000 GB. Packs are valid 30 days from activation, so buy for the crawl you will run this month.

Deduplicate at the document and near-duplicate level, strip boilerplate before counting tokens, and keep a provenance record for every page: URL, fetch time, HTTP status, exit country and the extraction method. The record is what lets you honour an opt-out or takedown later without rebuilding the corpus, and it is what makes a dataset auditable.

Not legal advice. Follow robots.txt, honour text-and-data-mining opt-outs (including the machine-readable kinds) and site terms, and handle personal data under the rules of the jurisdictions you collect in and serve from. Providers of general-purpose AI models in the EU carry a copyright-policy duty under the AI Act, which includes respecting rights reservations. Keep the provenance log, and review the pipeline with counsel before it feeds a model.