429 Too Many Requests through a proxy: find the limit

Most likely first.

1. Too many requests per second to one site. Scraping and monitoring jobs usually start at full speed. A burst at start-up, or parallel workers fetching the same pages, is the most common trigger.
2. Retries without backoff. A client that retries every failure at once raises its rate exactly when the site is asking it to slow down, and can turn a 30-second limit into a long one.
3. One sticky IP carries all the traffic. A sticky session is the right choice for logins and carts, but it puts every request from that worker on one address.
4. The limit follows an account, cookie or API key. If every worker sends the same session cookie, token or API key, the site counts them together whatever address each one uses.
5. Other traffic on the same address. Residential and mobile exits are shared networks: a household or carrier address can carry other traffic to the same site, which counts against a per-IP limit.

1. 1Read the headers that came with the 429

   Retry-After gives the wait. RateLimit or X-RateLimit headers, where a site sends them, show the quota left and the window.

   Rate-limit headersCopy

   curl -s -D - -o /dev/null -x http://HOST:PORT -U "USER:PASS" https://TARGET/ \
     | grep -iE "^(HTTP|retry-after|x-ratelimit|ratelimit)"

2. 2Confirm who sent it

   A 429 after 200 Connection established is the site's. A 429 instead of the tunnel is the proxy's.

   Proxy's answer, then the site'sCopy

   curl -v -x http://HOST:PORT -U "USER:PASS" https://TARGET/ -o /dev/null 2>&1 \
     | grep -iE "^(> CONNECT|< HTTP|< x-proxy-reason|\* CONNECT)"

3. 3Measure the rate that triggers it

   Run a slow loop and note when the first 429 appears. That tells you more about the limit than any guess.

   One request every 2 secondsCopy

   for i in $(seq 1 30); do
     curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" \
       -x http://HOST:PORT -U "USER:PASS" https://TARGET/
     sleep 2
   done

4. 4Find what the limit follows

   Send the same request from two sticky sessions without cookies, then with the same cookie. If the second exit is limited only when it shares the cookie or token, the limit follows the session, not the IP.

• Honor Retry-After and back off

  Wait at least the Retry-After time before the next request to that site. Without one, start at a second or two and double the wait after each further 429, with random jitter and a cap on retries.

  Python requestsCopy

  import random, time
  import requests
  
  proxy = "http://USER:PASS@HOST:PORT"
  session = requests.Session()
  session.proxies = {"http": proxy, "https": proxy}
  
  def get(url, tries=5):
      for attempt in range(tries):
          r = session.get(url, timeout=30)
          if r.status_code != 429:
              return r
          wait = r.headers.get("Retry-After", "")
          delay = int(wait) if wait.isdigit() else 2 ** attempt
          time.sleep(delay + random.uniform(0, 1))
      return r

• The same in Node.js

  Release the body of each refused response before waiting, so the connection goes back to the pool.

  Node.js with undiciCopy

  import { fetch, ProxyAgent } from "undici";
  
  const dispatcher = new ProxyAgent("http://USER:PASS@HOST:PORT");
  
  async function get(url, tries = 5) {
    let res;
    for (let attempt = 0; attempt < tries; attempt++) {
      res = await fetch(url, { dispatcher });
      if (res.status !== 429) return res;
      await res.body?.cancel();
      const seconds = Number(res.headers.get("retry-after"));
      const delay = seconds > 0 ? seconds * 1000 : 2 ** attempt * 1000;
      await new Promise((resolve) => setTimeout(resolve, delay + Math.random() * 1000));
    }
    return res;
  }

• Send less

  Lower concurrency per site, start workers gradually instead of all at once, cache pages you already have, and use the site's API where it offers one: API limits are documented. Spreading one job across more addresses does nothing for limits on cookies, keys or accounts, and working around a limit the site has stated can breach its terms.

• 429 Too Many Requests (RFC 6585, section 4) — IETF / RFC Editor
• Retry-After (RFC 9110, section 10.2.3) — IETF / RFC Editor
• RateLimit header fields for HTTP (Internet-Draft) — IETF HTTPAPI Working Group