Skip to main content
Web scraping requests occasionally fail for transient reasons, like a rate limit or a temporary server hiccup. The SDK can retry these automatically so you don’t have to write that logic yourself.

Basic usage

Retries are off by default (retries=0): the client sends each request once. Pass a retries count to the client constructor to turn them on:
With retries=2, a failed request is tried up to 3 times in all. The client then returns the last response it got.
This retry policy needs Python SDK 1.13.0 or later. If you upgrade with retries already set, see Upgrading from earlier versions.

When to enable retries

Enable retries (2–3 is a reasonable starting point) whenever you’re running requests at scale or in production. Anywhere a transient 429 or 5xx shouldn’t fail your whole job. Leave retries=0 (the default) for quick one-off scripts or debugging, where you’d rather see the raw failure immediately than wait through automatic retry attempts.
Don’t also wrap the client in your own retry loop: the tries multiply. With retries=2 (3 tries) inside a loop of 4 tries, one call can send up to 12 requests.

Which failures get retried

With retries set, the client retries a GET after:
  1. A 429, 500, 502, 503 or 504.
  2. A 422, except Extract’s 422 with code no_data_extracted, which is a final answer.
  3. A failure to connect: a refused connection, a name that did not resolve, or a connect timeout. The request never reached the API.
It never retries:
  1. A POST or PUT, whatever its headers, or a request with files or a stream body. An Idempotency-Key header is forwarded to the target site and does not change this.
  2. Any other network error, such as a read timeout or a connection that closed after the request was sent. The API may still finish that request and charge it.
  3. Any other status, such as 400, 401, 402, 403 or 404.
  4. A status from the target site: any status when original_status is set, and a status listed in allowed_status_codes.
To retry a POST or PUT, write your own retry, and only when the target site can safely receive the same request twice.

How long the SDK waits

  1. When the API sends Retry-After in whole seconds, the client waits that long, from 0.25 to 30 seconds. retry_max_wait does not change this. Any other Retry-After, such as an HTTP date, is ignored.
  2. Otherwise it waits a random time from 0.25 seconds up to 1 second before the first retry, 2 seconds before the second, 4 before the third, and so on, capped at retry_max_wait.
  3. A retry starts only if its wait ends within retry_budget seconds of the first try. A running try is not cut short, so a call can take longer than that.
A wait cannot be cancelled, also in fetch_async and the other async methods. While an async call waits, it keeps its worker in the client’s thread pool (concurrency, default 5), so other async calls on the same client can wait too.
retry_max_wait and retry_budget must be finite numbers above 0, or the constructor raises ValueError. Pass retries as an int: a bool or a float still works with a DeprecationWarning (True is 1, 2.5 is 2), and None or a string raises TypeError. You can also set client.retries after you create the client; the next call uses it.
You aren’t charged for retried attempts that fail. Only successful responses count toward your usage. See the Retry Failed Requests guide for more on how this works.

Upgrading from earlier versions

From Python SDK 1.13.0, an existing retries setting behaves differently. Calls with retries=0 are unchanged. If you already set retries, check these changes:
  • A PUT is no longer retried on a 429 or 5xx, and a POST or PUT is no longer retried on a 422 or a failed connection.
  • A read timeout or a connection that closed after the request was sent is no longer retried. A read timeout now raises requests.ReadTimeout instead of requests.ConnectionError, so catch requests.ReadTimeout too.
  • A TLS error or any other error that is not a failed connection is no longer retried.
  • A status from the target site (original_status, allowed_status_codes) is no longer retried.
  • Waits follow Retry-After or a random backoff, instead of a fixed 0, 1, 2, 4 seconds. No retry starts after retry_budget (default 90 seconds).

Handling failures

The SDK treats HTTP error status codes and network-level problems differently. Pick the tab that matches what you’re dealing with:
Even after all retry attempts are exhausted, the SDK returns the final requests.Response object. It does not raise an exception for a non-2xx status. Always check the response before trusting its content:
If you’d rather fail loudly, call response.raise_for_status(), which raises requests.exceptions.HTTPError for 4xx/5xx responses:

Troubleshooting

  • Response still fails after retries are exhausted: Check response.status_code and look it up in API Error Codes. Some failures (like an invalid apikey) won’t be fixed by retrying and need a code change instead.
  • Retries seem to take a long time: That’s the backoff by design. Lower retries or retry_budget if latency matters more than success rate for your use case.
  • Getting exceptions instead of error responses: That only happens for network-level failures (timeouts, DNS errors), not HTTP error status codes. Wrap the call in try/except requests.exceptions.RequestException as shown above.
  • 429 errors even with retries: You’re likely exceeding your plan’s concurrency limit, not hitting a one-off rate limit. See Async Requests and Concurrency.

Pricing

Retried attempts that fail are not billed. Only the final successful response counts. This makes retries effectively free insurance against transient failures:
404 and 410 responses count as successful and are billed, since the request itself completed correctly.

FAQ

The default number of retries is 0. Retries are off unless you pass retries=N to the constructor.
No. Only successful requests consume your Zenrows balance.
Yes for a connect timeout, once you set retries: that request never reached the API. A read timeout is not retried. It raises requests.ReadTimeout at once, because the API may still finish and charge that request. Handle both with your own try/except, as shown above.

Look up a specific error code

For a full breakdown of Zenrows error codes (like AUTH002 or RESP001) and how to resolve each one, see the API Error Codes reference.