> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Scrape pages protected by anti-bot systems

> Get the real content of pages behind Cloudflare and other anti-bot systems with Zenrows Fetch, and see what each request cost.

Send a protected URL to Fetch with `mode=auto` and get the page a real visitor sees, not the challenge screen. Reach for this recipe whenever a plain request returns a block page, a CAPTCHA or a `REQS002` error.

**Uses:** [Fetch](https://docs.zenrows.com/fetch/api-reference.md) with [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode.md) (`mode=auto`), [`js_render`](https://docs.zenrows.com/fetch/features/js-rendering) and [`premium_proxy`](https://docs.zenrows.com/fetch/features/premium-proxy)

## Ask your AI agent

With the [Zenrows MCP server](https://docs.zenrows.com/mcp/overview.md) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"theme":"css-variables"}
Use Zenrows to read https://www.scrapingcourse.com/antibot-challenge
and tell me the exact message shown in the challenge box.
```

## Let Adaptive Stealth Mode pick the configuration

Set `mode=auto`. Zenrows starts with the cheapest configuration, adds JavaScript rendering and premium proxies only when the page needs them, and bills only the configuration that returns the page.

```python Python (Requests) theme={"theme":"css-variables"}
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/antibot-challenge",
    "mode": "auto",
}

response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

print("Bypassed:", "You bypassed the Antibot challenge" in response.text)
print("Credits billed:", response.headers["X-Request-Credits"])
print("Cost (USD):", response.headers["X-Request-Cost"])
```

```javascript Node.js theme={"theme":"css-variables"}
const params = new URLSearchParams({
  apikey: "YOUR_ZENROWS_API_KEY",
  url: "https://www.scrapingcourse.com/antibot-challenge",
  mode: "auto",
});

const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
  signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);

const html = await response.text();
console.log("Bypassed:", html.includes("You bypassed the Antibot challenge"));
console.log("Credits billed:", response.headers.get("X-Request-Credits"));
console.log("Cost (USD):", response.headers.get("X-Request-Cost"));
```

```bash cURL theme={"theme":"css-variables"}
curl -s -D - -o page.html "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto"
```

```bash CLI theme={"theme":"css-variables"}
zenrows fetch https://www.scrapingcourse.com/antibot-challenge --json
```

The cURL command prints the response headers and saves the page to `page.html`. The CLI uses `mode=auto` by default and prints the billed credits after every request.

## Example output

```text theme={"theme":"css-variables"}
Bypassed: True
Credits billed: 25
Cost (USD): 0.025
```

The CLI's `--json` result carries the same numbers:

```json theme={"theme":"css-variables"}
{
  "ok": true,
  "mode": "auto",
  "httpStatus": 200,
  "costUsd": 0.025,
  "costCredits": 25,
  "finalUrl": "https://www.scrapingcourse.com/antibot-challenge"
}
```

## Read which configuration was billed

Every Fetch response carries the charge in its headers, so you can log it next to each URL:

| Header | Meaning | Example |
| - | - | - |
| `X-Request-Credits` | Credits this request consumed | `25` |
| `X-Request-Cost` | The same charge in USD (`0.001` = 1 credit) | `0.025` |
| `X-Request-Id` | ID to quote when you contact support | `6ad0054e...` |

With `mode=auto`, the billed credits tell you which configuration succeeded: `1` for a basic request, `5` with JavaScript rendering, `10` with premium proxies, `25` with both. A domain that always bills `25` is a good candidate for the fixed configuration below. Requests that don't succeed aren't charged.

## Fix the configuration with js\_render and premium\_proxy

When you already know a domain needs the full stack, set `js_render=true` and `premium_proxy=true` yourself. Every request then goes straight to that configuration. Send these parameters instead of `mode=auto`, never together with it.

```python Python (Requests) theme={"theme":"css-variables"}
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/antibot-challenge",
    "js_render": "true",
    "premium_proxy": "true",
}

response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

print("Bypassed:", "You bypassed the Antibot challenge" in response.text)
print("Credits billed:", response.headers["X-Request-Credits"])
```

```javascript Node.js theme={"theme":"css-variables"}
const params = new URLSearchParams({
  apikey: "YOUR_ZENROWS_API_KEY",
  url: "https://www.scrapingcourse.com/antibot-challenge",
  js_render: "true",
  premium_proxy: "true",
});

const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
  signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);

const html = await response.text();
console.log("Bypassed:", html.includes("You bypassed the Antibot challenge"));
console.log("Credits billed:", response.headers.get("X-Request-Credits"));
```

```bash cURL theme={"theme":"css-variables"}
curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true"
```

```bash CLI theme={"theme":"css-variables"}
zenrows fetch https://www.scrapingcourse.com/antibot-challenge --manual --js-render --premium-proxy --out page.html
```

The same code works on Cloudflare-protected pages: swap the URL for `https://www.scrapingcourse.com/cloudflare-challenge` and check for `You bypassed the Cloudflare challenge`.

## Customize it

| Change | How |
| - | - |
| Content from a specific country | Add `proxy_country=us` (any ISO code), see [Get content as seen from any country](https://docs.zenrows.com/academy/fetch/geotargeting.md) |
| Wait for content that loads late | Use `js_render=true` with `wait_for=<selector>`, see [Scrape JavaScript-rendered pages](https://docs.zenrows.com/academy/fetch/javascript-pages.md) |
| Markdown instead of HTML | Add `response_type=markdown`, see [Turn any web page into LLM-ready markdown](https://docs.zenrows.com/academy/fetch/markdown-for-llms.md) |
| Get the target's own status code | Add `original_status=true`, see [Original HTTP code](https://docs.zenrows.com/fetch/features/other.md#original-http-code) |
| Click or fill before reading | Add `js_instructions`, see [Click, fill, and scroll before you scrape](https://docs.zenrows.com/academy/fetch/page-interactions.md) |

## Troubleshooting

* **`400` with code `REQS002`**: the domain needs JavaScript rendering, premium proxies or both. Send `mode=auto`, or add `js_render=true` and `premium_proxy=true`. See [REQS002](https://docs.zenrows.com/api-error-codes.md#REQS002).
* **`400` with code `REQS004` mentioning `mode`**: `js_render` or `premium_proxy` was sent together with `mode=auto`. Keep one approach per request.
* **`422` with code `RESP001`**: the page did not come back on that attempt. Retry with backoff and, on a fixed configuration, switch to `mode=auto`. See [Retry failed requests](https://docs.zenrows.com/academy/retry-failed-requests.md).
* **`499` or a client-side timeout**: your HTTP client gave up first. Set its timeout to 180 seconds or more.
* **The page loads but a section is empty**: the content arrives after the first render. Add `wait_for` with a selector of that section.

## Related recipes

> [card] [Scrape JavaScript-rendered pages](https://docs.zenrows.com/academy/fetch/javascript-pages.md)
>
> Render the page and wait for the exact element you need.

> [card] [Retry failed requests](https://docs.zenrows.com/academy/retry-failed-requests.md)
>
> Back off, retry and escalate to mode=auto on 422.

> [card] [Get content as seen from any country](https://docs.zenrows.com/academy/fetch/geotargeting.md)
>
> Route the request through a residential IP in any country.

> [card] [Track usage and control spend](https://docs.zenrows.com/academy/production/control-costs.md)
>
> Pick the cheapest configuration that works for each domain.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.