> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Crawl Setup

> Start a Zenrows crawl, read its status and results with a cursor, download every result in one file, get each page as HTML or Markdown, stop a crawl, and get notified by webhook when it starts and ends.

This guide takes one start URL through a full crawl. Every request sends your API key in the `X-API-Key` header. Parameters, limits and errors are in the [endpoint reference](https://docs.zenrows.com/crawl/endpoints.md).

## Start a crawl

Send the start URL to `POST /v1/crawls`. Add `depth` to limit how many link steps it follows. Without it, the crawl follows links until `max_items` or `max_pages` stops it. Add `max_items` to keep more than the default 100 URLs. `include_patterns` keeps a URL only if it contains one of the strings you pass, and Crawl opens only those URLs. A homepage crawl with only `/product/` doesn't open listing paths that lack that text, such as `/page/`.

```python Python theme={"theme":"css-variables"}
# pip install requests
import requests

response = requests.post(
    'https://api.zenrows.com/v1/crawls',
    headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'},
    json={
        'url': 'https://www.scrapingcourse.com/ecommerce/',
        'depth': 2,
        'max_pages': 20,
        'include_patterns': ['/product/'],
    },
)
crawl = response.json()
print(response.status_code, crawl['crawl_id'])
```

```javascript Node.js theme={"theme":"css-variables"}
const response = await fetch('https://api.zenrows.com/v1/crawls', {
    method: 'POST',
    headers: {
        'X-API-Key': 'YOUR_ZENROWS_API_KEY',
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({
        url: 'https://www.scrapingcourse.com/ecommerce/',
        depth: 2,
        max_pages: 20,
        include_patterns: ['/product/'],
    }),
});
const crawl = await response.json();
console.log(response.status, crawl.crawl_id);
```

```bash cURL theme={"theme":"css-variables"}
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "depth": 2,
    "max_pages": 20,
    "include_patterns": ["/product/"]
  }'
```

The response is `202 Accepted`. The crawl runs in the background:

```json Response theme={"theme":"css-variables"}
{
  "crawl_id": "c_01M3XP7RZD4JT88F7PNPGBFRAF",
  "status": "running",
  "url": "https://www.scrapingcourse.com/ecommerce/",
  "depth": 2,
  "max_items": 100,
  "max_pages": 20,
  "include_patterns": ["/product/"],
  "exclude_patterns": [],
  "discovery": ["links"],
  "coverage": {
    "pages_fetched": 0,
    "pages_failed": 0,
    "items_found": 0
  },
  "duplicates_removed": 0,
  "created_at": "2026-10-09T09:00:00Z"
}
```

Keep the `crawl_id`. Every other call takes it.

## Read the status and the results

`GET /v1/crawls/{crawl_id}` returns the crawl's `status`, its `coverage` so far and one page of results. Results grow while the crawl runs, so read them as you go:

1. Read without `cursor` to get the first results.
2. Send the `next_cursor` you received as `cursor` to get the URLs found since.
3. Stop when `next_cursor` is `null`. This happens only once the crawl has ended and you have read its last results.

```python Python theme={"theme":"css-variables"}
# pip install requests
import time
import requests

crawl_id = 'YOUR_CRAWL_ID'
url = f'https://api.zenrows.com/v1/crawls/{crawl_id}'
headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}

urls, cursor = [], None
while True:
    params = {'cursor': cursor} if cursor else {}
    body = requests.get(url, headers=headers, params=params).json()
    urls += [result['url'] for result in body['results']]
    if body['next_cursor'] is None:
        break
    cursor = body['next_cursor']
    if not body['results']:
        time.sleep(5)  # nothing new yet

print(body['status'], len(urls))
```

```javascript Node.js theme={"theme":"css-variables"}
const crawlId = 'YOUR_CRAWL_ID';
const url = `https://api.zenrows.com/v1/crawls/${crawlId}`;
const headers = { 'X-API-Key': 'YOUR_ZENROWS_API_KEY' };

const urls = [];
let cursor = null;
let body;
while (true) {
    const params = cursor ? `?cursor=${encodeURIComponent(cursor)}` : '';
    body = await (await fetch(url + params, { headers })).json();
    urls.push(...body.results.map((result) => result.url));
    if (body.next_cursor === null) break;
    cursor = body.next_cursor;
    if (body.results.length === 0) {
        await new Promise((resolve) => setTimeout(resolve, 5000)); // nothing new yet
    }
}

console.log(body.status, urls.length);
```

```bash cURL theme={"theme":"css-variables"}
curl "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY"
```

A crawl that has ended reads like this:

```json Response theme={"theme":"css-variables"}
{
  "crawl_id": "c_01M3XP7RZD4JT88F7PNPGBFRAF",
  "status": "completed",
  "stop_reason": "max_items",
  "url": "https://www.scrapingcourse.com/ecommerce/",
  "depth": 2,
  "max_items": 100,
  "max_pages": 20,
  "include_patterns": ["/product/"],
  "exclude_patterns": [],
  "discovery": ["links"],
  "coverage": {
    "pages_fetched": 12,
    "pages_failed": 0,
    "items_found": 100
  },
  "duplicates_removed": 214,
  "created_at": "2026-10-09T09:00:00Z",
  "finished_at": "2026-10-09T09:01:12Z",
  "results": [
    { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/" },
    { "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/" }
  ],
  "next_cursor": null
}
```

`stop_reason: max_items` means the site may hold more URLs. To get them, start a new crawl with a higher `max_items` (and a higher `max_pages` if `stop_reason` is `max_pages`).

## Download all results

`GET /v1/crawls/{crawl_id}/download` returns every URL in one NDJSON file: one JSON object per line.

```bash cURL theme={"theme":"css-variables"}
curl "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID/download" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -o crawl.jsonl
```

```json Response theme={"theme":"css-variables"}
{"url":"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"}
{"url":"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/"}
```

You can download while the crawl runs. The `X-Crawl-Status` header tells you the crawl's status: `running` means a later download may hold more lines. So can a download in the 10 minutes after a [stop](#stop-a-crawl), while the header reads `stopped`.

## Get each page

Send `"output_format"` to get each page the crawl keeps, not only its URL: `"html"` for the page as fetched, `"markdown"` for it converted to Markdown. Each kept page counts toward `max_pages`, so set `max_pages` to at least `max_items` plus the pages the crawl walks:

```bash cURL theme={"theme":"css-variables"}
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "depth": 1,
    "max_items": 10,
    "max_pages": 20,
    "include_patterns": ["/product/"],
    "output_format": "html"
  }'
```

Each result then has a `content_status`, and a `content_url` once the page is fetched:

```json Response theme={"theme":"css-variables"}
{
  "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
  "content_status": "fetched",
  "content_url": "/v1/crawls/c_01M3XP24VJ55Q96QQYKT6FQW1J/contents/ct_4bf794e572e17fdd50710f61cc730066"
}
```

`content_status` is `pending` until the page is fetched, then `fetched` or `failed`. After a [stop](#stop-a-crawl), a page not fetched yet stays `pending`. Read a fetched page at its `content_url`:

```bash cURL theme={"theme":"css-variables"}
curl "https://api.zenrows.com/v1/crawls/c_01M3XP24VJ55Q96QQYKT6FQW1J/contents/ct_4bf794e572e17fdd50710f61cc730066" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -o page.html
```

The [download](#download-all-results) also includes each fetched page, as HTML or Markdown, in a `content` field.

## Stop a crawl

`POST /v1/crawls/{crawl_id}/stop` stops a running crawl:

```bash cURL theme={"theme":"css-variables"}
curl -X POST "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID/stop" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY"
```

The crawl then reads `status: stopped` with `stop_reason: user`. The URLs found before the stop stay readable. Pages already being fetched finish, are billed, and stay in the crawl: for up to 10 minutes after the stop, the links on them join the results. Keep reading with `next_cursor` until it is `null` for the final results. A stopped crawl can't resume.

## Get notified by webhook

Instead of polling, give the crawl a `webhook` when you start it. Zenrows then posts an event to your `https` URL when the crawl starts and when it ends. Your account needs a webhook signing key first: create it in the dashboard, under **Settings**, **API Keys**, and store the secret, since it is shown only once. Without the key, the request returns `422 webhook_signing_key_required`.

`events` narrows what you get: the request below asks only for the crawl's end. Leave it out to get all four events. `metadata` comes back in every event, to match the event to your own records.

```python Python theme={"theme":"css-variables"}
# pip install requests
import requests

response = requests.post(
    'https://api.zenrows.com/v1/crawls',
    headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'},
    json={
        'url': 'https://www.scrapingcourse.com/ecommerce/',
        'max_pages': 20,
        'webhook': {
            'url': 'https://hooks.example.com/zenrows',
            'events': ['crawl.completed', 'crawl.stopped', 'crawl.failed'],
            'metadata': {'site': 'scrapingcourse', 'run': 42},
        },
    },
)
crawl = response.json()
print(response.status_code, crawl['crawl_id'], crawl['webhook'])
```

```javascript Node.js theme={"theme":"css-variables"}
const response = await fetch('https://api.zenrows.com/v1/crawls', {
    method: 'POST',
    headers: {
        'X-API-Key': 'YOUR_ZENROWS_API_KEY',
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({
        url: 'https://www.scrapingcourse.com/ecommerce/',
        max_pages: 20,
        webhook: {
            url: 'https://hooks.example.com/zenrows',
            events: ['crawl.completed', 'crawl.stopped', 'crawl.failed'],
            metadata: { site: 'scrapingcourse', run: 42 },
        },
    }),
});
const crawl = await response.json();
console.log(response.status, crawl.crawl_id, crawl.webhook);
```

```bash cURL theme={"theme":"css-variables"}
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "max_pages": 20,
    "webhook": {
      "url": "https://hooks.example.com/zenrows",
      "events": ["crawl.completed", "crawl.stopped", "crawl.failed"],
      "metadata": {"site": "scrapingcourse", "run": 42}
    }
  }'
```

The crawl in the response, and on every read, shows the webhook it runs with:

```json theme={"theme":"css-variables"}
"webhook": {
  "url": "https://hooks.example.com/zenrows",
  "events": ["crawl.completed", "crawl.stopped", "crawl.failed"],
  "metadata": {"site": "scrapingcourse", "run": 42}
}
```

When the crawl ends, your URL receives one event, `crawl.completed`, `crawl.stopped` or `crawl.failed`, with the crawl's ID and its API URL in `resource`. The event carries no results: read the crawl at `resource.url` for them. Every field and header is in the [endpoint reference](https://docs.zenrows.com/crawl/endpoints.md#webhooks).

### Verify an event

Every event carries an `X-Signature` header, `t=<unix seconds>,v1=<hex>,kid=<key id>`. Check it before you trust the event:

1. Pick your secret by `kid`. While you rotate the key, either key may sign, so keep both.
2. Base64-decode the secret. The HMAC key is the decoded bytes, not the text.
3. Compute the hex HMAC-SHA256 of `t`, a dot, and the raw request body exactly as received, before any JSON parsing.
4. Compare it to `v1` in constant time.
5. Reject a `t` more than 5 minutes from your clock, so a captured event can't be replayed.

Then answer `2xx` within 10 seconds and process the event afterwards. A delivery that gets no `2xx` in time is retried with backoff for up to 3 days, so the same event can arrive twice: skip an `X-ZenRows-Event-Id` you have already handled. Events can arrive out of order, so order them by `occurred_at`.

```python Python theme={"theme":"css-variables"}
import base64, hashlib, hmac, time

# Your signing keys by key id, as base64 strings.
SECRETS = {'YOUR_KEY_ID': 'YOUR_BASE64_SIGNING_KEY'}

def verify(raw_body: bytes, signature_header: str) -> bool:
    if not signature_header:
        return False
    parts = dict(p.split('=', 1) for p in signature_header.split(',') if '=' in p)
    secret = SECRETS.get(parts.get('kid', ''))
    if secret is None or 'v1' not in parts:
        return False
    try:
        t = int(parts['t'])
    except (KeyError, ValueError):
        return False
    if abs(time.time() - t) > 300:  # older than 5 minutes
        return False
    key = base64.b64decode(secret)
    expected = hmac.new(key, f'{t}.'.encode() + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts['v1'])
```

```javascript Node.js theme={"theme":"css-variables"}
import crypto from 'node:crypto';

// Your signing keys by key id, as base64 strings.
const SECRETS = { YOUR_KEY_ID: 'YOUR_BASE64_SIGNING_KEY' };

function verify(rawBody, signatureHeader) {
    if (!signatureHeader) return false;
    const parts = Object.fromEntries(
        signatureHeader.split(',').map((p) => {
            const i = p.indexOf('=');
            return [p.slice(0, i), p.slice(i + 1)];
        }),
    );
    const secret = SECRETS[parts.kid];
    if (!secret || !parts.v1) return false;
    const t = Number(parts.t);
    if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > 300) return false; // older than 5 minutes
    const expected = crypto
        .createHmac('sha256', Buffer.from(secret, 'base64'))
        .update(`${parts.t}.`)
        .update(rawBody) // the raw body Buffer, before JSON parsing
        .digest('hex');
    const a = Buffer.from(expected);
    const b = Buffer.from(parts.v1);
    return a.length === b.length && crypto.timingSafeEqual(a, b);
}

// With Express, keep the raw body:
// app.post('/zenrows', express.raw({ type: 'application/json' }), (req, res) => {
//     if (!verify(req.body, req.get('X-Signature'))) return res.sendStatus(401);
//     res.sendStatus(200);
//     handle(JSON.parse(req.body)); // skip an event_id you have already handled
// });
```

## Next steps

> [card] [Crawl Endpoints](https://docs.zenrows.com/crawl/endpoints.md)
>
> Every parameter, limit, status and error code, plus pricing.

> [card] [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode.md)
>
> How Fetch picks the settings for each page a crawl opens.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.