> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Crawl Endpoints

> Reference for the Zenrows Crawl endpoints: parameters, response fields, statuses, webhook events, limits, pricing and error codes.

Crawl's endpoints live under `https://api.zenrows.com/v1/crawls`. Send your API key in the `X-API-Key` header or the `apikey` query parameter.

> [!TIP]
>
> Each endpoint on this page also has its own page in the [API Reference](https://docs.zenrows.com/api-reference.md#endpoint-reference), with typed parameters, a response example for every status code, and a playground that sends the request with your key.

| Method | Path | Does |
| - | - | - |
| `POST` | `/v1/crawls` | [Start a crawl](#start-a-crawl) |
| `GET` | `/v1/crawls` | [List your crawls](#list-your-crawls) |
| `GET` | `/v1/crawls/{crawl_id}` | [Read a crawl and its results](#read-a-crawl) |
| `GET` | `/v1/crawls/{crawl_id}/download` | [Download all results](#download-the-results) |
| `GET` | `/v1/crawls/{crawl_id}/contents/{content_id}` | [Read one page](#read-a-page) |
| `POST` | `/v1/crawls/{crawl_id}/stop` | [Stop a crawl](#stop-a-crawl) |

## Start a crawl

### `POST /v1/crawls`

Starts a crawl and returns `202 Accepted` at once, with the crawl in the body and its path in the `Location` header. The body is JSON.

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **url** `required` | `string` | | The start URL: a public `http` or `https` URL, up to 2,048 characters. The crawl starts on this exact URL, so one ending in `?page=3` starts on page 3. It stays on the URL's domain, subdomains included. Two sites on a shared hosting domain, such as two `github.io` sites, count as two sites. Crawl never opens or returns a link that would change something on the site, such as a log-out, unsubscribe, delete, checkout or add-to-cart link, or a link to a host that is not public. A link longer than 2,048 characters on a crawled page is skipped. |
| **depth** | `integer` | | How many link steps to follow from the start URL, from 1 to 100,000. At `1`, Crawl returns the links on the start page. At `2`, it also opens those pages and returns their links. Leave it out to follow links until `max_items` or `max_pages` stops the crawl. |
| **max\_items** | `integer` | `100` | Stops the crawl after it keeps this many URLs, from 1 to 100,000. With `output_format`, `max_pages` can stop it first, because each kept page takes a fetch. |
| **max\_pages** | `integer` | `100` | Stops the crawl after it fetches this many pages, from 1 to 100,000. With `output_format`, each kept page's fetch counts too. If both limits are reached together, `stop_reason` is `max_pages`. |
| **include\_patterns** | `array` of `string` | | Keeps a URL only if it contains at least one of these texts, such as `/product/`, and opens only those URLs. A homepage crawl with only `/product/` skips listing URLs that lack that text, such as `/page/`. The start URL is never returned. |
| **exclude\_patterns** | `array` of `string` | | Drops a URL that contains any of these texts, and doesn't open it, even if it matches an include pattern. |
| **output\_format** | `string` | | Also returns each kept page: `html` as fetched, `markdown` converted to Markdown. Leave it out to get URLs only. |
| **discovery** | `array` of `string` | `["links"]` | How the crawl finds the URLs it follows on each page it walks. `links` follows the links in the page's HTML. |
| **webhook** | `object` | | Where to send the crawl's events, and which ones. Set it when you start the crawl: it can't be changed later. Your account needs a webhook signing key. See [Webhooks](#webhooks). |
| **webhook.url** `required` | `string` | | The absolute `https` URL each event is posted to, up to 2,048 characters. Its host must resolve to a public address. |
| **webhook.events** | `array` of `string` | all four | The events to send, each named once: `crawl.started`, `crawl.completed`, `crawl.stopped`, `crawl.failed`. Leave it out to get all four. |
| **webhook.metadata** | `object` | | Your own data, sent back unchanged in every event, to match it to your records. Up to 20 keys of 1 to 64 characters. Each value is any JSON, up to 1,024 bytes as compact UTF-8 JSON. |

Send an `Idempotency-Key` header (up to 255 characters) to make a retry safe: a retry with the same key and body returns the same crawl and starts no second one.

## List your crawls

### `GET /v1/crawls`

Returns your crawls, newest first, without their results.

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **cursor** | `string` | | The `next_cursor` from the previous page. |
| **limit** | `integer` | `20` | How many crawls to return, from 1 to 100. |

The response holds a `crawls` array and a `next_cursor`. `next_cursor` is absent on the last page.

## Read a crawl

### `GET /v1/crawls/{crawl_id}`

Returns the crawl, its coverage so far, and one page of results.

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **cursor** | `string` | | The `next_cursor` from the previous read. Leave it out to start from the first result. |
| **limit** | `integer` | `1000` | How many results to return, from 1 to 10,000. |

While the crawl runs, and while a stopped crawl still reads the pages fetched before the stop, `next_cursor` is never `null`: read again with the same cursor to get only the new URLs. `next_cursor` is `null` once the crawl has ended and you have read its last results. Each URL is returned once.

### Response fields

| Field | Description |
| - | - |
| `crawl_id` | The crawl's ID. |
| `status` | `running`, `completed`, `stopped` or `failed`. See [Statuses](#statuses). |
| `stop_reason` | Why the crawl ended early: `max_items`, `max_pages` or `user`. Absent when nothing was left to open, when the crawl failed, and when it stopped because your account ran out of credits or the API key hit a credit cap. |
| `error` | Why the crawl failed or stopped on credits, with a `code` and a `detail`. Present when `status` is `failed`, and when it is `stopped` with `error.code` `insufficient_credits` or `api_key_cap_reached`. |
| `url`, `depth`, `max_items`, `max_pages`, `include_patterns`, `exclude_patterns`, `output_format` | The parameters the crawl runs with. `depth` is absent when you left it out. |
| `discovery` | The strategies the crawl finds URLs with, such as `["links"]`: the links in each page's HTML. |
| `webhook` | The crawl's webhook, when you started it with one: its `url`, `events` (the events it sends, all four when you left them out) and `metadata` (`null` when you set none). Absent on a crawl started without a webhook. |
| `coverage.pages_fetched` | Pages fetched so far, successful or not, at most `max_pages`. With `output_format`, this includes each kept page fetched for its content. |
| `coverage.pages_failed` | Pages that failed to load, or loaded empty. |
| `coverage.items_found` | URLs the crawl returns so far, after duplicates and your patterns are applied. At most `max_items`. |
| `duplicates_removed` | Links the crawl met again and dropped. The results never hold a duplicate. |
| `created_at`, `finished_at` | When the crawl started and ended, in UTC. `finished_at` is absent while it runs. |
| `results[].url` | A URL the crawl found, normalized. Spellings of one URL count as one: `http` or `https`, with or without `www.` and a trailing slash. Tracking parameters, such as `utm_*`, `gclid` and `fbclid`, are removed. |
| `results[].content_status` | With `output_format` only: `pending`, `fetched` or `failed`. |
| `results[].content_url` | With `output_format` only, once `content_status` is `fetched`: the path to [read the page](#read-a-page). |
| `next_cursor` | Send it as `cursor` on the next read. |

### Statuses

| `status` | Meaning |
| - | - |
| `running` | The crawl is opening pages. Results grow as it goes. |
| `completed` | The crawl finished. Nothing was left to open, or it reached `max_items` or `max_pages` (see `stop_reason`). A crawl whose pages held nothing new to follow also ends `completed`, with no results and no `error`, even when the start page was the only page it read: the pages had no link, only links to other sites, only links your patterns reject, or only links it already met. If a crawl with `include_patterns` or `exclude_patterns` ends with no results, check them against the URLs the start page links to. A block, consent or challenge page that shows text is read like any other page: the crawl follows its links, and ends `completed` with no results when it holds no link on the site. |
| `stopped` | You stopped it with [`POST /v1/crawls/{crawl_id}/stop`](#stop-a-crawl), and `stop_reason` is `user`. Or your account ran out of credits, or the API key hit one of its credit caps, while the crawl ran: then `stop_reason` is absent and `error.code` is `insufficient_credits` or `api_key_cap_reached`. The results hold the URLs kept before the stop and the links on the pages already fetched by then. |
| `failed` | The crawl could not go on for any other reason. `error.code` says why. The URLs found so far stay readable. Ending with no results is not a failure: see `completed`. |

| `error.code` | Meaning | What to do |
| - | - | - |
| `seed_unreachable` | The start URL could not be fetched, or it returned an empty page: no visible text and no link, such as an empty body or a challenge page without its content. A later page that fails or is empty counts in `coverage.pages_failed` and does not fail the crawl. | Check the URL. The site may be blocking it. |
| `no_items_found` | The crawl read the start page but found no data it could use, so it had nothing to follow. A crawl started with the parameters on this page doesn't return this code: when it has nothing to follow, it ends `completed` with no results. | Start from a listing or product page. |
| `insufficient_credits` | Your account ran out of credits. The crawl ends `stopped`, and the pages it fetched before that stay readable. | Add credits or upgrade your plan, then start a new crawl. |
| `api_key_cap_reached` | The API key that started the crawl reached one of its credit caps. The crawl ends `stopped`, and the pages it fetched before that stay readable. | Raise or remove the cap, or wait until it resets, then start a new crawl. |
| `internal_error` | Something failed on our side. | Start a new crawl. If it fails again, contact support with the `crawl_id`. |

## Download the results

### `GET /v1/crawls/{crawl_id}/download`

Returns every result as one NDJSON file (one JSON object per line), named `{crawl_id}.jsonl`. Each line has the `url`. With `output_format`, each line also has `content_status` and, once fetched, the page in `content`.

The `X-Crawl-Status` header holds the crawl's status. While it is `running`, a later download may hold more lines. After a stop, it reads `stopped`, yet a later download may still hold more lines for up to 10 minutes: read the crawl until `next_cursor` is `null` to know the results are final.

## Read a page

### `GET /v1/crawls/{crawl_id}/contents/{content_id}`

Returns one kept page, in the crawl's `output_format`, for a crawl started with one. Take the path from the result's `content_url`.

Add `?download=true` to get the page as a file to save rather than one to show in the browser; the page itself is the same.

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **download** | `boolean` | `false` | `true` returns the page as a file to save. |

## Stop a crawl

### `POST /v1/crawls/{crawl_id}/stop`

Stops a running crawl and returns `200` with its `crawl_id`, `status`, `stop_reason` and `finished_at`. Send no request body.

```json Response theme={"theme":"css-variables"}
{
  "crawl_id": "c_01M3XP7RZD4JT88F7PNPGBFRAF",
  "status": "stopped",
  "stop_reason": "user",
  "finished_at": "2026-10-09T09:01:30Z"
}
```

No new page is fetched after the stop. Pages already being fetched finish and are billed, and the crawl keeps them: for up to 10 minutes after the stop, it reads each one, counts it in `coverage`, and keeps the links on it within your patterns and `max_items`. With `output_format`, a URL kept after the stop reads `content_status: pending`, since no new page is fetched. `status`, `stop_reason` and `finished_at` stay as the stop set them. Read the crawl until `next_cursor` is `null` for its final results. A crawl that already ended returns its final state. If the stop returns `500 internal_error`, send it again. If it returns `503 crawl_busy`, the crawl is still running: send it again after `Retry-After` seconds.

## Webhooks

A crawl started with a `webhook` posts an event to your URL when it starts and when it ends, so you don't have to poll it. [Crawl setup](https://docs.zenrows.com/crawl/setup.md#get-notified-by-webhook) shows how to start one and verify its events.

### Events

| `event_type` | Sent when |
| - | - |
| `crawl.started` | The crawl was created and is running. Sent once. |
| `crawl.completed` | The crawl reached `status: completed`. |
| `crawl.stopped` | The crawl reached `status: stopped`: you stopped it, or your account ran out of credits or the API key hit a credit cap. Sent after the crawl has read the pages that were still being fetched at the stop. |
| `crawl.failed` | The crawl reached `status: failed`. |

Exactly one of `crawl.completed`, `crawl.stopped` and `crawl.failed` is sent per crawl, and only if it is in `webhook.events`.

### Payload

Each event is a `POST` with a JSON body:

```json crawl.completed theme={"theme":"css-variables"}
{
  "schema": "zenrows.webhook/v1",
  "event_id": "ev_01M3XP7RZD4JT88F7PNPGBFRAF_completed",
  "event_type": "crawl.completed",
  "occurred_at": "2026-10-09T09:04:12Z",
  "resource": {
    "type": "crawl",
    "id": "c_01M3XP7RZD4JT88F7PNPGBFRAF",
    "url": "https://api.zenrows.com/v1/crawls/c_01M3XP7RZD4JT88F7PNPGBFRAF"
  },
  "metadata": {
    "site": "scrapingcourse",
    "run": 42
  }
}
```

| Field | Description |
| - | - |
| `schema` | Always `zenrows.webhook/v1`. |
| `event_id` | The event's ID: `ev_`, the crawl ID without its `c_` prefix, `_`, and the last part of the event type. It is the same on every delivery of one event, so drop an `event_id` you have already handled. |
| `event_type` | One of the [events](#events). |
| `occurred_at` | When the crawl was created, for `crawl.started`; its `finished_at`, for the others. In UTC. |
| `resource.type` | Always `crawl`. |
| `resource.id` | The crawl's `crawl_id`. |
| `resource.url` | The crawl on the API. [Read it](#read-a-crawl) for its status, coverage and results. |
| `metadata` | The `webhook.metadata` you set, as you sent it. Absent when you set none. |

The event tells you that something happened, not the results. Read the crawl at `resource.url` for them.

### Headers

| Header | Value |
| - | - |
| `Content-Type` | `application/json` |
| `User-Agent` | `zenrows-webhook/v1` |
| `X-ZenRows-Event-Id` | The event's `event_id`. |
| `X-Signature` | `t=<unix seconds>,v1=<hex>,kid=<key id>`. `v1` is the lowercase hex HMAC-SHA256 of `<t>.<raw body>`, keyed with your account's webhook signing key. `kid` names the key that signed it. |

Every event is signed. To verify one, base64-decode your signing key, compute the HMAC over `t`, a dot and the raw body exactly as received, compare it to `v1` in constant time, and reject a `t` more than 5 minutes from your clock. Code for this is in [Crawl setup](https://docs.zenrows.com/crawl/setup.md#verify-an-event).

### Signing key

Create your account's webhook signing key in the dashboard, under **Settings**, **API Keys**. The secret is shown once, base64-encoded: store it when you create it. While you rotate the key, either the old or the new key may sign an event, so keep both and pick one by the header's `kid`. A crawl with a `webhook` can't start without a signing key: it returns `422 webhook_signing_key_required`.

### Delivery and retries

Answer with any `2xx` within 10 seconds, and do your processing afterwards. A delivery that gets another status, or no answer in time, is retried with backoff for up to 3 days, then dropped.

Because of retries, one event can arrive more than once: deduplicate on `event_id` or the `X-ZenRows-Event-Id` header. Events can also arrive out of order, so a short crawl's `crawl.completed` can come before its `crawl.started`. Order them by `occurred_at`, or read the crawl at `resource.url` for its current status.

## Limits

| Limit | Value |
| - | - |
| Active crawls per account | 3 by default, shared with your [Batch](https://docs.zenrows.com/batch/introduction.md) jobs |
| `max_items` | 100 by default, 100,000 at most |
| `max_pages` | 100 by default, 100,000 at most |
| `depth` | No limit by default, from 1 to 100,000 when set |
| Results per read | 1,000 by default, 10,000 at most |

An account runs 3 active jobs by default, shared with [Batch](https://docs.zenrows.com/batch/introduction.md). One more returns `429 too_many_crawls`, with a `Retry-After` header of 30 seconds. To run more than 3 at once, use the chat in your dashboard. Email [success@zenrows.com](mailto:success@zenrows.com) if you can't use the chat.

[Open the dashboard](https://app.zenrows.com)

## Pricing

Crawl has no fee of its own. Each page a crawl opens is one Fetch (formerly, Universal Scraper API) request with [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode.md), billed at that mode's price: from 1 to 25 credits per page, depending on the site. A page that fails to load (blocked, timed out or server error) is not billed. A page that answers 404 or 410 counts as failed but is billed, as in Fetch. A refused request starts no crawl and costs nothing.

`max_pages` caps how many pages a crawl fetches, so it caps what the crawl costs. With `output_format`, each kept page counts as one fetch. A kept page that the crawl also opens is fetched once, unless the crawl fetches it for its content first and reaches it later from a page nearer the start: then it is fetched, and billed, twice.

## Errors

Errors return as `application/problem+json`, with a `code`, a `title`, a `detail` that says what to do, the `status` and an `instance` to quote to support.

| Code | Status | Meaning |
| - | - | - |
| `invalid_request` | `400` | The body is not valid JSON, the `Idempotency-Key` is over 255 characters, or the list's `limit` or `cursor` is not valid. |
| `unknown_parameter` | `400` | The request has a body field or query parameter this endpoint does not accept. `detail` names it. |
| `invalid_cursor` | `400` | The `cursor` does not belong to this crawl. |
| `invalid_parameter` | `400`, `422` | A value is missing or out of range. `detail` names the field. |
| `crawl_not_found` | `404` | No crawl with this ID exists on your account. |
| `content_not_found` | `404` | The page is not fetched yet, its fetch failed, or the crawl has no `output_format`. |
| `idempotency_request_in_flight` | `409` | A request with this `Idempotency-Key` is still in progress. Retry when it ends. |
| `invalid_start_url` | `422` | The start URL is not a public `http` or `https` URL: its host is an IP address or a private address, or it is over 2,048 characters. |
| `idempotency_key_reused` | `422` | This `Idempotency-Key` was used with a different body. |
| `invalid_webhook` | `422` | The `webhook` breaks a rule: it is not an object, its `url` is missing, not an absolute `https` URL, over 2,048 characters, or on a host that does not resolve to a public address, or its `events` or `metadata` are out of bounds. `detail` names the field. |
| `webhook_signing_key_required` | `422` | The request has a `webhook`, but your account has no webhook signing key. Create one in the dashboard, under **Settings**, **API Keys**, then retry. No crawl was started. |
| `too_many_crawls` | `429` | Your account already runs 3 active jobs by default, shared with [Batch](https://docs.zenrows.com/batch/introduction.md). Wait for one to end. |
| `crawl_busy` | `503` | The crawl was saving its progress and the stop could not be saved. Retry after `Retry-After` seconds. |
| `internal_error` | `500` | Something failed on our side. Retry later. |

For authentication and plan errors, see the [API error codes](https://docs.zenrows.com/api-error-codes.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.