Skip to main content
This guide takes one start URL through a full crawl. Every request sends your API key in the X-API-Key header. Parameters, limits and errors are in the endpoint reference.

Start a crawl

Send the start URL to POST /v1/crawls. Add depth to limit how many link steps it follows. Without it, the crawl follows links until max_items or max_pages stops it. Add max_items to keep more than the default 100 URLs. include_patterns keeps a URL only if it contains one of the strings you pass, and Crawl opens only those URLs. A homepage crawl with only /product/ doesn’t open listing paths that lack that text, such as /page/.
The response is 202 Accepted. The crawl runs in the background:
Response
Keep the crawl_id. Every other call takes it.

Read the status and the results

GET /v1/crawls/{crawl_id} returns the crawl’s status, its coverage so far and one page of results. Results grow while the crawl runs, so read them as you go:
  1. Read without cursor to get the first results.
  2. Send the next_cursor you received as cursor to get the URLs found since.
  3. Stop when next_cursor is null. This happens only once the crawl has ended and you have read its last results.
A crawl that has ended reads like this:
Response
stop_reason: max_items means the site may hold more URLs. To get them, start a new crawl with a higher max_items (and a higher max_pages if stop_reason is max_pages).

Download all results

GET /v1/crawls/{crawl_id}/download returns every URL in one NDJSON file: one JSON object per line.
cURL
Response
You can download while the crawl runs. The X-Crawl-Status header tells you the crawl’s status: running means a later download may hold more lines. So can a download in the 10 minutes after a stop, while the header reads stopped.

Get each page

Send "output_format" to get each page the crawl keeps, not only its URL: "html" for the page as fetched, "markdown" for it converted to Markdown. Each kept page counts toward max_pages, so set max_pages to at least max_items plus the pages the crawl walks:
cURL
Each result then has a content_status, and a content_url once the page is fetched:
Response
content_status is pending until the page is fetched, then fetched or failed. After a stop, a page not fetched yet stays pending. Read a fetched page at its content_url:
cURL
The download also includes each fetched page, as HTML or Markdown, in a content field.

Stop a crawl

POST /v1/crawls/{crawl_id}/stop stops a running crawl:
cURL
The crawl then reads status: stopped with stop_reason: user. The URLs found before the stop stay readable. Pages already being fetched finish, are billed, and stay in the crawl: for up to 10 minutes after the stop, the links on them join the results. Keep reading with next_cursor until it is null for the final results. A stopped crawl can’t resume.

Get notified by webhook

Instead of polling, give the crawl a webhook when you start it. Zenrows then posts an event to your https URL when the crawl starts and when it ends. Your account needs a webhook signing key first: create it in the dashboard, under Settings, API Keys, and store the secret, since it is shown only once. Without the key, the request returns 422 webhook_signing_key_required. events narrows what you get: the request below asks only for the crawl’s end. Leave it out to get all four events. metadata comes back in every event, to match the event to your own records.
The crawl in the response, and on every read, shows the webhook it runs with:
When the crawl ends, your URL receives one event, crawl.completed, crawl.stopped or crawl.failed, with the crawl’s ID and its API URL in resource. The event carries no results: read the crawl at resource.url for them. Every field and header is in the endpoint reference.

Verify an event

Every event carries an X-Signature header, t=<unix seconds>,v1=<hex>,kid=<key id>. Check it before you trust the event:
  1. Pick your secret by kid. While you rotate the key, either key may sign, so keep both.
  2. Base64-decode the secret. The HMAC key is the decoded bytes, not the text.
  3. Compute the hex HMAC-SHA256 of t, a dot, and the raw request body exactly as received, before any JSON parsing.
  4. Compare it to v1 in constant time.
  5. Reject a t more than 5 minutes from your clock, so a captured event can’t be replayed.
Then answer 2xx within 10 seconds and process the event afterwards. A delivery that gets no 2xx in time is retried with backoff for up to 3 days, so the same event can arrive twice: skip an X-ZenRows-Event-Id you have already handled. Events can arrive out of order, so order them by occurred_at.

Next steps

Crawl Endpoints

Every parameter, limit, status and error code, plus pricing.

Adaptive Stealth Mode

How Fetch picks the settings for each page a crawl opens.