X-API-Key header. Parameters, limits and errors are in the endpoint reference.
Start a crawl
Send the start URL toPOST /v1/crawls. Add depth to limit how many link steps it follows. Without it, the crawl follows links until max_items or max_pages stops it. Add max_items to keep more than the default 100 URLs. include_patterns keeps a URL only if it contains one of the strings you pass, and Crawl opens only those URLs. A homepage crawl with only /product/ doesn’t open listing paths that lack that text, such as /page/.
202 Accepted. The crawl runs in the background:
Response
crawl_id. Every other call takes it.
Read the status and the results
GET /v1/crawls/{crawl_id} returns the crawl’s status, its coverage so far and one page of results. Results grow while the crawl runs, so read them as you go:
- Read without
cursorto get the first results. - Send the
next_cursoryou received ascursorto get the URLs found since. - Stop when
next_cursorisnull. This happens only once the crawl has ended and you have read its last results.
Response
stop_reason: max_items means the site may hold more URLs. To get them, start a new crawl with a higher max_items (and a higher max_pages if stop_reason is max_pages).
Download all results
GET /v1/crawls/{crawl_id}/download returns every URL in one NDJSON file: one JSON object per line.
cURL
Response
X-Crawl-Status header tells you the crawl’s status: running means a later download may hold more lines. So can a download in the 10 minutes after a stop, while the header reads stopped.
Get each page
Send"output_format" to get each page the crawl keeps, not only its URL: "html" for the page as fetched, "markdown" for it converted to Markdown. Each kept page counts toward max_pages, so set max_pages to at least max_items plus the pages the crawl walks:
cURL
content_status, and a content_url once the page is fetched:
Response
content_status is pending until the page is fetched, then fetched or failed. After a stop, a page not fetched yet stays pending. Read a fetched page at its content_url:
cURL
content field.
Stop a crawl
POST /v1/crawls/{crawl_id}/stop stops a running crawl:
cURL
status: stopped with stop_reason: user. The URLs found before the stop stay readable. Pages already being fetched finish, are billed, and stay in the crawl: for up to 10 minutes after the stop, the links on them join the results. Keep reading with next_cursor until it is null for the final results. A stopped crawl can’t resume.
Get notified by webhook
Instead of polling, give the crawl awebhook when you start it. Zenrows then posts an event to your https URL when the crawl starts and when it ends. Your account needs a webhook signing key first: create it in the dashboard, under Settings, API Keys, and store the secret, since it is shown only once. Without the key, the request returns 422 webhook_signing_key_required.
events narrows what you get: the request below asks only for the crawl’s end. Leave it out to get all four events. metadata comes back in every event, to match the event to your own records.
crawl.completed, crawl.stopped or crawl.failed, with the crawl’s ID and its API URL in resource. The event carries no results: read the crawl at resource.url for them. Every field and header is in the endpoint reference.
Verify an event
Every event carries anX-Signature header, t=<unix seconds>,v1=<hex>,kid=<key id>. Check it before you trust the event:
- Pick your secret by
kid. While you rotate the key, either key may sign, so keep both. - Base64-decode the secret. The HMAC key is the decoded bytes, not the text.
- Compute the hex HMAC-SHA256 of
t, a dot, and the raw request body exactly as received, before any JSON parsing. - Compare it to
v1in constant time. - Reject a
tmore than 5 minutes from your clock, so a captured event can’t be replayed.
2xx within 10 seconds and process the event afterwards. A delivery that gets no 2xx in time is retried with backoff for up to 3 days, so the same event can arrive twice: skip an X-ZenRows-Event-Id you have already handled. Events can arrive out of order, so order them by occurred_at.
Next steps
Crawl Endpoints
Every parameter, limit, status and error code, plus pricing.
Adaptive Stealth Mode
How Fetch picks the settings for each page a crawl opens.