> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Introduction to Crawl

> Give Zenrows Crawl one start URL, such as a homepage, a category or a search page, and get back the pages it finds: their URLs, and each page as HTML or Markdown if you want it.

> [!NOTE]
>
> Crawl is New. Any active Zenrows API key can use it, with no extra setup. Crawl is still evolving: more features are on the way, limits may be tuned, and every change is announced in the [changelog](https://www.zenrows.com/changelog).

Zenrows Crawl takes one start URL and returns the pages it finds from there. Point it at a homepage, a category page or a search results page, and Crawl follows the links for you and removes duplicates. You choose what you get back for each page: only its URL, or the page as HTML or Markdown. You don't write a link loop, a filter or a deduplication step.

A crawl is a background job. You start it with one request, then read its status and its results while it runs.

## How it works

1. Crawl opens the start URL with Fetch (formerly, Universal Scraper API) and [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode.md) (`mode=auto`).
2. It reads every link in the page's HTML. It keeps the links on the same domain (subdomains included) that match your filters, and drops duplicates: `http` and `https` links to one page count as one. It skips a link longer than 2,048 characters, a link that would change something on the site, such as log-out, delete or add-to-cart, and a link to a host that is not public.
3. It opens the kept links and reads their links too, up to `depth` when you set one. One step from a page to its links is one level of `depth`.
4. It stops when nothing is left to open, or when it reaches `max_items` or `max_pages`.

`include_patterns` chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only `/product/` opens the product links on that page and skips listing and pagination URLs that don't contain that text, such as `/page/`, so it returns almost nothing from the rest of the site. To walk those pages, include their path as well, and raise `depth` if you set one. Crawl returns the URLs that match.

## Basic usage

Start a crawl with a `POST` to `/v1/crawls`:

```bash cURL theme={"theme":"css-variables"}
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "include_patterns": ["/product/"],
    "max_pages": 10
  }'
```

The request above sets no `depth`, so Crawl has no depth limit. It opens the product pages linked from the start page, then the product links on those pages, until it has fetched 10 pages (`max_pages`). It returns the product URLs it found on those pages, up to `max_items` (100 by default), so a run can return more URLs than it fetched pages. With only `/product/`, it doesn't open `/page/` or other listing URLs.

The response is `202 Accepted` with a `crawl_id`. Read the crawl with `GET /v1/crawls/{crawl_id}`, sending each response's `next_cursor` as `cursor` on the next read, until `next_cursor` is `null`: then it has ended and you have every result.

You can also try Crawl without code in the Crawl playground.

[Open the Crawl playground](https://app.zenrows.com/crawl)

## Next steps

> [card] [Crawl Setup](https://docs.zenrows.com/crawl/setup.md)
>
> Start a crawl, read its results, download them, and get each page as HTML or Markdown.

> [card] [Crawl Endpoints](https://docs.zenrows.com/crawl/endpoints.md)
>
> Every endpoint, parameter, limit and error code, plus pricing.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.