> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Turn a whole catalog into a structured dataset

> Turn an entire online store into a structured dataset: find every product with Zenrows Crawl, extract them with Batch, and save CSV and JSON.

Point Crawl at a store, let it find every listing page, then send those pages to Batch with `extract=auto`. You get the full catalog (names, prices, SKUs, URLs, images) as one CSV or JSON file, with no selectors to write or maintain.

**Uses:** [Crawl](https://docs.zenrows.com/crawl/introduction.md) with `include_patterns`, [Batch](https://docs.zenrows.com/batch/introduction.md) with [Extract](https://docs.zenrows.com/extract/introduction.md) (`extract=auto`)

## How it works

1. **Crawl discovers the pages.** Starting from the store's home page, Crawl follows links and keeps the URLs that match `include_patterns`, here the paginated listing pages (`/ecommerce/page/`).
2. **Batch extracts every page.** One Batch job runs `extract=auto` on every listing page in parallel. Each result is JSON with a `products` array.
3. **Your code merges the rows.** The script flattens every page's `products` into one list, removes duplicates by product URL, and writes `catalog.csv` and `catalog.json`.

Listing pages carry a full row per product, so 12 requests cover a 188-product store. To collect detail-page fields as well, point `include_patterns` at product URLs (`/product/`) and run a Batch step with a `css_extractor` for the fields you need.

## Ask your AI agent

With the [Zenrows MCP server](https://docs.zenrows.com/mcp/overview.md) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"theme":"css-variables"}
Use Zenrows to find every listing page of https://www.scrapingcourse.com/ecommerce/,
scrape them all in one batch, and combine every product into one table with
name, price, SKU and product URL. Save it as catalog.csv.
```

The same prompt works on a catalog with thousands of pages: the crawl and the batch run in the cloud, not in the conversation.

## Build the dataset

Install the [Zenrows SDK](https://docs.zenrows.com/sdk/overview.md) (`pip install zenrows` or `npm install zenrows`). The script runs the three steps end to end. `max_items=3` keeps the first run small: raise it, or remove it, to cover the whole catalog.

```python Python (SDK) theme={"theme":"css-variables"}
import csv
import json

from zenrows import ZenRowsBatchClient, ZenRowsCrawlClient

API_KEY = "YOUR_ZENROWS_API_KEY"
START_URL = "https://www.scrapingcourse.com/ecommerce/"

# 1. Crawl: discover every listing page of the catalog
crawler = ZenRowsCrawlClient(api_key=API_KEY)
crawl = crawler.create(
    START_URL,
    include_patterns=["/ecommerce/page/"],
    max_items=3,  # raise to cover the whole catalog
)
crawl = crawler.wait(crawl.crawl_id)
listing_urls = [START_URL] + [result.url for result in crawler.results(crawl.crawl_id)]
print(f"Crawl {crawl.status.value}: {len(listing_urls)} listing pages")

# 2. Batch: extract every listing page as structured JSON
batch = ZenRowsBatchClient(api_key=API_KEY)
job = batch.submit_regular(listing_urls, zenrows_params={"extract": "auto"})
run = job.run.wait(timeout=900)
print(f"Batch {run.data.status.value}: {run.stats.successful}/{run.stats.total} pages extracted")

# 3. Merge the products into one dataset, one row per product
products = {}
for task in run.results(status="successful"):
    parsed = json.loads(run.download_task_to_memory(task))["parsed"]
    for product in parsed.get("products", []):
        products[product["product_url"]] = product

rows = list(products.values())
fields = ["name", "price", "currency", "sku", "product_id", "product_url", "image_url"]

with open("catalog.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
    writer.writeheader()
    writer.writerows(rows)

with open("catalog.json", "w", encoding="utf-8") as f:
    json.dump(rows, f, indent=2, ensure_ascii=False)

print(f"Saved {len(rows)} products to catalog.csv and catalog.json")
```

```javascript Node.js theme={"theme":"css-variables"}
import { writeFile } from "node:fs/promises";
import { ZenRowsBatchClient, ZenRowsCrawlClient } from "zenrows";

const API_KEY = "YOUR_ZENROWS_API_KEY";
const START_URL = "https://www.scrapingcourse.com/ecommerce/";

// 1. Crawl: discover every listing page of the catalog
const crawler = new ZenRowsCrawlClient(API_KEY);
const crawl = await crawler.create({
  url: START_URL,
  includePatterns: ["/ecommerce/page/"],
  maxItems: 3, // raise to cover the whole catalog
});
const ended = await crawler.wait(crawl.crawl_id);
const listingUrls = [START_URL];
for await (const result of crawler.results(crawl.crawl_id)) listingUrls.push(result.url);
console.log(`Crawl ${ended.status}: ${listingUrls.length} listing pages`);

// 2. Batch: extract every listing page as structured JSON
const batch = new ZenRowsBatchClient(API_KEY);
const job = await batch.submitRegular(listingUrls, undefined, { zenrowsParams: { extract: "auto" } });
const run = await job.run.wait();
console.log(`Batch ${run.data.status}: ${run.data.stats.successful}/${run.data.stats.total} pages extracted`);

// 3. Merge the products into one dataset, one row per product
const products = new Map();
for await (const task of run.results({ status: "successful" })) {
  const { parsed } = JSON.parse(await run.downloadTaskToMemory(task));
  for (const product of parsed.products ?? []) products.set(product.product_url, product);
}

const rows = [...products.values()];
const fields = ["name", "price", "currency", "sku", "product_id", "product_url", "image_url"];
const csvCell = (value) => `"${String(value ?? "").replaceAll('"', '""')}"`;
const csv = [fields.join(","), ...rows.map((row) => fields.map((f) => csvCell(row[f])).join(","))].join("\n");

await writeFile("catalog.csv", csv);
await writeFile("catalog.json", JSON.stringify(rows, null, 2));
console.log(`Saved ${rows.length} products to catalog.csv and catalog.json`);
```

`crawler.wait` and `job.run.wait` poll until each step finishes, so the script runs top to bottom with no extra polling code.

## Build it from the terminal

The [Zenrows CLI](https://docs.zenrows.com/cli/introduction.md) runs the same pipeline with four commands and [jq](https://jqlang.org/). Install it with `npm install -g @zenrows/cli` and log in with `zenrows login`.

```bash CLI theme={"theme":"css-variables"}
# 1. Crawl the listing pages and save their URLs as JSONL
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ \
  --include-patterns /ecommerce/page/ --max-items 3 --follow --json > crawl.json
zenrows crawl results "$(jq -r .crawl_id crawl.json)" --out pages.jsonl
echo '{"url": "https://www.scrapingcourse.com/ecommerce/"}' >> pages.jsonl  # the start page has products too

# 2. Turn each URL into a Batch task with extract=auto, then run the job
jq -c '{url, extract: "auto"}' pages.jsonl > jobs.jsonl
zenrows batch create jobs.jsonl --follow --timeout 2h --json > job.json

# 3. Download every result and merge the products into CSV
zenrows batch results "$(jq -r .job_id job.json)" --download ./pages
jq -s '[.[].parsed.products[]] | unique_by(.product_url)' pages/*.json > catalog.json
jq -r '(["name","price","sku","product_url"]), (.[] | [.name, .price, .sku, .product_url]) | @csv' \
  catalog.json > catalog.csv
```

## Example output

The Python script's console output, from a run with `max_items=3`:

```text theme={"theme":"css-variables"}
Crawl completed: 4 listing pages
Batch completed: 4/4 pages extracted
Saved 64 products to catalog.csv and catalog.json
```

The first rows of `catalog.csv`:

```text theme={"theme":"css-variables"}
name,price,currency,sku,product_id,product_url,image_url
Atlas Fitness Tank,18,USD,MT11,1364,https://www.scrapingcourse.com/ecommerce/product/atlas-fitness-tank/,https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mt11-blue_main.jpg
Atomic Endurance Running Tee (Crew-Neck),29,USD,MS12,1025,https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-running-tee-crew-neck/,https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/ms12-red_main.jpg
Augusta Pullover Jacket,57,USD,WJ03,1761,https://www.scrapingcourse.com/ecommerce/product/augusta-pullover-jacket/,https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/wj03-red_main.jpg
```

Each entry in `catalog.json` keeps every field Extract returned:

```json theme={"theme":"css-variables"}
{
  "availability": null,
  "currency": "USD",
  "image_url": "https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mt11-blue_main.jpg",
  "name": "Atlas Fitness Tank",
  "price": 18,
  "product_id": "1364",
  "product_type": "variable",
  "product_url": "https://www.scrapingcourse.com/ecommerce/product/atlas-fitness-tank/",
  "sku": "MT11"
}
```

## Customize it

| Change | How |
| - | - |
| Cover the whole catalog | Remove `max_items` or raise it. `max_pages` sets how many pages the crawl fetches, see [Crawl endpoints](https://docs.zenrows.com/crawl/endpoints.md#start-a-crawl) |
| Crawl one category | Start from the category URL and set `include_patterns` to its path, for example `/product-category/clothing/men/` |
| Keep only some fields | Add `extract_fields` to `zenrows_params`, for example `"extract_fields": "products.name,products.price"`, see [Extract](https://docs.zenrows.com/extract/endpoints.md) |
| Use your own selectors | Replace `extract` with `css_extractor` in `zenrows_params`, see [CSS extractor](https://docs.zenrows.com/fetch/features/css-extractor.md) |
| Target a country's prices | Add `"mode": "auto", "proxy_country": "us"` to `zenrows_params`, see [Proxy country](https://docs.zenrows.com/fetch/features/proxy-country.md) |
| Get notified instead of waiting | Attach a webhook to the Batch job, see [Batch webhooks](https://docs.zenrows.com/batch/developer-guide-restapi.md#set-up-webhooks) |
| Refresh the dataset every day | Submit the Batch step as a scheduled job, see [Scheduled jobs](https://docs.zenrows.com/batch/developer-guide-restapi.md#scheduled-jobs) |

## Troubleshooting

* **The crawl returns no listing pages**: the pattern doesn't match the store's pagination URLs. Open page 2 in your browser, copy the part of the URL that repeats (`/page/`, `?page=`, `/p/`) and use it as `include_patterns`.
* **The dataset is smaller than the store**: `max_items` stopped the crawl early. Raise it, and check `crawl.coverage.items_found` against the store's page count.
* **A page returns `REQS004`**: the `extract_fields` you asked for aren't on that page. The error's `detail` lists the available fields: copy the ones you need, or remove `extract_fields`.
* **Some Batch tasks failed**: rerun only the failures with `job.retry_failed()` (Python), `job.retryFailed()` (Node.js) or `zenrows batch retry-failed <job-id>`. Successful results carry over.

## Related recipes

> [card] [Map every URL on a website](https://docs.zenrows.com/academy/crawl/map-a-website.md)
>
> Run Crawl on its own to list a site's pages.

> [card] [Get structured JSON from any page](https://docs.zenrows.com/academy/extract/auto-extraction.md)
>
> How `extract=auto` works on one page.

> [card] [Scrape a list of URLs](https://docs.zenrows.com/academy/scrape-from-a-list-of-urls.md)
>
> Send thousands of URLs as one Batch job.

> [card] [Build a price monitor](https://docs.zenrows.com/academy/build-price-monitoring.md)
>
> Track the prices in your dataset over time.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.