> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Run many requests concurrently

> Scrape many URLs at once with Zenrows Fetch, using all of your plan's capacity, or let the Zenrows SDKs run them in parallel for you.

Sending Fetch requests one at a time leaves most of your plan idle. Run them in parallel, with as many in flight as your plan's concurrency allows, and a list of URLs finishes in a fraction of the time. Each response tells you that number in its `Concurrency-Limit` header.

**Uses:** [Fetch](https://docs.zenrows.com/fetch/api-reference.md) with [concurrency headers](https://docs.zenrows.com/fetch/features/concurrency.md#concurrency-headers), the [Python SDK](https://docs.zenrows.com/fetch/sdk/python/async-concurrency.md) and the [Node.js SDK](https://docs.zenrows.com/fetch/sdk/nodejs/async-concurrency.md)

## Ask your AI agent

With the [Zenrows MCP server](https://docs.zenrows.com/mcp/overview.md) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"theme":"css-variables"}
Use Zenrows to scrape all of these pages at once and give me each product's name and price:
https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/
https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/
https://www.scrapingcourse.com/ecommerce/product/aeon-capri/
https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee/
```

Your agent sends the list as one Batch job, so the same prompt scales from four URLs to thousands.

## Size a worker pool to your plan

Make one request, read `Concurrency-Limit`, then run that many workers over the rest of the list. Every worker reuses the same pattern: send, check, parse.

```python Python (Requests) theme={"theme":"css-variables"}
import re
from concurrent.futures import ThreadPoolExecutor, as_completed

import requests

API_KEY = "YOUR_ZENROWS_API_KEY"
SLUGS = [
    "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee",
    "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt",
]
URLS = [f"https://www.scrapingcourse.com/ecommerce/product/{slug}/" for slug in SLUGS]

session = requests.Session()

def scrape(url):
    params = {"apikey": API_KEY, "url": url, "mode": "auto"}
    response = session.get("https://api.zenrows.com/v1/", params=params, timeout=180)
    response.raise_for_status()
    title = re.search(r"<h1[^>]*>(.*?)</h1>", response.text, re.S).group(1).strip()
    return title, response.headers

# Read your plan's limit from the first response, then size the pool to it.
first_title, headers = scrape(URLS[0])
limit = int(headers["Concurrency-Limit"])
print(f"Concurrency-Limit: {limit}")
print(first_title)

with ThreadPoolExecutor(max_workers=limit) as pool:
    futures = {pool.submit(scrape, url): url for url in URLS[1:]}
    for future in as_completed(futures):
        try:
            title, _ = future.result()
            print(title)
        except Exception as error:  # HTTP errors, timeouts, pages without an <h1>
            print(f"{futures[future]} failed: {error}")
```

```javascript Node.js theme={"theme":"css-variables"}
const API_KEY = "YOUR_ZENROWS_API_KEY";
const slugs = [
  "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee",
  "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt",
];
const urls = slugs.map((slug) => `https://www.scrapingcourse.com/ecommerce/product/${slug}/`);

async function scrape(url) {
  const params = new URLSearchParams({ apikey: API_KEY, url, mode: "auto" });
  const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
    signal: AbortSignal.timeout(180_000),
  });
  if (!response.ok) throw new Error(`${url} failed: ${response.status}`);
  const html = await response.text();
  const title = html.match(/<h1[^>]*>([\s\S]*?)<\/h1>/)[1].trim();
  return { title, limit: Number(response.headers.get("Concurrency-Limit")) };
}

// Read your plan's limit from the first response, then run that many workers.
const first = await scrape(urls[0]);
console.log(`Concurrency-Limit: ${first.limit}`);
console.log(first.title);

const queue = urls.slice(1);
async function worker() {
  while (queue.length > 0) {
    const url = queue.shift();
    try {
      const { title } = await scrape(url);
      console.log(title);
    } catch (error) {
      console.error(error.message);
    }
  }
}
await Promise.all(Array.from({ length: first.limit }, worker));
```

## Example output

Results print in the order they finish, not the order of the list:

```text theme={"theme":"css-variables"}
Concurrency-Limit: 5
Abominable Hoodie
Aeon Capri
Affirm Water Bottle
Aero Daily Fitness Tee
Adrienne Trek Jacket
Aether Gym Pant
Aim Analog Watch
Ajax Full-Zip Sweatshirt
```

## Let the SDK queue requests for you

The Zenrows SDKs keep an internal queue: fire every request at once and the client keeps `concurrency` of them in flight. `retries` re-sends transient failures. Install with `pip install zenrows` or `npm install zenrows`.

```python Python (SDK) theme={"theme":"css-variables"}
import asyncio
import re

from zenrows import ZenRowsClient

SLUGS = [
    "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee",
    "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt",
]
URLS = [f"https://www.scrapingcourse.com/ecommerce/product/{slug}/" for slug in SLUGS]

# concurrency sets the requests in flight; retries re-sends transient failures.
client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1)

async def main():
    responses = await asyncio.gather(
        *[client.fetch_async(url, params={"mode": "auto"}, timeout=180) for url in URLS]
    )
    for url, response in zip(URLS, responses):
        match = re.search(r"<h1[^>]*>(.*?)</h1>", response.text, re.S)
        print(response.status_code, match.group(1).strip() if match else url)

asyncio.run(main())
```

```javascript Node.js (SDK) theme={"theme":"css-variables"}
import { ZenRows } from "zenrows";

const slugs = [
  "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee",
  "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt",
];
const urls = slugs.map((slug) => `https://www.scrapingcourse.com/ecommerce/product/${slug}/`);

// concurrency sets the requests in flight; retries re-sends transient failures.
const client = new ZenRows("YOUR_ZENROWS_API_KEY", { concurrency: 5, retries: 1 });

const results = await Promise.allSettled(urls.map((url) => client.fetch(url, { mode: "auto" })));

for (const [i, result] of results.entries()) {
  if (result.status === "rejected") {
    console.error(urls[i], result.reason);
    continue;
  }
  const html = await result.value.text();
  const title = html.match(/<h1[^>]*>([\s\S]*?)<\/h1>/)?.[1].trim();
  console.log(result.value.status, title);
}
```

Output, in list order:

```text theme={"theme":"css-variables"}
200 Abominable Hoodie
200 Adrienne Trek Jacket
200 Aeon Capri
200 Aero Daily Fitness Tee
200 Aether Gym Pant
200 Affirm Water Bottle
200 Aim Analog Watch
200 Ajax Full-Zip Sweatshirt
```

Set `concurrency` to your plan's `Concurrency-Limit`. Use one client for the whole run, so every request shares the same queue.

## Send very large lists to Batch

For thousands of URLs, or jobs you want to run without keeping a process alive, send the list to [Batch](https://docs.zenrows.com/batch/introduction.md). Zenrows queues the URLs, runs them at your plan's concurrency, retries and stores every result for you to download. See [Scrape a list of URLs](https://docs.zenrows.com/academy/scrape-from-a-list-of-urls.md).

## Customize it

| Change | How |
| - | - |
| Back off and retry failures | See [Retry failed requests](https://docs.zenrows.com/academy/retry-failed-requests.md) |
| Watch free slots live | Read `Concurrency-Remaining` on each response, see [Concurrency headers](https://docs.zenrows.com/fetch/features/concurrency.md#concurrency-headers) |
| Different parameters per URL | Build `params` inside `scrape()` from the URL or a per-URL dict |
| Thousands of URLs, results stored for you | Use [Batch](https://docs.zenrows.com/academy/scrape-from-a-list-of-urls.md) |
| Many parallel real browsers | See [Run browser sessions in parallel](https://docs.zenrows.com/academy/browser/parallel-sessions.md) |

## Troubleshooting

* **`429` with code `AUTH006`**: more requests are in flight than your plan's concurrency. Lower the pool size to `Concurrency-Limit`, and keep one pool or one SDK client for the whole run.
* **`429` right after a burst of cancelled requests**: a request cancelled on your side keeps its slot until Zenrows finishes it. Keep client timeouts at 180 seconds or more so you don't cancel work in progress.
* **Throughput is lower than expected**: slow pages hold their slot longer. Use the cheapest configuration that works, for example `mode=auto` instead of a fixed `js_render`, so each request returns sooner.
* **One failure stops the whole run**: an exception escaped a worker. Catch errors per URL, as the samples do, and collect failures for a retry pass.

## Related recipes

> [card] [Scrape a list of URLs](https://docs.zenrows.com/academy/scrape-from-a-list-of-urls.md)
>
> Send thousands of URLs as one Batch job.

> [card] [Retry failed requests](https://docs.zenrows.com/academy/retry-failed-requests.md)
>
> Retry 429s and 5xx errors with backoff.

> [card] [Lesson 4: Scrape every product with Batch](https://docs.zenrows.com/academy/foundations/scrape-at-scale.md)
>
> The Foundations lesson on moving from one URL to many.

> [card] [Track usage and control spend](https://docs.zenrows.com/academy/production/control-costs.md)
>
> Keep the per-request cost low on large runs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.