> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Capture the network requests behind a page

> Find the hidden API calls a page makes in the background and get their data as clean JSON with Zenrows.

Many pages load their data through background XHR and fetch calls, and those calls often return clean JSON. Capture them and you get structured data without parsing any HTML. Fetch returns every call in one response; the Zenrows Browser lets you listen to them live from Playwright or Puppeteer.

**Uses:** [Fetch](https://docs.zenrows.com/fetch/api-reference.md) with [`json_response`](https://docs.zenrows.com/fetch/features/json-response), [`js_render`](https://docs.zenrows.com/fetch/features/js-rendering) and [`wait_for`](https://docs.zenrows.com/fetch/features/wait-for); [the Zenrows Browser](https://docs.zenrows.com/browser-sessions/introduction.md) with [Playwright](https://docs.zenrows.com/browser-sessions/get-started/playwright.md) or [Puppeteer](https://docs.zenrows.com/browser-sessions/get-started/puppeteer.md)

## Ask your AI agent

With the [Zenrows MCP server](https://docs.zenrows.com/mcp/overview.md) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"theme":"css-variables"}
Use the Zenrows browser to open https://www.scrapingcourse.com/javascript-rendering, wait for the
products to load, and list the background data requests the page made. Close the session.
Then use Zenrows to read the request that returns the products and give me the name,
SKU and price of the first three.
```

## Find the request that carries the data

Before you capture anything, find the call in your own browser:

1. Open `https://www.scrapingcourse.com/javascript-rendering` in Chrome.
2. Right-click the page, choose **Inspect**, open the **Network** tab and select the **Fetch/XHR** filter.
3. Reload the page. A `GET` to `https://www.scrapingcourse.com/ajax/products/json` appears.
4. Click it and open **Response**: it is a JSON array with every product's name, SKU, price and images.

The URL fragment you spotted, here `/ajax/products`, is what you filter on in the code below. Calls triggered by an action, such as a "Load more" click, show up when you perform that action with DevTools open.

## Capture requests with Fetch

Add `json_response=true` to a rendered request. The response becomes JSON with the page `html` and an `xhr` list: one entry per background call, with its `url`, `method`, `status_code`, `headers` and `body`. `wait_for` holds the capture until the data has arrived.

```python Python (Requests) theme={"theme":"css-variables"}
import json
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/javascript-rendering",
    "js_render": "true",
    "wait_for": ".product-name",
    "json_response": "true",
}

response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

data = response.json()
for call in data["xhr"]:
    print(call["method"], call["status_code"], call["url"])

# Keep the call that returns the product list and parse its body
products_call = next(call for call in data["xhr"] if "/ajax/products" in call["url"])
products = json.loads(products_call["body"])
print(len(products), "products")
for product in products[:3]:
    print(product["sku"], product["name"], product["regular_price"])
```

```javascript Node.js theme={"theme":"css-variables"}
const params = new URLSearchParams({
  apikey: "YOUR_ZENROWS_API_KEY",
  url: "https://www.scrapingcourse.com/javascript-rendering",
  js_render: "true",
  wait_for: ".product-name",
  json_response: "true",
});

const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
  signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);

const data = await response.json();
for (const call of data.xhr) console.log(call.method, call.status_code, call.url);

// Keep the call that returns the product list and parse its body
const productsCall = data.xhr.find((call) => call.url.includes("/ajax/products"));
const products = JSON.parse(productsCall.body);
console.log(products.length, "products");
for (const p of products.slice(0, 3)) console.log(p.sku, p.name, p.regular_price);
```

```bash cURL theme={"theme":"css-variables"}
curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fjavascript-rendering&js_render=true&wait_for=.product-name&json_response=true"
```

## Example output

```text theme={"theme":"css-variables"}
GET 200 https://www.scrapingcourse.com/ajax/products/json
POST 204 https://www.scrapingcourse.com/cdn-cgi/rum?
12 products
MH01 Chaz Kangeroo Hoodie 52
MH02 Teton Pullover Hoodie 70
MH03 Bruno Compete Hoodie 63
```

Each `body` is a string: parse it with `json.loads` or `JSON.parse` when the call returns JSON.

## Call the endpoint directly

Once you know the endpoint, you can often request it straight away. It skips rendering, so it is faster and cheaper than capturing it from the page.

```python Python (Requests) theme={"theme":"css-variables"}
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/ajax/products/json",
    "mode": "auto",
}

response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

products = response.json()
print(len(products), "products,", response.headers["X-Request-Credits"], "credit")
```

Output: `12 products, 1 credit`. When the endpoint needs the page's cookies or tokens, keep capturing it from the page instead.

## Capture requests with the Zenrows Browser

For multi-step flows, or when you want each response as it happens, connect Playwright or Puppeteer to the Zenrows Browser and register a `response` listener before you navigate. Filter on the resource type and read the body with `.json()`.

```python Python (Playwright) theme={"theme":"css-variables"}
import asyncio

from playwright.async_api import async_playwright

CONNECTION_URL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

async def main():
    captured = []

    async def on_response(response):
        if response.request.resource_type in ("xhr", "fetch"):
            captured.append(response)

    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(CONNECTION_URL)
        try:
            page = await browser.new_page()
            page.on("response", on_response)
            await page.goto("https://www.scrapingcourse.com/javascript-rendering")
            await page.wait_for_selector(".product-name")

            for response in captured:
                print(response.request.method, response.status, response.url[:80])

            products_response = next(r for r in captured if "/ajax/products" in r.url)
            products = await products_response.json()
            print(len(products), "products")
            for product in products[:3]:
                print(product["sku"], product["name"], product["regular_price"])
        finally:
            await browser.close()

asyncio.run(main())
```

```javascript Node.js (Puppeteer) theme={"theme":"css-variables"}
import puppeteer from "puppeteer-core";

const connectionURL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY";

const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
try {
  const page = await browser.newPage();

  const captured = [];
  page.on("response", (response) => {
    const type = response.request().resourceType();
    if (type === "xhr" || type === "fetch") captured.push(response);
  });

  await page.goto("https://www.scrapingcourse.com/javascript-rendering");
  await page.waitForSelector(".product-name");

  for (const response of captured) {
    console.log(response.request().method(), response.status(), response.url().slice(0, 80));
  }

  const productsResponse = captured.find((r) => r.url().includes("/ajax/products"));
  const products = await productsResponse.json();
  console.log(products.length, "products");
  for (const p of products.slice(0, 3)) console.log(p.sku, p.name, p.regular_price);
} finally {
  await browser.close();
}
```

Install the client first: `pip install playwright` or `npm install puppeteer-core`. Output:

```text theme={"theme":"css-variables"}
GET 200 https://www.scrapingcourse.com/ajax/products/json
POST 204 https://www.google-analytics.com/g/collect?v=2&tid=G-NZGD14H87G&gtm=45je6ad1v918
POST 204 https://www.scrapingcourse.com/cdn-cgi/rum?
12 products
MH01 Chaz Kangeroo Hoodie 52
MH02 Teton Pullover Hoodie 70
MH03 Bruno Compete Hoodie 63
```

The `finally` block closes the browser even when a step fails, which ends the session and frees its slot.

## Customize it

| Change | How |
| - | - |
| Capture calls triggered by a click or scroll | Add `js_instructions` to the Fetch request, see [Click, fill, and scroll before you scrape](https://docs.zenrows.com/academy/fetch/page-interactions.md) |
| Protected site | Add `premium_proxy=true` to the Fetch request, see [Scrape pages protected by anti-bot systems](https://docs.zenrows.com/academy/fetch/protected-sites.md) |
| The Zenrows Browser from a specific country | Add `proxy_country` to the connection URL, see [Country](https://docs.zenrows.com/browser-sessions/features/country.md) |
| Send the page's cookies or tokens to the endpoint | Use `custom_headers=true`, see [Keep the same IP and submit forms](https://docs.zenrows.com/academy/fetch/sessions-and-forms.md) |
| Wait for the network instead of an element | In Fetch, `js_instructions` with `{"wait_event": "networkidle"}` |

## Troubleshooting

* **The `xhr` list is empty or missing the call**: the capture finished before the call ran. Add `wait_for` with a selector of the content the call fills, or a `wait` of a few seconds.
* **`400` with code `REQS004` mentioning `json_response`**: the request has no rendering. Add `js_render=true`.
* **`json.loads` fails on a `body`**: that call returned HTML, text or an empty body (status `204`). Filter by URL first, and check `status_code` before parsing.
* **The Zenrows Browser listener sees nothing**: it was registered after navigation. Call `page.on("response", ...)` before `page.goto`.
* **`await response.json()` fails in the Zenrows Browser**: the response was a redirect or had no body. Check `response.status` and read the body of `200` responses.

## Related recipes

> [card] [Scrape JavaScript-rendered pages](https://docs.zenrows.com/academy/fetch/javascript-pages.md)
>
> Render the page and wait for the content you need.

> [card] [Connect Playwright or Puppeteer to the Zenrows Browser](https://docs.zenrows.com/academy/browser/connect-playwright-and-puppeteer.md)
>
> Drive the Zenrows Browser from your existing scripts.

> [card] [Click, fill, and scroll before you scrape](https://docs.zenrows.com/academy/fetch/page-interactions.md)
>
> Trigger the calls you want to capture.

> [card] [Get structured JSON from any page](https://docs.zenrows.com/academy/extract/auto-extraction.md)
>
> Get product, listing or article fields as JSON.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.