Basic Troubleshooting Guide

This guide helps you resolve common web scraping issues by organizing solutions around specific use cases. Whether you’re dealing with dynamic content, geo-restrictions, or session management, these targeted troubleshooting techniques will get you back on track quickly.

Dynamic Content & Timing Issues

When scraping JavaScript-heavy sites, SPAs, or pages with delayed content loading, timing becomes critical.

Multiple Wait-For Selectors

The wait_for parameter handles dynamic content, but edge cases like empty results require careful selector planning:

const response = await axios.get('https://api.zenrows.com/v1/', {
  params: {
    apikey: YOUR_ZENROWS_API_KEY,
    url: TARGET_URL,
    js_render: true,
    // Wait for either product listings OR a "no results" message
    wait_for: '.product-listing, .no-results-message'
  }
});

response = requests.get(
    'https://api.zenrows.com/v1/',
    params={
        'apikey': YOUR_ZENROWS_API_KEY,
        'url': TARGET_URL,
        'js_render': True,
        # Wait for either product listings OR a "no results" message
        'wait_for': '.product-listing, .no-results-message'
    }
)

This ensures your request completes when the expected content appears OR when a “no results” indicator is displayed, avoiding unnecessary timeouts.

Testing CSS Selectors Before Scraping

Before using CSS selectors in your scraping code, test them directly in the browser console:

Open the target page in your browser
Open Developer Tools (F12 or ⌘+⌥+I)
In the Console tab, test your selector:

document.querySelectorAll('.your-selector')

Verify it returns the expected elements across different page states

Anti-Bot Protection & Access Issues

When sites block your requests or return unexpected errors, these debugging techniques help identify the root cause.

Error 422: Unprocessable Entity

This typically means anti-bot protection is blocking access. Solutions include:

Enable JavaScript rendering: js_render: true to mimic real browser behavior
Use premium proxies: Set premium_proxy: true to route traffic through residential IPs
Add a referer header: Set custom_headers=true and send Referer: https://www.google.com/ to mimic typical browser behavior
Wait for content to load: Use wait_for or wait parameters for slower sites

If adding parameters makes it worse, see Still failing after stacking parameters.

If your success rate suddenly drops, it’s often due to new anti-bot challenges or region-specific blocks. Enabling js_render, switching proxy regions, or using premium_proxy: true typically resolves this.

Still failing after stacking parameters? Retest with the minimal config

Extra parameters change how the request behaves, and on some protected sites they can lower your success rate instead of raising it. The referer value is a header, so it needs custom_headers=true. If you still get 422 after adding wait, wait_for, or a custom referer header on top of js_render=true and premium_proxy=true, follow these steps:

Remove the extra parameters and retest with only js_render=true and premium_proxy=true.
Add the extras back one at a time, and keep only the ones that help.
Keep proxy_country if the site is region-restricted. See Error 422 with Geo-Restrictions.

The minimal request looks like this:

const response = await axios.get('https://api.zenrows.com/v1/', {
  params: {
    apikey: YOUR_ZENROWS_API_KEY,
    url: TARGET_URL,
    js_render: true,
    premium_proxy: true
    // No wait, wait_for, or custom headers yet
  }
});

response = requests.get(
    'https://api.zenrows.com/v1/',
    params={
        'apikey': YOUR_ZENROWS_API_KEY,
        'url': TARGET_URL,
        'js_render': True,
        'premium_proxy': True
        # No wait, wait_for, or custom headers yet
    }
)

If the minimal request still returns 422, contact support with the X-Request-Id response header so the team can check what the target site is doing.

Retrieving Original Status Codes

When a target website returns an error, Zenrows handles it gracefully by default. Use original_status to see what’s actually happening:

const response = await axios.get('https://api.zenrows.com/v1/', {
  params: {
    apikey: YOUR_ZENROWS_API_KEY,
    url: TARGET_URL,
    original_status: true  // Return original status code
  }
});
console.log('Original status code:', response.status);

response = requests.get(
    'https://api.zenrows.com/v1/',
    params={
        'apikey': YOUR_ZENROWS_API_KEY,
        'url': TARGET_URL,
        'original_status': True  # Return original status code
    }
)

print(f'Original status code: {response.status_code}')

This helps you identify if the target site is returning errors like 403 Forbidden or 429 Too Many Requests.

Allowing Specific Status Codes

Examine error pages or handle specific status codes by allowing them to pass as successful responses:

const response = await axios.get('https://api.zenrows.com/v1/', {
  params: {
    apikey: YOUR_ZENROWS_API_KEY,
    url: TARGET_URL,
    // Allow 404 and 500 responses to be treated as success
    allowed_status_codes: '404,500'
  }
});

response = requests.get(
    'https://api.zenrows.com/v1/',
    params={
        'apikey': YOUR_ZENROWS_API_KEY,
        'url': TARGET_URL,
        # Allow 404 and 500 responses to be treated as success
        'allowed_status_codes': '404,500'
    }
)

Region-Restricted Sites

Geographic restrictions require specific approaches to access localized content.

Error 422 with Geo-Restrictions

When content is region-locked, adjust your geographic approach:

Specify target country: Use proxy_country: 'us' or another accessible region
Enable premium proxies: Set premium_proxy: true for residential IPs
Test different regions: Some content may be available in multiple countries

Performance & Resource Management

Optimize your scraping efficiency and manage API limits effectively.

Error 429: Too Many Requests

This indicates you’ve reached your concurrency limit. Solutions include:

Request queue implementation: Space out requests using a queue
Concurrency monitoring: Check the concurrency-remaining response header
Backoff strategy: Increase delays when 429 errors occur
Complete requests fully: Don’t cancel requests - let them finish to release concurrency slots

If you begin to notice a lower-than-usual success rate, monitor your concurrency usage and ensure requests are being spaced out appropriately. Sudden spikes can trigger errors due to the concurrency limit being reached.

Error 413: Content Too Large

When responses exceed your plan’s size limits:

Use CSS selectors: Extract only needed data with css_extractor
Disable JSON response: Remove json_response if not needed to reduce payload size
Implement pagination: Split large data sets into multiple requests

Session-Dependent Sites

For multi-step workflows requiring consistent session state.

Session management issues

When session state needs to persist across multiple requests:

Use consistent session IDs: Apply the same session_id across related requests
Maintain custom headers: Include necessary cookies and authentication headers
Test session flow: Verify each step maintains the required state

Getting Support

If you’re still encountering issues after applying these targeted solutions, Zenrows’ support team is ready to help. Prepare Request Details:

Target URL and all parameters used
Request ID from the X-Request-Id response header
Expected vs. actual results

Contact Support: Use live chat on the Zenrows website or email.

Get Started

Fetch

Extract

Batch

Browser Sessions

Residential Proxies (Legacy)

Guides

Basic Troubleshooting Guide

Dynamic Content & Timing Issues

Multiple Wait-For Selectors

Testing CSS Selectors Before Scraping

Anti-Bot Protection & Access Issues

Error 422: Unprocessable Entity

Still failing after stacking parameters? Retest with the minimal config

Retrieving Original Status Codes

Allowing Specific Status Codes

Region-Restricted Sites

Error 422 with Geo-Restrictions

Performance & Resource Management

Error 429: Too Many Requests

Error 413: Content Too Large

Session-Dependent Sites

Session management issues

Getting Support

​Dynamic Content & Timing Issues

​Multiple Wait-For Selectors

​Testing CSS Selectors Before Scraping

​Anti-Bot Protection & Access Issues

​Error 422: Unprocessable Entity

​Still failing after stacking parameters? Retest with the minimal config

​Retrieving Original Status Codes

​Allowing Specific Status Codes

​Region-Restricted Sites

​Error 422 with Geo-Restrictions

​Performance & Resource Management

​Error 429: Too Many Requests

​Error 413: Content Too Large

​Session-Dependent Sites

​Session management issues

​Getting Support

Dynamic Content & Timing Issues

Multiple Wait-For Selectors

Testing CSS Selectors Before Scraping

Anti-Bot Protection & Access Issues

Error 422: Unprocessable Entity

Still failing after stacking parameters? Retest with the minimal config

Retrieving Original Status Codes

Allowing Specific Status Codes

Region-Restricted Sites

Error 422 with Geo-Restrictions

Performance & Resource Management

Error 429: Too Many Requests

Error 413: Content Too Large

Session-Dependent Sites

Session management issues

Getting Support