Sync vs Async Scraping: When to Use Each Mode

What are sync and async scraping modes?

UltraWebScrapingAPI offers two scraping modes: sync (synchronous) and async (asynchronous). Each is designed for different use cases.

Sync mode blocks until the scrape is complete and returns the HTML directly in the response. It’s simple but limited to one URL per request.

Async mode accepts up to 100 URLs, returns a subscription ID immediately, and processes all URLs in the background. You retrieve results by polling or via webhooks.

When to use sync mode

Use sync mode when you need a single page’s HTML immediately:

• Real-time price checking
• Single-page data extraction
• Interactive tools and dashboards
• Simple scripts that process one URL at a time

response = requests.post(
    "https://api.ultrawebscrapingapi.com/v1/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={
        "urls": [{"url": "https://example.com/product/123"}],
        "mode": "sync"
    }
)
html = response.json()["html"]
# Parse and use the HTML immediately

When to use async mode

Use async mode when you need to scrape multiple pages efficiently:

• Batch scraping product catalogs
• Monitoring multiple competitor pages
• Scheduled data collection jobs
• Any workflow with more than one URL

response = requests.post(
    "https://api.ultrawebscrapingapi.com/v1/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={
        "urls": [
            {"url": "https://site.com/page1"},
            {"url": "https://site.com/page2"},
            {"url": "https://site.com/page3"},
        ],
        "mode": "async"
    }
)
subscription_id = response.json()["subscriptionId"]

Webhooks vs polling

With async mode, you have two ways to know when scraping is done:

Polling

Simple but requires repeated API calls:

import time

while True:
    status = requests.get(
        f"https://api.ultrawebscrapingapi.com/v1/subscription/{subscription_id}",
        headers={"X-API-Key": "your_api_key"},
    ).json()

    if status["processing"] == 0 and status["queued"] == 0:
        break
    time.sleep(5)

Webhooks (recommended)

Register a webhook URL once, then get notified automatically:

# One-time setup
endpoint = requests.post(
    "https://api.ultrawebscrapingapi.com/v1/endpoint",
    headers={"X-API-Key": "your_api_key"},
    json={"url": "https://your-server.com/webhook"}
).json()

# Include endpointId in scrape requests
requests.post(
    "https://api.ultrawebscrapingapi.com/v1/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={
        "urls": urls,
        "mode": "async",
        "endpointId": endpoint["endpointId"]
    }
)

Your webhook receives progress updates and a final “completed” notification. All webhooks are signed with HMAC-SHA256 so you can verify they’re from UltraWebScrapingAPI.

Error handling differences

Sync and async modes handle errors differently, and understanding this is critical for building reliable scraping pipelines.

In sync mode, errors are returned immediately in the response. If the target site blocks the request or the page times out, you get an error status right away and can retry or handle it in your code:

response = requests.post(
    "https://api.ultrawebscrapingapi.com/v1/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={"urls": [{"url": "https://example.com"}], "mode": "sync"}
)

data = response.json()
if data.get("status") == "error":
    print(f"Failed: {data.get('error')}")
    # Retry logic here

In async mode, each URL in the batch has its own status. Some URLs may succeed while others fail. When you retrieve results, check each job’s status individually:

for job in status["jobs"]:
    if job["status"] == "completed":
        # Process successful result
        result = fetch_result(subscription_id, job["index"])
    elif job["status"] == "failed":
        # Log and optionally resubmit this URL
        print(f"Failed: {job['url']} - {job.get('error')}")

This per-URL error handling is one of async mode’s advantages — a single failed URL doesn’t block the rest of the batch.

Performance considerations

Sync mode latency depends entirely on the target site. Simple pages return in 2-5 seconds. Sites with heavy anti-bot protection like Akamai or Cloudflare Turnstile may take 10-30 seconds as our browser solves challenges.

Async mode throughput scales much better. When you submit 100 URLs, we process them concurrently across our browser pool. The total time to complete all 100 URLs is far less than 100× the single-URL time. For large-scale scraping operations — monitoring thousands of product prices, tracking airline fares across routes — async mode is the only practical option.

Cost implications

Both modes cost the same per URL — there’s no premium for sync mode. However, async mode can be more cost-effective in practice because:

• Batch submission reduces API call overhead
• Webhook delivery eliminates polling costs (each poll is a free API call, but polling adds latency and complexity)
• Failed URLs in a batch can be resubmitted without reprocessing successful ones

Summary

Feature	Sync	Async
URLs per request	1	Up to 100
Response	HTML directly	Subscription ID
Best for	Single page, real-time	Batch scraping
Webhooks	N/A	Supported
Error handling	Immediate	Per-URL status

For most production use cases, async mode with webhooks gives you the best combination of throughput and simplicity. Use sync mode for quick, interactive scraping needs. Try sync mode instantly in the playground.

Read the full API documentation for complete details on both modes, or check the getting started guide to set up your first scraping pipeline. View our pricing to find the right plan for your volume. If you’re scraping anti-bot protected sites, see how we handle DataDome and PerimeterX.