Sync vs Async Scraping: When to Use Each Mode

What are sync and async scraping modes?

UltraWebScrapingAPI offers two scraping modes: sync (synchronous) and async (asynchronous). Each is designed for different use cases.

Sync mode blocks until the scrape is complete and returns the HTML directly in the response. It’s simple but limited to one URL per request.

Async mode accepts up to 100 URLs, returns a subscription ID immediately, and processes all URLs in the background. You retrieve results by polling or via webhooks.

When to use sync mode

Use sync mode when you need a single page’s HTML immediately:

• Real-time price checking
• Single-page data extraction
• Interactive tools and dashboards
• Simple scripts that process one URL at a time

response = requests.post(
    "https://api.ultrawebscrapingapi.com/api/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={
        "urls": [{"url": "https://example.com/product/123"}],
        "mode": "sync"
    }
)
html = response.json()["html"]
# Parse and use the HTML immediately

When to use async mode

Use async mode when you need to scrape multiple pages efficiently:

• Batch scraping product catalogs
• Monitoring multiple competitor pages
• Scheduled data collection jobs
• Any workflow with more than one URL

response = requests.post(
    "https://api.ultrawebscrapingapi.com/api/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={
        "urls": [
            {"url": "https://site.com/page1"},
            {"url": "https://site.com/page2"},
            {"url": "https://site.com/page3"},
        ],
        "mode": "async"
    }
)
subscription_id = response.json()["subscriptionId"]

Webhooks vs polling

With async mode, you have two ways to know when scraping is done:

Polling

Simple but requires repeated API calls:

import time

while True:
    status = requests.get(
        f"https://api.ultrawebscrapingapi.com/api/subscription/{subscription_id}",
        headers={"X-API-Key": "your_api_key"},
    ).json()

    if status["processing"] == 0 and status["queued"] == 0:
        break
    time.sleep(5)

Webhooks (recommended)

Register a webhook URL once, then get notified automatically:

# One-time setup
endpoint = requests.post(
    "https://api.ultrawebscrapingapi.com/api/endpoint",
    headers={"X-API-Key": "your_api_key"},
    json={"url": "https://your-server.com/webhook"}
).json()

# Include endpointId in scrape requests
requests.post(
    "https://api.ultrawebscrapingapi.com/api/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={
        "urls": urls,
        "mode": "async",
        "endpointId": endpoint["endpointId"]
    }
)

Your webhook receives progress updates and a final “completed” notification. All webhooks are signed with HMAC-SHA256 so you can verify they’re from UltraWebScrapingAPI.

Summary

Feature	Sync	Async
URLs per request	1	Up to 100
Response	HTML directly	Subscription ID
Best for	Single page, real-time	Batch scraping
Webhooks	N/A	Supported

For most production use cases, async mode with webhooks gives you the best combination of throughput and simplicity. Use sync mode for quick, interactive scraping needs.

Read the full API documentation for complete details on both modes.