SDK to access Zenrows APIs directly from Python. Zenrows handles proxies rotation, headless browsers, and CAPTCHAs for you.
This package ships three clients:
ZenRowsClient— the original synchronous scraping client. One URL in, one HTML/JSON response out. Best for ad-hoc scraping.ZenRowsBatchClient— the async-job / Batch API client. Submit thousands of URLs as one job, poll for results, optionally upload a CSV of URLs in one call. Best for offline / bulk pipelines.ZenRowsCrawlClient— the Crawl API client. Give it one start URL; it follows links and returns the URLs (and, optionally, the pages) it finds behind it. (New.)
- Installation
- Quickstart — synchronous scraping (
ZenRowsClient) - Quickstart — Batch API (
ZenRowsBatchClient) - Quickstart — Crawl API (new) (
ZenRowsCrawlClient) - Contributing
- License
pip install zenrowsStart using the API by creating your API Key.
The SDK uses requests for HTTP requests. The client's response will be a requests Response.
It also uses Retry to automatically retry failed requests (status codes 429, 500, 502, 503, and 504). Retries are not active by default; you need to specify the number of retries, as shown below. It already includes an exponential back-off retry delay between failed requests.
client.fetch() is the primary method for the main page-scraping product; client.get() still works and is kept as a deprecated alias.
from zenrows import ZenRowsClient
client = ZenRowsClient("YOUR-API-KEY", retries=1)
url = "https://www.zenrows.com/"
response = client.fetch(url, params={
# Our algorithm allows to automatically extract content from any website
"autoparse": False,
# CSS Selectors for data extraction (i.e. {"links":"a @href"} to get href attributes from links)
"css_extractor": "",
# Enable Javascript with a headless browser (5 credits)
"js_render": False,
# Use residential proxies (10 credits)
"premium_proxy": False,
# Make your request from a given country. Requires premium_proxy
"proxy_country": "us",
# Wait for a given CSS Selector to load in the DOM. Requires js_render
"wait_for": ".content",
# Wait a fixed amount of time in milliseconds. Requires js_render
"wait": 2500,
# Block specific resources from loading, check docs for the full list. Requires js_render
"block_resources": "image,media,font",
# Change the browser's window width and height. Requires js_render
"window_width": 1920,
"window_height": 1080,
# Will automatically use either desktop or mobile user agents in the headers
"device": "desktop",
# Will return the status code returned by the website
"original_status": False,
}, headers={
"Referrer": "https://www.google.com",
"User-Agent": "MyCustomUserAgent",
})
print(response.text)You can also pass optional params and headers; the list above is a reference. For more info, check out the documentation page.
Sending headers to the target URL will overwrite our defaults. Be careful when doing it and contact us if there is any problem.
Set mode="auto" to let Zenrows pick the request configuration for you — it starts with the cheapest viable setup and escalates to js_render/premium_proxy only when the target needs it, billing only for the configuration that succeeds.
response = client.get(url, params={"mode": "auto"})Compatible with proxy_country, js_instructions, and custom_headers.
The SDK also offers POST requests by calling the client.post method. It can receive a new parameter data that represents the data sent in, for example, a form.
from zenrows import ZenRowsClient
client = ZenRowsClient("YOUR-API-KEY", retries=1)
url = "https://httpbin.org/anything"
response = client.post(url, data={
"key1": "value1",
"key2": "value2",
})
print(response.text)The SDK also offers PUT requests by calling the client.put method. It can receive a new parameter data that represents the data sent in, for example, a form.
from zenrows import ZenRowsClient
client = ZenRowsClient("YOUR-API-KEY", retries=1)
url = "https://httpbin.org/anything"
response = client.put(url, data={
"key1": "value1",
"key2": "value2",
})
print(response.text)To limit the concurrency, it uses asyncio, which will simultaneously send a maximum of requests. The concurrency is determined by the plan you are in, so take a look at the pricing and set it accordingly. Take into account that each client instance will have its own limit, meaning that two different scripts will not share it, and 429 (Too Many Requests) errors might arise.
The main difference with the sequential snippet above is client.fetch_async instead of client.fetch. The rest will work exactly the same. But the async is necessary to parallelize calls and allow async/await syntax. Remember to run the scripts with asyncio.run or it will fail with a coroutine 'main' was never awaited error.
We use asyncio.gather in the example below. It will wait for all the calls to finish, and the results are stored in a responses array. The whole list of URLs will run, even if some fail. Then each response will have the status, request, response content, and other values as usual.
from zenrows import ZenRowsClient
import asyncio
client = ZenRowsClient("YOUR-API-KEY", concurrency=5, retries=1)
async def main():
urls = [
"https://www.zenrows.com/",
# ...
]
responses = await asyncio.gather(*[client.fetch_async(url) for url in urls])
for response in responses:
print(response.text)
asyncio.run(main())Extract (beta) runs a page through Zenrows' AI-powered structured extraction instead of returning raw HTML. Use client.extract() — it's the same request as fetch(), with the extract param set for you (defaults to "auto"; pass "native" or "standard" for the other contracts).
from zenrows import ZenRowsClient
client = ZenRowsClient("YOUR-API-KEY")
url = "https://www.zenrows.com/"
response = client.extract(url) # extract: "auto"
# response = client.extract(url, mode="native")
print(response.json())Batch is open to every account and spends the same credits as a regular Fetch request.
For workflows where you have many URLs to scrape and don't want to manage retries, concurrency, and pagination yourself, the Batch API submits a job (a list of tasks), runs it asynchronously on Zenrows' infrastructure, and lets you poll results when they're ready.
from zenrows import ZenRowsBatchClient
client = ZenRowsBatchClient(api_key="YOUR-API-KEY")
# 1. Submit a job — URLs as bare strings or dicts with per-task fields.
# submit_* returns a JobRef (no GET yet).
job = client.submit_regular(
[
{"url": "https://example.com/a", "external_id": "order-1"},
{"url": "https://example.com/b", "external_id": "order-2"},
],
zenrows_params={"js_render": "true", "premium_proxy": "true"},
)
# 2. Block until the current run is terminal; returns a RunHandle.
run = job.run.wait()
print(f"{run.stats.successful}/{run.stats.total} succeeded")
# 3. Inspect failures — each failed task carries an RFC 7807 error.
for task in run.results(status="failed"):
code = task.error.code if task.error else "unknown"
print(f" {task.external_id or task.task_id} failed: {code}")
# 4. Download every successful body to ./out — one file per task.
count = run.download_to_dir("./out")
print(f"downloaded {count} results to ./out/")Need full control over the request body?
client.submit_job({...})accepts a raw dict (the wire shape) or a typedSubmitJobRequest, and returns the sameJobRef.
Set extract in zenrows_params to run tasks through Extract — structured data
instead of raw HTML. It works job-wide or per task, and per-task values win on
collision.
job = client.submit_regular(
[
# Every task in this job runs through Extract...
{"url": "https://example.com/products", "external_id": "p1"},
# ...unless it overrides the job-level params.
{"url": "https://example.com/raw", "zenrows_params": {}},
],
zenrows_params={"extract": "auto"},
)An Extract task's result carries two keys — html (the raw page) and parsed
(the structured data). It costs the same as a regular task, so estimate_cost
prices it correctly. extract_fields is not supported in Batch yet.
file_input_id = client.upload_csv(
"leads.csv",
fields={"url": "Page URL", "external_id": "Lead Ref"},
header=True,
)
job = client.submit_regular(file_input_id=file_input_id)upload_csv allocates the slot, PUTs your file to the presigned URL,
and returns the file_input_id — one call instead of three.
Pricing is per successful request (base 1, js_render 5,
premium_proxy 10, both 25; mode=auto is dynamic 1–25).
client.estimate_cost answers "if every URL succeeds once, what's the
charge?" — pass the same body you'd submit, and get back a credit
interval (min == max unless the job uses mode=auto) with a per-tier
breakdown.
est = client.estimate_cost(
{
"type": "regular",
"tasks": [{"url": "https://a"}, {"url": "https://b"}],
"zenrows_params": {"js_render": "true"},
}
)
print(est) # "10 credits (2 tasks)"
print(est.format()) # per-tier breakdown tableEstimation is computed client-side from the SDK's rate card today, so it costs no API call; it may move server-side in a future release, so it lives on the client (
client.estimate_cost) to keep your call sites stable.
After a run finishes, read the actual credits it used from
run.stats.spend ({credits, cost}). Only successful requests are
charged, so realized spend is ≤ the estimate. (Per-task task.spend is
indicative and often absent on successful rows — rely on the run-level
rollup for actuals.)
After a run finishes with some failures, retry_failed() starts a new
run that re-executes only the failed tasks — successes are inherited
verbatim, so you don't pay to re-scrape them. (Thin shortcut for
rerun(status="failed").) Pass include_pending=True to also pick up
tasks that never started.
# If you still hold the `job` ref from submit, act on it directly:
run = job.retry_failed()
# Or, when you only kept the id (a CLI arg, your DB, a webhook),
# `client.job(id)` acts on it with no GET:
run = client.job(job_id).retry_failed()
run.wait()The Batch API doesn't ship page bodies inline — results() gives you
metadata plus a presigned result_url per task. The SDK downloads bodies
straight from that URL (no extra API round-trip). Four ways, on the
current run (job.run.*) or any specific run
(client.get_run(job_id, run_id=...)):
| You want… | Use | Where it lands |
|---|---|---|
| Every body, one file per task | run.download_to_dir(dir) |
files on disk |
| Every body, kept in the program | run.download_to_memory() |
list[DownloadedResult] |
| The whole run as a single artifact | run.download_all_results("out.zip") |
one .zip |
| One task's body (inside a loop) | run.download_task_to_file(task, path) / run.download_task_to_memory(task) |
file / bytes |
Bulk → disk. Iterates results, fetches each body, writes one file
per task. Tunable parallelism, a progress bar, and safety caps that
raise DownloadLimitExceeded before a runaway job fills the disk:
run.download_to_dir(
"./out",
status="successful", # default; pass status=None for everything
use_external_id=True, # name files <external_id>.<ext> vs <task_id>.<ext>
concurrency=8, # parallel body fetches
progress=True, # tqdm bar (soft dep)
max_files=20_000, # caps that raise DownloadLimitExceeded
max_bytes_per_file=10 * 1024 * 1024,
)Bulk → memory. Same iteration, no disk — each item carries
task_id, external_id, content_type, and body: bytes:
for r in run.download_to_memory(status="successful"):
process(r.body, r.content_type) # e.g. parse in-process, push to a queueWhole run → one zip. A server-side export: kick off the zip, poll until ready, stream it to disk in one call. Best when you want a single artifact:
run.download_all_results("results.zip")The server-side zip is capped at 1 GiB per run. For larger runs — or one file per task — use
download_to_dir, which fetches bodies client-side with no size limit (slower; tune withconcurrency=).
One task at a time. When you're iterating results() and want only
some bodies (a Python-side filter the status= argument can't express),
download them individually — straight from each task's result_url:
for task in run.results(status="successful"):
if task.external_id in wanted:
run.download_task_to_file(task, f"./out/{task.external_id}.html")
body = run.download_task_to_memory(task) # or just the raw bytesdownload_task_to_file also accepts an open binary file object, not
just a path. Every result's result_url is a plain presigned URL, so
you can equally GET it with your own HTTP client (no auth header) — it's
TTL-limited, so fetch it promptly. See examples 02 and 08.
Already have a job or run id (from a webhook, a queue, your own DB)?
client.job(id) / client.run(id, run_id) mint a JobRef / RunRef with
no network call — lifecycle ops act on the id directly; call .load()
for a loaded handle when you actually want the data:
client.job(job_id).delete() # DELETE — no preceding GET
client.job(job_id).run.stop() # stop the current run
client.run(job_id, run_id).delete() # scrub one specific run
client.job(job_id).load().data.status # explicit GET → JobHandleget_job / get_run remain the eager variants (a client.job(id).load()
in one call), for when you want the data up front. See example 10.
Submit a recurring job with a typed schedule builder, then pause /
resume / re-schedule it via the job.schedule facet. (This is distinct
from job.run.pause(), which reversibly suspends the current run's
dispatcher — two different endpoints the API keeps separate.)
from zenrows.batch import Rate
job = client.submit_scheduled(Rate(every=6, unit="hour"), ["https://example.com"])
job.schedule.pause() # stop firing (schedule keeps ticking)
job.schedule.resume() # fire again
job.schedule.update(Rate(every=1, unit="day")) # replace the scheduleRegister a completion webhook at submit time, or manage it later via
PUT / DELETE /jobs/{id}/webhook. Set signature=True to have each
delivery HMAC-signed (manage signing keys via the /hmac/keys
endpoints); it defaults to unsigned. Deliveries are at-least-once, so
dedup on the X-ZenRows-Event-Id header.
client.submit_regular(
["https://example.com/a"],
webhook={"url": "https://hooks.example.com/zr", "signature": True},
)No webhook receiver? Just poll for completion with
job.run.wait()/run.wait().
Every non-2xx surfaces as BatchAPIError; the code attribute carries
the stable code from the RFC 7807 body (file_input_not_found,
idempotency_key_conflict, api_key_cap_reached, etc.) and detail the
human-readable explanation (for a credit cap: which cap, and when it resets).
from zenrows.batch import BatchAPIError
try:
client.get_job("does-not-exist")
except BatchAPIError as exc:
if exc.code == "not_found":
...
raiseEvery enum the Batch API returns (RunStatus, FailureReason, ...) is open:
a value the server adds after your SDK version was released parses as an
UNKNOWN member whose .value is the raw string, instead of failing the
whole response. Branch on the members you know and keep a fallback:
from zenrows.batch.models import FailureReason
run = client.get_run(job_id, run_id=run_id).data
if run.failure_reason is FailureReason.API_KEY_CAP_REACHED:
print(run.failure_detail) # which cap was reached and when it resets
elif run.failure_reason is not None:
print("failed:", run.failure_reason.value) # includes values unknown to this SDKRuns that fail because an API key reached its credit cap carry
failure_reason = "api_key_cap_reached". Older SDK versions without open enums cannot
parse such runs, so upgrade before enabling API key credit caps with Batch.
The full Batch surface (jobs, runs, tasks, results, content, history,
file_inputs, HMAC keys) is reachable via methods on ZenRowsBatchClient.
See src/zenrows/batch/client.py or help(ZenRowsBatchClient).
Crawl is still evolving: new features are coming, limits may be tuned, and the changelog announces each change.
A crawl starts from one URL, follows the links on each page, and keeps
the URLs that match your patterns, up to max_items. It stops at
max_items or max_pages, or when no link is left. depth limits the
link hops from the start URL; without it the crawl has no depth limit.
It stays on the start URL's registrable domain;
subdomains count. It runs in the background: create returns at once with
status="running", and wait blocks until it ends. Without
output_format a crawl returns URLs only; output_format="html" also
returns each kept URL's page.
import os
from zenrows import ZenRowsCrawlClient
client = ZenRowsCrawlClient(api_key=os.environ["ZENROWS_API_KEY"])
crawl = client.create(
"https://www.scrapingcourse.com/ecommerce/",
depth=1, # link hops from the start URL (default: no limit)
max_items=50, # stop after keeping 50 URLs (default 100)
max_pages=100, # stop after fetching 100 pages (default 100)
include_patterns=["/product/"], # keep only URLs containing this
)
crawl = client.wait(crawl.crawl_id) # polls up to timeout=600 s
print(crawl.status.value, crawl.stop_reason, crawl.coverage)
for result in client.results(crawl.crawl_id): # follows next_cursor
print(result.url)wait returns the crawl when it ends, or when timeout (seconds,
default 600) runs out. It does not raise: on timeout the crawl is still
running (call wait again, or stop), and a failed crawl comes
back with crawl.error set. results does not poll: it returns at the first empty page, so on a
running crawl it yields only the URLs kept so far. Call wait first to
read every result. results(crawl_id, limit=...) sets the page size of
each request. get(crawl_id, cursor=..., limit=...) returns one raw page.
The client takes these options: base_url (default
https://api.zenrows.com/v1), retries (default 3; set 0 to disable)
and timeout (seconds per request, default 30):
client = ZenRowsCrawlClient(api_key, base_url="https://api.zenrows.com/v1", retries=3, timeout=30)Pass output_format="html" to also fetch the page of every kept URL:
crawl = client.create(start_url, depth=1, output_format="html")
client.wait(crawl.crawl_id)
for result in client.results(crawl.crawl_id):
if result.content_url: # content_status == fetched
html = client.content(crawl.crawl_id, result)
# Or everything in one NDJSON file: {"url", "content_status", "content"} per line.
with client.download(crawl.crawl_id) as download: # streamed, parsed
print(download.status) # from X-Crawl-Status
for line in download.lines:
print(line.url, line.content_status)content takes a result, its content_url, or its content id.
download returns a CrawlDownload: its status is the crawl's status
when the file was read, and its lines yields parsed lines. On a running crawl the
file holds only the results kept so far.
page = client.list(limit=20) # newest first, without results
for c in page.crawls:
print(c.crawl_id, c.status.value)
# Pass page.next_cursor as cursor= for the next page; it is None on the last.
client.stop(crawl_id) # idempotent; read final counts with get()Every non-2xx raises CrawlAPIError. Branch on status_code and code:
from zenrows import CrawlAPIError
from zenrows.crawl import CRAWL_NOT_ENABLED_CODE
try:
client.create(start_url, depth=1)
except CrawlAPIError as exc:
if exc.code == CRAWL_NOT_ENABLED_CODE: # 403 REQS008
print("Crawl is not enabled for this account")
elif exc.code == "too_many_crawls": # 429: active jobs limit reached
print("retry in", exc.retry_after, "s")
elif exc.status_code in (400, 422):
print(exc.detail)
else:
raise| Status | code |
What to do |
|---|---|---|
| 400 | invalid_request, unknown_parameter, invalid_cursor |
Fix the request. Do not retry it as is. |
| 403 | REQS008 |
Crawl is not enabled for the account (CRAWL_NOT_ENABLED_CODE). |
| 404 | crawl_not_found, content_not_found |
Check the id. |
| 409 | idempotency_request_in_flight |
Retry after the first request with this key finishes. |
| 422 | invalid_parameter, invalid_start_url, domain_not_allowed |
Fix the request. Do not retry it as is. |
| 422 | idempotency_key_reused |
Use a new key, or no key. Do not retry it as is. |
| 429 | too_many_crawls |
The account has reached its limit of active jobs (3 by default), shared with its Batch jobs. Retry after exc.retry_after seconds. create does not retry it. |
| 503 | crawl_busy |
stop could not be saved yet; the crawl is still running. stop retries it up to retries times, waiting exc.retry_after seconds, then raises it. Repeating the stop is safe. |
code is None when the response has no problem body with a code (for
example a 502 from a proxy). A crawl that ended failed is not an error
response: read crawl.error.code and crawl.error.detail on it. Response enums (CrawlStatus, StopReason,
ContentStatus, ...) are open: a value added later parses as an
UNKNOWN member that keeps the raw value.
Pull requests are welcome. For significant changes, please open an issue first to discuss what you would like to change.