Docs examples target https://api.scrapy.infralyon.com (not production https://api.scrapy.io).
Skip to main content

Python SDK

Official Python client for the Scrapy.io Platform API and publisher execution APIs.

Published version: 0.1.3
Package: scrapyio-sdk

Server-side only. Keep API keys off client devices.

Install​

pip install scrapyio-sdk

Initialize​

import os
from scrapyio_sdk import ScrapyIO

client = ScrapyIO(api_key=os.environ["x-api-key"])

The SDK always sends x-api-key: <api_key>.

Use x-api-key for SDKs and MCP. Raw curl / HTTP docs on this site use SCRAPYIO_API_KEY as a shell variable for the same key value — see Authentication.

Optional constructor fields: base_url (default https://api.scrapy.infralyon.com/v1), max_retries (default 2), retry_backoff_ms (default 250).

Discover tools​

tools = client.tools.list(q="instagram", limit=20)
# {"items", "total", "offset", "limit"}

tool = client.tools.get("datadoping", "instagram-profile-scraper")
# or:
same = client.tool("datadoping/instagram-profile-scraper").get()

Run a scraper​

Sync — .call()​

result = client.tool("datadoping/instagram-profile-scraper").call({
    "username": "nasa",
})

Async — .start() + wait + dataset​

run = client.tool("datadoping/instagram-profile-scraper").start({
    "usernames": ["nasa"],
})

finished = client.run(run["id"]).wait(
    poll_interval_ms=3000,
    timeout_ms=15 * 60_000,
)

page = client.run(run["id"]).list_items(offset=0, limit=100)

Also available: client.runs.list(...), client.runs.get(run_id), client.run(run_id).get().

Schedules​

schedule = client.schedules.create(
    {
        "publisher": "datadoping",
        "slug": "instagram-profile-scraper",
        "runName": "nightly-ig",
        "timezone": "UTC",
        "frequency": "one-time",
        "date": "2026-09-20",
        "time": "23:50",
        "inputs": ["nasa"],
    },
    idempotency_key="nightly-ig-v1",
)

client.schedules.update(schedule["id"], {"isActive": False})
client.schedules.delete(schedule["id"])

Also: client.schedules.list(...), client.schedules.get(id).

Errors​

from scrapyio_sdk import ScrapyIO, ScrapyIOError

try:
    client.tools.get("nope", "missing")
except ScrapyIOError as err:
    # err.type, err.status, err.message, err.doc_url
    ...

Retries​

GET/HEAD Platform requests may retry on transient 5xx / 429 (default up to 2 retries with exponential backoff). Execution POSTs (.call() / .start()) are not auto-retried.

Limitations (v0.1)​

  • list_items() is JSON-only (omit format or pass "json"). For CSV/JSONL, use the Platform HTTP dataset export endpoints.
  • Requires Python 3.10+.

Mental model​

client
├── tools.list / tools.get / tools.get_readme / tools.list_reviews
├── account.get / account.usage
├── tool("publisher/slug").get / .call / .start
├── runs.list / runs.get / runs.abort / runs.resume
├── run(id).get / .wait / .list_items / .abort / .resume
└── schedules.list / get / create / update / delete