Docs examples target https://api.scrapy.infralyon.com (not production https://api.scrapy.io).
Skip to main content

Getting started

Scrapy.io is a marketplace for web scrapers and data APIs. You discover tools in the Store, run them with an API key, and pay per successful result.

Follow Run scrapers in the sidebar the first time. Open API reference when you already know the endpoint you need.

Two hosts, one product​

Scrapy.io splits platform concerns from execution:

HostRoleExample
https://api.scrapy.infralyon.comPlatform API — catalog, runs, datasets, schedulesGET /v1/tools
https://{publisher}.p.scrapy.infralyon.comExecution — start a sync or async scrapePOST /instagram-profile-scraper/v1/api

The Platform API does not start scrapes. Publisher hosts do. After a run starts, use the Platform API to poll status and download results.

Discover tool (api.scrapy.infralyon.com)
        ↓
Execute (publisher.p.scrapy.infralyon.com)
        ↓
Poll run + fetch dataset (api.scrapy.infralyon.com)

Prerequisites​

  1. A Scrapy.io account
  2. An API key from the console (Integrations / API keys)
  3. Enough credits for the tool’s pricePerResult
export SCRAPYIO_API_KEY="sc_..."

Always send the key as an x-api-key header (preferred) or Authorization: Bearer. Never put keys in query strings or commit them to git.

Five-minute walkthrough​

1. List tools​

curl -s "https://api.scrapy.infralyon.com/v1/tools?q=instagram&limit=5" \
  -H "x-api-key: $SCRAPYIO_API_KEY" | jq .

Response shape (collections always use this envelope):

{
  "data": {
    "items": [
      {
        "publisher": "datadoping",
        "slug": "instagram-profile-scraper",
        "name": "Instagram Profile Scraper",
        "pricePerResult": 0.01,
        "execution": {
          "apiUrl": "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/api",
          "scraperUrl": "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/scraper"
        }
      }
    ],
    "total": 12,
    "offset": 0,
    "limit": 5
  }
}

2. Inspect one tool​

curl -s "https://api.scrapy.infralyon.com/v1/tools/datadoping/instagram-profile-scraper" \
  -H "x-api-key: $SCRAPYIO_API_KEY" | jq '.data.inputSchema, .data.execution'

Use inputSchema to learn required fields. Use execution.apiUrl / execution.scraperUrl for the publisher host.

3. Run sync (one request → one result)​

curl -s -X POST "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/api" \
  -H "x-api-key: $SCRAPYIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"username":"nasa"}'

4. Or run async (batch)​

curl -s -X POST "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/scraper" \
  -H "x-api-key: $SCRAPYIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"usernames":["nasa","spacex"]}'

Save the returned run / task id.

5. Poll and download on the Platform API​

RUN_ID="your-run-id"

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID" \
  -H "x-api-key: $SCRAPYIO_API_KEY" | jq '.data.status'

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID/dataset/items?limit=100" \
  -H "x-api-key: $SCRAPYIO_API_KEY" | jq .

Where to go next​

  1. Authentication — create a key and send it on every request
  2. Find a scraper — pick a tool and copy its endpoint
  3. Run a scraper — inputs, sync, and async
  4. Download results — JSON, CSV, or JSONL

Call Scrapy from code with the JavaScript SDK, Python SDK, or MCP.

Look up an endpoint in the API reference. Machine-readable contract: https://docs.scrapy.io/openapi/platform-v1.yaml.