Getting started

Scrapy.io is a marketplace for web scrapers and data APIs. You discover tools in the Store, run them with an API key, and pay per successful result.

This documentation covers the public developer APIs: how to authenticate, discover tools, execute jobs, monitor runs, export datasets, and create schedules.

Two hosts, one product

Scrapy.io splits platform concerns from execution:

Host	Role	Example
https://api.scrapy.infralyon.com	Platform API — catalog, runs, datasets, schedules	GET /v1/tools
https://{publisher}.p.scrapy.infralyon.com	Execution — start a sync or async scrape	POST /instagram-profile-scraper/v1/api

The Platform API does not start scrapes. Publisher hosts do. After a run starts, use the Platform API to poll status and download results.

Discover tool (api.scrapy.infralyon.com)
        ↓
Execute (publisher.p.scrapy.infralyon.com)
        ↓
Poll run + fetch dataset (api.scrapy.infralyon.com)

Prerequisites

1. A Scrapy.io account
2. An API key from the console (Integrations / API keys)
3. Enough credits for the tool’s pricePerResult

export SCRAPY_API_KEY="scrapy_api_xxxxxxxx"

Always send the key as a Bearer token (preferred) or X-API-Key header. Never put keys in query strings or commit them to git.

Five-minute walkthrough

1. List tools

curl -s "https://api.scrapy.infralyon.com/v1/tools?q=instagram&limit=5" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq .

Response shape (collections always use this envelope):

{
  "data": {
    "items": [
      {
        "publisher": "datadoping",
        "slug": "instagram-profile-scraper",
        "name": "Instagram Profile Scraper",
        "pricePerResult": 0.01,
        "execution": {
          "apiUrl": "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/api",
          "scraperUrl": "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/scraper"
        }
      }
    ],
    "total": 12,
    "offset": 0,
    "limit": 5
  }
}

2. Inspect one tool

curl -s "https://api.scrapy.infralyon.com/v1/tools/datadoping/instagram-profile-scraper" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data.inputSchema, .data.execution'

Use inputSchema to learn required fields. Use execution.apiUrl / execution.scraperUrl for the publisher host.

3. Run sync (one request → one result)

curl -s -X POST "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/api" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"username":"nasa"}'

4. Or run async (batch)

curl -s -X POST "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/scraper" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"usernames":["nasa","spacex"]}'

Save the returned run / task id.

5. Poll and download on the Platform API

RUN_ID="your-run-id"

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data.status'

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID/dataset/items?limit=100" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq .

What to read next

Goal	Page
Auth, keys, ownership	Authentication
Publisher / slug model	Publishers and tools
Sync vs batch	Sync vs async
End-to-end execute guide	Run a tool
Polling + exports	Monitor and dataset
Recurring jobs	Schedules
Full Platform reference	Platform overview
Error types	Errors

OpenAPI (machine-readable): https://docs.scrapy.io/openapi/platform-v1.yaml