Docs examples target https://api.scrapy.infralyon.com (not production https://api.scrapy.io).
Skip to main content

Monitor a run and retrieve dataset

After you start a sync or async job on a publisher host, use the Platform API to track progress and download rows.

Get one run​

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq .

Example payload:

{
  "data": {
    "id": "…",
    "kind": "async_batch",
    "status": "running",
    "publisher": "datadoping",
    "toolSlug": "instagram-profile-scraper",
    "runName": "ig-profiles-demo",
    "totalItems": 3,
    "processedItems": 1,
    "completedItems": 1,
    "failedItems": 0,
    "billedAmount": null,
    "estimatedCost": 0.03,
    "createdAt": "2026-09-07T10:00:00.000Z",
    "completedAt": null
  }
}

List runs​

# Latest runs for your account
curl -s "https://api.scrapy.infralyon.com/v1/runs?limit=20" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data'

# Only async scraper runs
curl -s "https://api.scrapy.infralyon.com/v1/runs?kind=scraper&limit=20" \
  -H "Authorization: Bearer $SCRAPY_API_KEY"

# Filter by public status (succeeded maps from completed)
curl -s "https://api.scrapy.infralyon.com/v1/runs?status=succeeded&limit=20" \
  -H "Authorization: Bearer $SCRAPY_API_KEY"

Filters:

QueryValues
kindapi_sync, async_batch, or aliases api, scraper
toolIdInternal tool id (when you already have it)
statusqueued, running, succeeded, failed, cancelled, partial
offset / limitPagination

Polling pattern​

while true; do
  STATUS=$(curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID" \
    -H "Authorization: Bearer $SCRAPY_API_KEY" | jq -r '.data.status')
  echo "status=$STATUS"
  case "$STATUS" in
    succeeded|partial|failed|cancelled) break ;;
  esac
  sleep 4
done

Use exponential backoff in production clients if you poll often.

Dataset items (JSON)​

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID/dataset/items?offset=0&limit=100" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data.items | length'

Empty items with total: 0 usually means the run has no successful rows yet, or failed before producing output — check status and run error fields in the console if needed.

Ownership​

You can only read runs created by your API key’s account. Another account’s runId returns:

{
  "error": {
    "type": "not_found",
    "message": "Run was not found",
    "doc_url": "https://docs.scrapy.io/errors/not-found"
  }
}