Sync vs async
Scrapy.io tools can be called in two execution modes on the publisher host. Both use the same API key. After the run exists, use the Platform API to inspect status and datasets.
Comparison
Sync (/v1/api) | Async (/v1/scraper) | |
|---|---|---|
| Best for | Single input, low latency | Many inputs, long jobs |
| Request | One JSON body (e.g. {"username":"nasa"}) | Batch fields (e.g. {"usernames":[...]}) |
| Response | Result inline (or error) | Run accepted; returns run / task id |
Platform kind | api_sync | async_batch |
| Typical flow | Call → done | Call → poll → dataset |
Sync example
curl -s -X POST \
"https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/api" \
-H "Authorization: Bearer $SCRAPY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"username":"nasa"}'
Use sync when you need a single record in one HTTP round-trip (enrichment, agent tools, form handlers).
Async example
curl -s -X POST \
"https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/scraper" \
-H "Authorization: Bearer $SCRAPY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"usernames":["nasa","spacex","esa"]}'
Optional: some publisher scraper routes support ?wait=true to block until the run finishes (with a timeout). For production batch jobs, prefer fire-and-forget + Platform polling.
After execution — Platform
# List recent runs (filter by kind)
curl "https://api.scrapy.infralyon.com/v1/runs?kind=async_batch&limit=10" \
-H "Authorization: Bearer $SCRAPY_API_KEY"
# One run
curl "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID" \
-H "Authorization: Bearer $SCRAPY_API_KEY"
# Rows
curl "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID/dataset/items?limit=100" \
-H "Authorization: Bearer $SCRAPY_API_KEY"
kind query aliases: api → api_sync, scraper → async_batch.
Public run statuses
| Status | Meaning |
|---|---|
queued | Accepted, not started |
running | In progress |
succeeded | Finished successfully (maps from internal completed) |
partial | Finished with some item failures |
failed | Failed |
cancelled | Cancelled by user or system |
Poll until a terminal status: succeeded, partial, failed, or cancelled.