Docs examples target https://api.scrapy.infralyon.com (not production https://api.scrapy.io).
Skip to main content

Run a tool

This guide walks through discovering a tool, executing it on a publisher host, and connecting that run to the Platform API.

Before you start​

  • API key set as SCRAPY_API_KEY
  • Enough credits for the tool’s pricePerResult × number of items
  • Know whether you need sync (one item) or async (many items)

Step 1 — Discover the tool​

curl -s "https://api.scrapy.infralyon.com/v1/tools?q=instagram+profile&limit=5" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data.items[] | {publisher, slug, name, execution}'

Or open the tool in the Store UI and copy publisher + slug from the URL / API tab.

Step 2 — Read the input contract​

curl -s "https://api.scrapy.infralyon.com/v1/tools/datadoping/instagram-profile-scraper" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data.inputSchema'

Also check the Store Input tab — it is generated from the same schema family.

Step 3a — Sync execution​

Use when you have one primary value and want the result in the HTTP response.

curl -s -X POST \
  "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/api" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"username":"nasa"}'

Notes:

  • Body fields are tool-specific (not a generic inputs[] wrapper on /api).
  • Do not send toolShortId on publisher URLs — the tool is already in the path.
  • On success you get publisher JSON. On failure you get an error response from the publisher API.

Sync runs also appear under Platform GET /v1/runs?kind=api_sync for history.

Step 3b — Async execution​

Use when you have many inputs or a long job.

curl -s -X POST \
  "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/scraper" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "usernames": ["nasa", "spacex", "esa"],
    "runName": "ig-profiles-demo"
  }'

Save the returned run / task id. Example field names vary by tool (usernames, post_urls, inputs, …) — follow inputSchema.

Optional wait​

Some scraper endpoints support:

curl -s -X POST \
  "https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/scraper?wait=true" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"usernames":["nasa"]}'

wait=true blocks until the run finishes or a timeout is hit. For large batches, prefer async without wait + Platform polling.

Step 4 — Monitor on Platform​

export RUN_ID="paste-id-here"

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data | {id, status, kind, processedItems, totalItems}'

Repeat until status is terminal (succeeded, partial, failed, cancelled).

Step 5 — Fetch results​

curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID/dataset/items?limit=100" \
  -H "Authorization: Bearer $SCRAPY_API_KEY" | jq '.data'

See Monitor and dataset and Export dataset for formats and pagination.

Common mistakes​

MistakeFix
Calling api.scrapy.infralyon.com to “run” a scraperUse {publisher}.p.scrapy.infralyon.com for execution
Wrong field names in bodyCheck inputSchema / Store Input tab
Expecting Platform to invent run idsCapture id from publisher response
Polling forever on failed runsStop on terminal statuses
Using another user’s runIdOwnership is enforced; you get 404