Web Scraping API Documentation
Scrapy.io is a web scraping API platform (also called a web scraper API or website scraping API): you call HTTP endpoints with an API key, run marketplace scrapers, and download structured datasets — without hosting browsers or proxies yourself.
This page is the SEO and product overview for developers searching for scraper API documentation. For the machine-readable contract, see the Platform API overview and OpenAPI.
What you can do with the Scrapy.io scraper API
| Capability | Where |
|---|---|
| Discover scrapers / tools | GET https://api.scrapy.infralyon.com/v1/tools |
| Run one result (sync) | POST https://{publisher}.p.scrapy.infralyon.com/{tool}/v1/api |
| Run a batch (async) | POST https://{publisher}.p.scrapy.infralyon.com/{tool}/v1/scraper |
| Poll job status | GET https://api.scrapy.infralyon.com/v1/runs/{runId} |
| Dataset API — export rows | GET https://api.scrapy.infralyon.com/v1/runs/{runId}/dataset/items |
| Schedule recurring scrapes | POST https://api.scrapy.infralyon.com/v1/schedules |
Why teams choose a scraping API
Compared with building your own crawlers, a hosted data extraction API gives you:
- Ready-made tools for Instagram, LinkedIn, Google Maps, TikTok, and more
- Pay-per-result billing (you are charged for successful rows)
- One API key for discovery, runs, datasets, and schedules
- JSON / CSV / JSONL exports for warehouses and AI agents
If you are evaluating an Apify alternative, Scrapy.io keeps a similar developer workflow (run → poll → dataset) with Scrapy.io’s own publisher URLs and Platform API.
Quick start (curl)
export SCRAPY_API_KEY="scrapy_api_..."
# 1) Find a tool
curl -s "https://api.scrapy.infralyon.com/v1/tools?q=instagram&limit=5" \
-H "Authorization: Bearer $SCRAPY_API_KEY"
# 2) Run the web scraper API (sync)
curl -s -X POST \
"https://datadoping.p.scrapy.infralyon.com/instagram-profile-scraper/v1/api" \
-H "Authorization: Bearer $SCRAPY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"username":"nasa"}'
# 3) Fetch dataset rows (dataset API)
curl -s "https://api.scrapy.infralyon.com/v1/runs/$RUN_ID/dataset/items?limit=100" \
-H "Authorization: Bearer $SCRAPY_API_KEY"
Python example
import os
import requests
API_KEY = os.environ["SCRAPY_API_KEY"]
headers = {"Authorization": f"Bearer {API_KEY}"}
tools = requests.get(
"https://api.scrapy.infralyon.com/v1/tools",
params={"q": "linkedin", "limit": 5},
headers=headers,
timeout=60,
).json()
print(tools["data"]["items"][0]["execution"])
See also: how to use the scraper API and web scraper API in Python.
Architecture (two hosts)
Web scraping API (docs + Platform) Execution (publisher)
api.scrapy.infralyon.com/v1 {publisher}.p.scrapy.infralyon.com
tools / runs / datasets / schedules /v1/api | /v1/scraper
The Platform API does not execute scrapes; publisher hosts do. That split keeps the platform API stable while tools evolve.
Popular tool verticals
Developers often search for niche scraper APIs. On Scrapy.io these are marketplace tools under a publisher slug, for example:
- Instagram scraper API
- LinkedIn scraper API / LinkedIn profile scraper API
- Google Maps scraper API
- TikTok, YouTube, Reddit, Facebook scraper APIs
Discover them with GET /v1/tools?q=... or the Store.