Docs examples target https://api.scrapy.infralyon.com (not production https://api.scrapy.io).
Skip to main content

MCP (Model Context Protocol)

Connect Scrapy.io to your AI tools with MCP. Agents can discover tools, run scrapers, wait for runs, read dataset items, and manage schedules.

Package: @scrapyio/mcp

You can connect in two ways:

  • Local (stdio) — your AI app starts @scrapyio/mcp as a process (typical for Cursor and Claude Desktop)
  • Remote (Streamable HTTP) — your AI app connects to https://mcp.scrapy.infralyon.com

What you can do​

  • Discover Scrapy.io tools and input schemas
  • Run tools synchronously or start async runs
  • Monitor runs and list JSON dataset items
  • Create and manage schedules

Authentication​

Use your own Scrapy.io API key (sc_...). Never put the key in MCP tool arguments.

  • Stdio: set x-api-key in the MCP server’s environment (see configs below)
  • Remote: send the key when the client connects — Authorization: Bearer YOUR_API_KEY or x-api-key: YOUR_API_KEY

OAuth is not supported. Create or regenerate a key in your Scrapy.io account settings.

Connect your AI client​

Cursor​

Paste into Cursor Settings → MCP (or your mcp.json):

{
  "mcpServers": {
    "scrapyio": {
      "command": "npx",
      "args": ["-y", "@scrapyio/mcp"],
      "env": {
        "x-api-key": "YOUR_API_KEY"
      }
    }
  }
}

Replace YOUR_API_KEY with your Scrapy.io API key.

Claude Desktop​

Use the same stdio config as Cursor in Claude Desktop’s MCP settings.

ChatGPT and other remote MCP clients​

  1. Point the client at the Scrapy.io MCP server:
    https://mcp.scrapy.infralyon.com
  2. Authenticate with your API key on connect:
Authorization: Bearer YOUR_API_KEY

or

x-api-key: YOUR_API_KEY

Example config shape (exact fields vary by client):

{
  "mcpServers": {
    "scrapyio": {
      "url": "https://mcp.scrapy.infralyon.com",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Other MCP clients​

  • If the client supports remote MCP, use the Streamable HTTP URL and headers above
  • Otherwise use the stdio npx -y @scrapyio/mcp config

Install (optional)​

npm install -g @scrapyio/mcp

Or run without a global install via npx -y @scrapyio/mcp (as in the Cursor config).

Available tools​

CategoryTools
Discoveryscrapyio_list_tools, scrapyio_get_tool, scrapyio_get_tool_readme, scrapyio_list_tool_reviews
Accountscrapyio_get_account, scrapyio_get_usage
Executionscrapyio_call_tool, scrapyio_start_tool, scrapyio_list_runs, scrapyio_get_run, scrapyio_abort_run, scrapyio_resume_run, scrapyio_wait_for_run, scrapyio_list_run_items
Schedulesscrapyio_list_schedules, scrapyio_get_schedule, scrapyio_create_schedule, scrapyio_update_schedule, scrapyio_delete_schedule

Example workflow​

  1. List tools with a search query (scrapyio_list_tools)
  2. Get a tool’s input schema (scrapyio_get_tool)
  3. Start an async run (scrapyio_start_tool), wait (scrapyio_wait_for_run), then list items (scrapyio_list_run_items)

Limitations​

  • Remote MCP uses API-key auth on connect (not OAuth)
  • Dataset export formats other than JSON are not exposed through MCP