Pagination
List endpoints on the Platform API use a consistent pagination model.
Query parameters
| Param | Default | Max | Description |
|---|---|---|---|
offset | 0 | — | Number of items to skip |
limit | 20 | 100 | Page size |
Invalid values return validation_error (HTTP 400).
Response envelope
{
"data": {
"items": [ ],
"total": 256,
"offset": 0,
"limit": 20
}
}
| Field | Meaning |
|---|---|
items | Current page |
total | Total matching rows (for the same filters) |
offset | Echo of request offset |
limit | Echo of request limit |
Walking all pages
OFFSET=0
LIMIT=100
while true; do
RESP=$(curl -s "https://api.scrapy.infralyon.com/v1/runs?offset=$OFFSET&limit=$LIMIT" \
-H "Authorization: Bearer $SCRAPY_API_KEY")
COUNT=$(echo "$RESP" | jq '.data.items | length')
TOTAL=$(echo "$RESP" | jq '.data.total')
echo "got $COUNT (offset=$OFFSET / total=$TOTAL)"
[ "$COUNT" -eq 0 ] && break
OFFSET=$((OFFSET + LIMIT))
[ "$OFFSET" -ge "$TOTAL" ] && break
done
Dataset items
GET /v1/runs/{runId}/dataset/items also supports offset and limit for JSON responses. Export formats (format=csv, etc.) apply the same windowing where applicable.