Crawlers

Queue crawler

Website crawlers currently queued or running. Authenticate with a bearer API key that includes the required scope.

Required scopecrawlers:read

curl -X GET https://api.agentency.com/v1/crawlers/queue \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

List crawlers

List crawlers. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND. Mutations accept Idempotency-Key.

Required scopecrawlers:read

{
  "object": "list",
  "data": [
    {
      "object": "website_crawler",
      "id": 1,
      "name": "Repudiandae laboriosam laboriosam quisquam.",
      "url": "http://keeling.com/voluptatem-incidunt-unde-voluptas-molestiae-corporis-eos-voluptatem",
      "status": "completed",
      "chatbot_id": 1,
      "error_code": null,
      "created_at": "2026-01-01T00:00:00+00:00",
      "updated_at": "2026-01-01T00:00:00+00:00"
    }
  ],
  "has_more": false,
  "next_cursor": null,
  "previous_cursor": null,
  "url": "/v1/crawlers"
}

curl -X GET https://api.agentency.com/v1/crawlers \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Create a crawler

Create a website crawler. Same orchestration as the dashboard (plan knowledge quota → 402 `KNOWLEDGE_PLAN_SIZE_EXCEEDED`, default collection + chatbot attachment, per-account concurrent cap → 429 `CRAWLER_LIMIT_REACHED`, in-flight dedupe → 200 with the existing crawler). Accepts Idempotency-Key and emits `crawler.created` when a crawler is created.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/crawlers \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "description": "description",
  "base_url": "base_url",
  "crawl_mode": "crawl_mode",
  "max_depth": "max_depth",
  "max_pages": "max_pages",
  "crawl_delay_seconds": "crawl_delay_seconds",
  "crawl_concurrency": "crawl_concurrency"
}'

Retrieve a crawler

Retrieve a crawler. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND.

Required scopecrawlers:read

{
  "object": "website_crawler",
  "id": 1,
  "name": "Repudiandae laboriosam laboriosam quisquam.",
  "url": "http://keeling.com/voluptatem-incidunt-unde-voluptas-molestiae-corporis-eos-voluptatem",
  "status": "completed",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

curl -X GET https://api.agentency.com/v1/crawlers/{id} \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Update a crawler

Update a website crawler. 409 `CRAWLER_RUNNING` while the crawl is in flight. Accepts Idempotency-Key and emits `crawler.updated`.

Required scopecrawlers:write

curl -X PATCH https://api.agentency.com/v1/crawlers/{id} \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "max_pages": 1,
  "crawl_delay_seconds": 1
}'

Delete a crawler

Delete a website crawler (cancels it first when in flight). Accepts Idempotency-Key and emits `crawler.deleted`.

Required scopecrawlers:write

curl -X DELETE https://api.agentency.com/v1/crawlers/{id} \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Pages crawler

Paginated crawled pages of one website crawler. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND.

Required scopecrawlers:read

curl -X GET https://api.agentency.com/v1/crawlers/{id}/pages \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Statistics crawler

Statistics crawler. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND.

Required scopecrawlers:read

curl -X GET https://api.agentency.com/v1/crawlers/{id}/statistics \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Sitemap index crawler

List sitemap crawlers. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND.

Required scopecrawlers:read

{
  "object": "list",
  "data": [
    {
      "object": "sitemap_crawler",
      "id": 1,
      "name": "Quia aut.",
      "url": "http://www.emard.com/porro-dolore-illum-dolorem-dolores/sitemap.xml",
      "status": "pending",
      "chatbot_id": 1,
      "error_code": null,
      "created_at": "2026-01-01T00:00:00+00:00",
      "updated_at": "2026-01-01T00:00:00+00:00"
    }
  ],
  "has_more": false,
  "next_cursor": null,
  "previous_cursor": null,
  "url": "/v1/sitemap_crawlers"
}

curl -X GET https://api.agentency.com/v1/sitemap_crawlers \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Store sitemap crawler

Create a sitemap crawler. Same orchestration as the dashboard (plan knowledge quota → 402, sitemap pre-flight → 422 `VALIDATION_FAILED`, default collection + chatbot attachment, per-account concurrent cap → 429 `CRAWLER_LIMIT_REACHED`, in-flight dedupe → 200 with the existing crawler). Accepts Idempotency-Key and emits `crawler.created` when a crawler is created.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/sitemap_crawlers \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "sitemap_url": "sitemap_url",
  "chatbot_id": 1,
  "knowledge_collection_id": 1,
  "auto_train": "auto_train",
  "urls_per_scan": "urls_per_scan",
  "max_total_urls": "max_total_urls",
  "crawl_delay_seconds": "crawl_delay_seconds"
}'

Sitemap show crawler

Retrieve a sitemap crawler. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND.

Required scopecrawlers:read

curl -X GET https://api.agentency.com/v1/sitemap_crawlers/{id} \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Update sitemap crawler

Update a sitemap crawler's own tunables (`name`, `urls_per_scan`, `max_total_urls`, `crawl_delay_seconds`, `scan_frequency`). 409 `CRAWLER_RUNNING` while a scan is in flight. Accepts Idempotency-Key and emits `crawler.updated`.

Required scopecrawlers:write

curl -X PATCH https://api.agentency.com/v1/sitemap_crawlers/{id} \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "urls_per_scan": 1,
  "max_total_urls": "max_total_urls",
  "crawl_delay_seconds": 1,
  "scan_frequency": "scan_frequency"
}'

Destroy sitemap crawler

Delete a sitemap crawler (cancels it first when in flight). Accepts Idempotency-Key and emits `crawler.deleted`.

Required scopecrawlers:write

curl -X DELETE https://api.agentency.com/v1/sitemap_crawlers/{id} \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Sitemap pages crawler

Paginated URL rows of one sitemap crawler. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND.

Required scopecrawlers:read

curl -X GET https://api.agentency.com/v1/sitemap_crawlers/{id}/pages \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Sitemap statistics crawler

Sitemap statistics crawler. Authenticate with a bearer API key that includes the required scope. Resources that belong to another account return 404 NOT_FOUND.

Required scopecrawlers:read

curl -X GET https://api.agentency.com/v1/sitemap_crawlers/{id}/statistics \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Validate url crawler

Pre-flight probe of a URL before creating a crawler. Mutates nothing (`crawlers:read`); rides the tight `crawling_preflight` budget because it is a synchronous outbound fetch.

Required scopecrawlers:read

curl -X POST https://api.agentency.com/v1/crawlers/validate_url \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "url": "url"
}'

Validate a sitemap URL

Fetches and parses a sitemap before you commit to crawling it, so you can surface "that URL is not a sitemap" to your own user instead of creating a crawler that immediately fails. Mutates nothing, which is why it only needs `crawlers:read`; rides the tight `crawling_preflight` budget because it is a synchronous outbound fetch.

Required scopecrawlers:read

curl -X POST https://api.agentency.com/v1/sitemap_crawlers/validate_url \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "sitemap_url": "sitemap_url"
}'

Cancel queued crawler

Cancel one website or sitemap crawler by `{crawler_id, crawler_type}`. 409 `CRAWLER_TERMINAL` when it already finished. Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/crawlers/queue/cancel \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "crawler_id": 1,
  "crawler_type": "crawler_type"
}'

Pause crawler

Pause a running (or queued) website crawler. 409 `CRAWLER_NOT_RUNNING` otherwise. Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/crawlers/{id}/pause \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Resume crawler

Resume a paused website crawler. 409 `CRAWLER_NOT_PAUSED` when it is not paused, 409 `CRAWLER_INACTIVE` when it was deactivated. Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/crawlers/{id}/resume \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Retry crawler

Re-run a website crawler that is not running. 409 `CRAWLER_RUNNING` while in flight, 409 `CRAWLER_INACTIVE` after cancellation / deactivation, 409 `CRAWL_NOT_RETRYABLE` while paused (resume it first). Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/crawlers/{id}/retry \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Cancel crawler

Cancel a website crawler that has not finished. 409 `CRAWLER_TERMINAL` for a completed / failed / cancelled one. Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/crawlers/{id}/cancel \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Pause sitemap crawler

Pause a running (or queued) sitemap crawler. 409 `CRAWLER_NOT_RUNNING` otherwise. Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/sitemap_crawlers/{id}/pause \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Resume sitemap crawler

Resume a paused sitemap crawler. 409 `CRAWLER_NOT_PAUSED` when it is not paused, 409 `CRAWLER_INACTIVE` when it was deactivated. Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/sitemap_crawlers/{id}/resume \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Scan sitemap crawler

Trigger a scan of a sitemap crawler now. 409 `CRAWLER_RUNNING` while a scan is in flight, 409 `CRAWLER_INACTIVE` after cancellation / deactivation, 409 `CRAWL_NOT_RETRYABLE` while paused (resume it first). Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/sitemap_crawlers/{id}/scan \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Retry sitemap crawler

Re-run a sitemap crawler that is not running. 409 `CRAWLER_RUNNING` while in flight, 409 `CRAWLER_INACTIVE` after cancellation / deactivation, 409 `CRAWL_NOT_RETRYABLE` while paused (resume it first). Accepts Idempotency-Key.

Required scopecrawlers:write

curl -X POST https://api.agentency.com/v1/sitemap_crawlers/{id}/retry \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"