Crawlers

Operations tagged Crawlers in the public API.

About Crawlers

A crawler collects a website's pages into knowledge, one dataset per page. There are two kinds, each with its own paths and ids: a website crawler (/crawlers) follows links from a start URL within the depth and page limits you set; a sitemap crawler (/sitemap_crawlers) fetches the pages a sitemap.xml lists, and is the better choice when the site has one.

Lifecycle. Creating a crawler checks and probes the URL, queues the first run and emits crawler.created. status moves pending → running → completed or failed, emitting crawl.completed or crawl.failed; pause, resume, cancel and retry move it by hand and emit nothing. A recurring crawler (crawl_frequency, scan_frequency) waits as pending between runs. With chatbot_id and auto_train (on by default) every collected page trains into the chatbot, and each page's dataset emits its own knowledge.dataset.trained or .failed.

Limits and duplicates. An account runs at most 5 website and 10 sitemap crawls at once by default; one more is 429 CRAWLER_LIMIT_REACHED, and resuming, retrying or re-scanning at the cap is refused the same way. Creating a crawler for a URL and chatbot that already has one pending or running returns that crawler with 200 instead of a second one, and emits nothing. A page a crawler already saved in its collection is updated in place on a later run, not added twice.

URLs. Only public http(s) addresses are crawled: a private, loopback, link-local or otherwise blocked address is refused before anything is fetched, every redirect is checked the same way, and the connection is pinned to the address that was checked. respect_robots_txt (on by default) obeys the site's robots.txt. POST /crawlers/validate_url and POST /sitemap_crawlers/validate_url run the pre-flight without creating anything.

Webhooks. crawler.* and crawl.* events carry only {object, id}, and website and sitemap crawlers are numbered separately: keep each id with its kind.

IDs you will meet.

id — a crawler id, an integer, in every /crawlers/{id} (website) or /sitemap_crawlers/{id} (sitemap) path.
crawler_id with crawler_type — how POST /crawlers/queue/cancel names a crawler of either kind.

IDs on this page

Knowledge collection id<COLLECTION_ID>: Integer.
Get it from
POST /v1/knowledge/collections returns it as id
GET /v1/knowledge/collections returns it as data[].id
POST /v1/knowledge/collections/auto_create returns it as id
GET /v1/knowledge/{id} returns it as collection_id
Webhook knowledge.collection.created carries it as data.object.id
Adding knowledge or a crawl with only chatbot_id files it in the chatbot's default collection, created on first use: read its id as collection_id on any of the chatbot's datasets.
Chatbot id<CHATBOT_ID>: Integer.
Get it from
GET /v1/chatbots returns it as data[].id
POST /v1/chatbots returns it as id
GET /v1/chatbots/by_handle/{handle} returns it as id
Webhook chatbot.created carries it as data.object.id
In the dashboard, it is the number in a chatbot's address: /dashboard/chatbots/{id}.
Website crawler id<CRAWLER_ID>: Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Get it from
POST /v1/crawlers returns it as id
GET /v1/crawlers returns it as data[].id
GET /v1/crawlers/queue returns it as data[].id
Webhook crawler.created carries it as data.object.id
crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.
Sitemap crawler id<SITEMAP_CRAWLER_ID>: Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Get it from
POST /v1/sitemap_crawlers returns it as id
GET /v1/sitemap_crawlers returns it as data[].id
Webhook crawler.created carries it as data.object.id
crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.
Crawler id<CRAWLER_ID>: Integer: a website crawler id with crawler_type: url, a sitemap crawler id with crawler_type: sitemap.
Get it from
GET /v1/crawlers/queue returns it as data[].id
POST /v1/crawlers returns it as id
POST /v1/sitemap_crawlers returns it as id
GET /v1/sitemap_crawlers returns it as data[].id

/v1/crawlers/queue

List queued and running website crawlers

Returns the account's website crawlers that are pending or running, newest first and cursor-paginated — including a recurring crawler waiting for its next run. Paused, finished and sitemap crawlers are not listed. Cancel one with POST /crawlers/queue/cancel or POST /crawlers/{id}/cancel.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Query parameters

limitintegerOptional
Page size, clamped to 1–100.
Default: 20Minimum: 1Maximum: 100
starting_afterstringOptional
Cursor from a previous page's next_cursor: returns the page after it. A cursor minted for another list, sort or order is 422 INVALID_CURSOR.
ending_beforestringOptional
Cursor from a previous page's previous_cursor: returns the page before it. Wins when both cursors are sent.
created[gte]stringOptional
Only rows created on or after this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.
created[lte]stringOptional
Only rows created on or before this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.

Responses

200
Success.
objectstringRequired
Alwayslist
dataarray of CrawlerResourceRequired
has_morebooleanRequired
next_cursorstringnullableRequired
previous_cursorstringnullableRequired
urlstringRequired
Always/v1/crawlers/queue
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
422
Validation failed. See error.details.
Error codesINVALID_CURSOR INVALID_PARAMETER
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/crawlers/queue' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "list",
  "data": [
    {
      "object": "website_crawler",
      "id": 1,
      "name": "Product catalogue",
      "url": "https://shop.example.com/products",
      "status": "running",
      "chatbot_id": 1,
      "error_code": null,
      "created_at": "2026-01-01T00:00:00+00:00",
      "updated_at": "2026-01-01T00:00:00+00:00"
    }
  ],
  "has_more": false,
  "next_cursor": null,
  "previous_cursor": null,
  "url": "/v1/crawlers/queue"
}

/v1/crawlers

List website crawlers

Returns the account's website crawlers, newest first and cursor-paginated; status narrows the list (an unknown value is 422 VALIDATION_FAILED). Sitemap crawlers are listed by GET /sitemap_crawlers.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Query parameters

statusstringnullableOptional
Crawl status: pending, running, paused, completed, failed or cancelled.
Possible valuespendingrunningpausedcompletedfailedcancellednull
limitintegerOptional
Page size, clamped to 1–100.
Default: 20Minimum: 1Maximum: 100
starting_afterstringOptional
Cursor from a previous page's next_cursor: returns the page after it. A cursor minted for another list, sort or order is 422 INVALID_CURSOR.
ending_beforestringOptional
Cursor from a previous page's previous_cursor: returns the page before it. Wins when both cursors are sent.
created[gte]stringOptional
Only rows created on or after this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.
created[lte]stringOptional
Only rows created on or before this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.

Responses

200
Success.
objectstringRequired
Alwayslist
dataarray of CrawlerResourceRequired
has_morebooleanRequired
next_cursorstringnullableRequired
previous_cursorstringnullableRequired
urlstringRequired
Always/v1/crawlers
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
422
Validation failed. See error.details.
Error codesINVALID_CURSOR INVALID_PARAMETER VALIDATION_FAILED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/crawlers' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "list",
  "data": [
    {
      "object": "website_crawler",
      "id": 1,
      "name": "Product catalogue",
      "url": "https://shop.example.com/products",
      "status": "running",
      "chatbot_id": 1,
      "error_code": null,
      "created_at": "2026-01-01T00:00:00+00:00",
      "updated_at": "2026-01-01T00:00:00+00:00"
    },
    {
      "object": "website_crawler",
      "id": 1,
      "name": "Help centre",
      "url": "https://shop.example.com/help",
      "status": "completed",
      "chatbot_id": 1,
      "error_code": null,
      "created_at": "2026-01-01T00:00:00+00:00",
      "updated_at": "2026-01-01T00:00:00+00:00"
    }
  ],
  "has_more": false,
  "next_cursor": null,
  "previous_cursor": null,
  "url": "/v1/crawlers"
}

/v1/crawlers

Create a website crawler

Creates a crawler that follows links from base_url and saves each page it collects as a knowledge dataset in knowledge_collection_id (or the chatbot's default collection), and returns it with 201, status: pending. The first run starts on its own within seconds; crawl.completed or crawl.failed tells you when it ends, and with auto_train the collected pages are trained into chatbot_id, each dataset emitting its own knowledge.dataset.trained or .failed. Before anything is created the URL is checked (only public http(s) addresses) and probed: a blocked address or a site that does not answer is 422 VALIDATION_FAILED. A chatbot already at its plan's knowledge allowance is 402 KNOWLEDGE_PLAN_SIZE_EXCEEDED; an account already running its maximum of concurrent website crawls (5 by default) is 429 CRAWLER_LIMIT_REACHED. When the same URL already has a pending or running crawler for the same chatbot, that crawler is returned with 200 and no event, instead of a second one.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: crawler.created
Rate limits

Before you call

knowledge_collection_id(body)Knowledge collection id
Integer.
Placeholder in the samples: <COLLECTION_ID>
Get it from
- POST /v1/knowledge/collections returns it as id
- GET /v1/knowledge/collections returns it as data[].id
- POST /v1/knowledge/collections/auto_create returns it as id
- GET /v1/knowledge/{id} returns it as collection_id
- Webhook knowledge.collection.created carries it as data.object.id
- Adding knowledge or a crawl with only chatbot_id files it in the chatbot's default collection, created on first use: read its id as collection_id on any of the chatbot's datasets.
chatbot_id(body)Chatbot id
Integer.
Placeholder in the samples: <CHATBOT_ID>
Get it from
- GET /v1/chatbots returns it as data[].id
- POST /v1/chatbots returns it as id
- GET /v1/chatbots/by_handle/{handle} returns it as id
- Webhook chatbot.created carries it as data.object.id
- In the dashboard, it is the number in a chatbot's address: /dashboard/chatbots/{id}.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Request body

Public /v1 start website crawling payload. Field set is pinned on this class so a dashboard FormRequest tweak cannot silently widen the public contract.

namestringRequired
Display name, at most 255 characters.
Max length: 255
descriptionstringnullableOptional
Optional longer description, at most 1000 characters.
Max length: 1000
base_urlstringRequired
The public http(s) page the crawl starts from, at most 2048 characters. Checked before anything is fetched — a private, loopback, link-local or otherwise blocked address is 422 VALIDATION_FAILED on this field — then probed once: a site that does not answer is 422 VALIDATION_FAILED. POST /crawlers/validate_url runs the same probe without creating anything.
Format: uriMax length: 2048
crawl_modestringnullableOptional
website (default) follows links from base_url; single_page collects base_url alone (max_depth and max_pages are then 1).
Possible valuessingle_pagewebsitenull
max_depthintegernullableOptional
How many links deep to follow from base_url: 1 to 10 by default. Required unless crawl_mode is single_page.
Minimum: 1Maximum: 10
max_pagesintegernullableOptional
The most pages to collect: 1 to 100,000 by default. Required unless crawl_mode is single_page.
Minimum: 1Maximum: 100000
crawl_delay_secondsintegernullableOptional
Seconds between requests to the site: 1 (default) to 60.
Minimum: 1Maximum: 60
crawl_concurrencyintegernullableOptional
Pages fetched at once: 1 (default, one at a time) to 5.
Minimum: 1Maximum: 5
follow_external_linksbooleanOptional
Whether to follow links to other hosts. Defaults to false.
respect_robots_txtbooleanOptional
Whether to obey the site's robots.txt (default true). A page it disallows is left out with error_code: robots_blocked.
include_patternsarray of stringnullableOptional
Only collect URLs matching one of these patterns (a glob or a plain substring, each at most 255 characters).
exclude_patternsarray of stringnullableOptional
Never collect URLs matching one of these patterns (a glob or a plain substring, each at most 255 characters), for example a tag or category archive.
content_selectorsobjectOptional
CSS selectors that pick each page's content. Omit for the platform defaults.
knowledge_collection_idintegernullableOptional
The collection to file the collected pages in. Optional with chatbot_id, when the chatbot's default collection is used (created on first use); required without it.
chatbot_idintegernullableOptional
The chatbot the collected pages train. A crawler of the same base_url for the same chatbot that is still pending or running is returned instead of a new one.
auto_trainbooleannullableOptional
Train each collected page into chatbot_id (default true).
crawl_frequencystringnullableOptional
once (default), or re-crawl the site daily, weekly or monthly.
Possible valuesoncedailyweeklymonthlynull
enable_javascriptbooleanOptional
JavaScript rendering is not available: true is 422 VALIDATION_FAILED.
user_agentstringnullableOptional
The User-Agent to send, at most 500 characters. Defaults to the platform crawler's.
Max length: 500
training_optionsobjectOptional
How collected pages become knowledge. Omitted keys take the platform defaults.

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
201
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
402
A billing gate refused the request (plan limit, monthly AI budget or message credits). See error.code.
Error codesKNOWLEDGE_PLAN_SIZE_EXCEEDED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesIDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesBLOCKED_DOMAIN CHALLENGE_BLOCKED CONTENT_TOO_LARGE CORRUPT_CONTENT_ENCODING CRAWLING_ERROR IDEMPOTENCY_KEY_REUSED INVALID_SITEMAP ROBOTS_BLOCKED UNSUPPORTED_CONTENT_ENCODING UNSUPPORTED_CONTENT_TYPE UPSTREAM_HTTP_ERROR VALIDATION_FAILED
429
Rate limited. Honour Retry-After.
Error codesCRAWLER_LIMIT_REACHED RATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/crawlers' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "description": "description",
  "base_url": "base_url",
  "crawl_mode": "crawl_mode",
  "max_depth": "max_depth",
  "max_pages": "max_pages",
  "crawl_delay_seconds": "crawl_delay_seconds",
  "crawl_concurrency": "crawl_concurrency"
}'

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 0,
  "name": "string",
  "url": "string",
  "status": "pending",
  "chatbot_id": "string",
  "error_code": "unreachable",
  "created_at": "string",
  "updated_at": "string"
}

Built from the response schema, not recorded from a live call.

/v1/crawlers/{id}

Retrieve a website crawler

Returns one website crawler: its start url, its status and, after a failure, an error_code. Poll it while a crawl runs, or subscribe to crawl.completed and crawl.failed. The pages it collected are listed by GET /crawlers/{id}/pages.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "name": "Help centre",
  "url": "https://shop.example.com/help",
  "status": "completed",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/crawlers/{id}

Update a website crawler

Changes the crawler's name, max_pages or crawl_delay_seconds for its next run, and emits crawler.updated. Nothing is re-crawled until you call POST /crawlers/{id}/retry or the next scheduled run. A crawler that is running is 409 CRAWLER_RUNNING. Other settings (start URL, patterns, chatbot) are fixed: create a new crawler to change them.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: crawler.updated
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Request body

Body for PATCH /v1/crawlers/{id} (website crawlers). is_active is deliberately NOT accepted (HI-22): flipping it through an edit left a pending row the jobs silently skip. Deactivation is the DELETE / cancel transition. Ceilings come from config/crawling.php so the public contract cannot drift from the dashboard's (ME-27).

namestringOptional
Display name, at most 255 characters.
Max length: 255
max_pagesintegerOptional
The most pages to collect: 1 to 100,000 by default. Applies from the next run.
Minimum: 1Maximum: 100000
crawl_delay_secondsintegerOptional
Seconds between requests to the site: 1 to 60 by default. Applies from the next run.
Minimum: 1Maximum: 60

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_RUNNING IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED VALIDATION_FAILED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X PATCH 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "max_pages": 1,
  "crawl_delay_seconds": 1
}'

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "name": "Help centre (EN)",
  "url": "https://example.com/help",
  "status": "pending",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/crawlers/{id}

Delete a website crawler

Deletes the crawler, cancelling it first when it has not finished, and emits crawler.deleted. The pages it already saved stay as knowledge datasets: remove them with POST /knowledge/remove_from_system if you no longer want them.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: crawler.deleted
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
Success.
objectstringRequired
Alwayswebsite_crawler
idintegerRequired
The crawler id.
deletedbooleanRequired
Always true.
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesIDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X DELETE 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "deleted": true
}

/v1/crawlers/{id}/pages

List a website crawler's pages

Returns every page the crawler discovered, newest first and cursor-paginated, with what became of it: processed pages are knowledge datasets now, failed ones carry the error_code that explains why (unreachable, blocked by robots.txt, a duplicate of another page, too short, …). The datasets themselves are listed by GET /chatbots/{id}/knowledge.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Query parameters

limitintegerOptional
Page size, clamped to 1–100.
Default: 20Minimum: 1Maximum: 100
starting_afterstringOptional
Cursor from a previous page's next_cursor: returns the page after it. A cursor minted for another list, sort or order is 422 INVALID_CURSOR.
ending_beforestringOptional
Cursor from a previous page's previous_cursor: returns the page before it. Wins when both cursors are sent.
created[gte]stringOptional
Only rows created on or after this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.
created[lte]stringOptional
Only rows created on or before this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.

Responses

200
Success.
objectstringRequired
Alwayslist
dataarray of CrawledPageResourceRequired
has_morebooleanRequired
next_cursorstringnullableRequired
previous_cursorstringnullableRequired
urlstringRequired
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
422
Validation failed. See error.details.
Error codesINVALID_CURSOR INVALID_PARAMETER
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>/pages' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "list",
  "data": [
    {
      "object": "crawled_page",
      "id": 1,
      "url": "https://shop.example.com/help/refunds",
      "status": "processed",
      "error_code": null
    }
  ],
  "has_more": false,
  "next_cursor": null,
  "previous_cursor": null,
  "url": "/v1/crawlers/1/pages"
}

/v1/crawlers/{id}/statistics

Check a website crawler's outcome

A light read of the crawler's status and failure error_code, without the rest of the crawler. It carries no page counts: count the pages of GET /crawlers/{id}/pages by status for those.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Responses

200
Success.
objectstringRequired
Alwayscrawler_statistics
idintegerRequired
The crawler id.
statusstringRequired
pending, running, paused, completed, failed or cancelled.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>/statistics' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "crawler_statistics",
  "id": 1,
  "status": "completed",
  "error_code": null
}

/v1/sitemap_crawlers

List sitemap crawlers

Returns the account's sitemap crawlers, newest first and cursor-paginated. Website crawlers are listed by GET /crawlers.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Query parameters

limitintegerOptional
Page size, clamped to 1–100.
Default: 20Minimum: 1Maximum: 100
starting_afterstringOptional
Cursor from a previous page's next_cursor: returns the page after it. A cursor minted for another list, sort or order is 422 INVALID_CURSOR.
ending_beforestringOptional
Cursor from a previous page's previous_cursor: returns the page before it. Wins when both cursors are sent.
created[gte]stringOptional
Only rows created on or after this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.
created[lte]stringOptional
Only rows created on or before this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.

Responses

200
Success.
objectstringRequired
Alwayslist
dataarray of CrawlerResourceRequired
has_morebooleanRequired
next_cursorstringnullableRequired
previous_cursorstringnullableRequired
urlstringRequired
Always/v1/sitemap_crawlers
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
422
Validation failed. See error.details.
Error codesINVALID_CURSOR INVALID_PARAMETER
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/sitemap_crawlers' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "list",
  "data": [
    {
      "object": "sitemap_crawler",
      "id": 1,
      "name": "Shop sitemap",
      "url": "https://shop.example.com/sitemap.xml",
      "status": "completed",
      "chatbot_id": 1,
      "error_code": null,
      "created_at": "2026-01-01T00:00:00+00:00",
      "updated_at": "2026-01-01T00:00:00+00:00"
    }
  ],
  "has_more": false,
  "next_cursor": null,
  "previous_cursor": null,
  "url": "/v1/sitemap_crawlers"
}

/v1/sitemap_crawlers

Create a sitemap crawler

Creates a crawler that fetches the pages sitemap_url lists (following a sitemap index into its sitemaps) and saves each as a knowledge dataset in knowledge_collection_id (or the chatbot's default collection), and returns it with 201, status: pending. The first scan starts on its own within seconds — there is no need to call POST /sitemap_crawlers/{id}/scan — and later scans follow scan_frequency; a page whose content has not changed since the last scan is marked unchanged and not retrained. With auto_train the collected pages are trained into chatbot_id, each dataset emitting its own knowledge.dataset.trained or .failed. Before anything is created the URL is checked (only public http(s) addresses) and read: a blocked address or a URL that is not a usable sitemap is 422 VALIDATION_FAILED (details.errors says why; POST /sitemap_crawlers/validate_url runs the same check alone). A chatbot already at its plan's knowledge allowance is 402 KNOWLEDGE_PLAN_SIZE_EXCEEDED; an account already running its maximum of concurrent sitemap crawls (10 by default) is 429 CRAWLER_LIMIT_REACHED. When the same sitemap already has a pending or running crawler for the same chatbot, that crawler is returned with 200 and no event, instead of a second one.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: crawler.created
Rate limits

Before you call

chatbot_id(body)Chatbot id
Integer.
Placeholder in the samples: <CHATBOT_ID>
Get it from
- GET /v1/chatbots returns it as data[].id
- POST /v1/chatbots returns it as id
- GET /v1/chatbots/by_handle/{handle} returns it as id
- Webhook chatbot.created carries it as data.object.id
- In the dashboard, it is the number in a chatbot's address: /dashboard/chatbots/{id}.
knowledge_collection_id(body)Knowledge collection id
Integer.
Placeholder in the samples: <COLLECTION_ID>
Get it from
- POST /v1/knowledge/collections returns it as id
- GET /v1/knowledge/collections returns it as data[].id
- POST /v1/knowledge/collections/auto_create returns it as id
- GET /v1/knowledge/{id} returns it as collection_id
- Webhook knowledge.collection.created carries it as data.object.id
- Adding knowledge or a crawl with only chatbot_id files it in the chatbot's default collection, created on first use: read its id as collection_id on any of the chatbot's datasets.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Request body

Public /v1 start sitemap crawling payload. Field set is pinned on this class so a dashboard FormRequest tweak cannot silently widen the public contract. Authorization is inherited: the SitemapCrawlPolicy::create gate (HI-15) applies to API-key callers too, so a member-bound key without knowledge.manage cannot start crawls the dashboard would refuse.

namestringRequired
Display name, at most 255 characters.
Max length: 255
sitemap_urlstringRequired
The public http(s) URL of a sitemap.xml or sitemap index, at most 2048 characters. Checked before it is fetched — a private, loopback, link-local or otherwise blocked address is 422 VALIDATION_FAILED on this field — then read once: a URL that is not a usable sitemap is 422 VALIDATION_FAILED with the reasons in details.errors.
Format: uriMax length: 2048
chatbot_idintegernullableOptional
The chatbot the collected pages train. A sitemap crawler of the same URL for the same chatbot that is still pending or running is returned instead of a new one.
knowledge_collection_idintegernullableOptional
The collection to file the collected pages in. Optional with chatbot_id, when the chatbot's default collection is used (created on first use); required without it.
auto_trainbooleannullableOptional
Train each collected page into chatbot_id (default true).
urls_per_scanintegernullableOptional
Sitemap URLs fetched per batch of a scan: 1 to 1000, default 500.
Minimum: 1Maximum: 1000
max_total_urlsintegernullableOptional
The most sitemap URLs to collect: 1 to 100,000 by default. Omit it for no cap of its own.
Minimum: 1Maximum: 100000
crawl_delay_secondsintegernullableOptional
Seconds between requests to the site: 0 to 60, default 1.
Minimum: 0Maximum: 60
respect_robots_txtbooleannullableOptional
Whether to obey the site's robots.txt (default true). A URL it disallows is skipped with error_code: robots_blocked.
user_agentstringnullableOptional
The User-Agent to send, at most 255 characters. Defaults to the platform crawler's.
Max length: 255
scan_frequencystringnullableOptional
How often the sitemap is scanned again for new and changed pages: hourly (default), daily, weekly, or manual (only when you call POST /sitemap_crawlers/{id}/scan).
Possible valueshourlydailyweeklymanualnull
content_selectorsobjectOptional
CSS selectors that pick each page's content. Omit for the platform defaults.
exclude_patternsarray of stringnullableOptional
Never collect URLs matching one of these patterns (a glob or a plain substring), at most 50 of 255 characters each — for example a tag or category archive.
Max items: 50
training_optionsobjectOptional
How collected pages become knowledge. Omitted keys take the platform defaults.

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
201
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
402
A billing gate refused the request (plan limit, monthly AI budget or message credits). See error.code.
Error codesKNOWLEDGE_PLAN_SIZE_EXCEEDED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesIDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesBLOCKED_DOMAIN CHALLENGE_BLOCKED CONTENT_TOO_LARGE CORRUPT_CONTENT_ENCODING CRAWLING_ERROR IDEMPOTENCY_KEY_REUSED INVALID_SITEMAP ROBOTS_BLOCKED UNSUPPORTED_CONTENT_ENCODING UNSUPPORTED_CONTENT_TYPE UPSTREAM_HTTP_ERROR VALIDATION_FAILED
429
Rate limited. Honour Retry-After.
Error codesCRAWLER_LIMIT_REACHED RATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/sitemap_crawlers' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "sitemap_url": "sitemap_url",
  "chatbot_id": 1,
  "knowledge_collection_id": 1,
  "auto_train": "auto_train",
  "urls_per_scan": "urls_per_scan",
  "max_total_urls": "max_total_urls",
  "crawl_delay_seconds": "crawl_delay_seconds"
}'

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 0,
  "name": "string",
  "url": "string",
  "status": "pending",
  "chatbot_id": "string",
  "error_code": "unreachable",
  "created_at": "string",
  "updated_at": "string"
}

Built from the response schema, not recorded from a live call.

/v1/sitemap_crawlers/{id}

Retrieve a sitemap crawler

Returns one sitemap crawler: its sitemap url, its status and, after a failure, an error_code. A recurring crawler reads pending while it waits for its next scan. Each URL the sitemap lists is reported by GET /sitemap_crawlers/{id}/pages.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 1,
  "name": "Shop sitemap",
  "url": "https://shop.example.com/sitemap.xml",
  "status": "completed",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/sitemap_crawlers/{id}

Update a sitemap crawler

Changes the crawler's name, urls_per_scan, max_total_urls, crawl_delay_seconds or scan_frequency for its next scan, and emits crawler.updated. A crawler with a scan running is 409 CRAWLER_RUNNING. The sitemap URL, patterns and chatbot are fixed: create a new crawler to change them.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: crawler.updated
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Request body

Body for PATCH /v1/sitemap_crawlers/{id}. Split from {@see UpdateCrawlerRequest} (ME-27): that request accepted max_pages, which is not a sitemap_crawls column — a MassAssignmentException off production and a silent no-op on it. The field set here is the sitemap crawl's own tunables, with ceilings read from config/crawling.php like the dashboard StartSitemapCrawlingRequest. is_active is not accepted (HI-22) — deactivation is the DELETE transition.

namestringOptional
Display name, at most 255 characters.
Max length: 255
urls_per_scanintegerOptional
Sitemap URLs fetched per batch of a scan: 1 to 1000.
Minimum: 1Maximum: 1000
max_total_urlsintegernullableOptional
The most sitemap URLs to collect: 1 to 100,000 by default; null removes the cap of its own.
Minimum: 1Maximum: 100000
crawl_delay_secondsintegerOptional
Seconds between requests to the site: 0 to 60 by default.
Minimum: 0Maximum: 60
scan_frequencystringOptional
How often the sitemap is scanned again: hourly, daily, weekly or manual (only when you call POST /sitemap_crawlers/{id}/scan).
Possible valueshourlydailyweeklymanual

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_RUNNING IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED VALIDATION_FAILED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X PATCH 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "name": "name",
  "urls_per_scan": 1,
  "max_total_urls": "max_total_urls",
  "crawl_delay_seconds": 1,
  "scan_frequency": "scan_frequency"
}'

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 1,
  "name": "Product pages",
  "url": "https://example.com/sitemap.xml",
  "status": "pending",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/sitemap_crawlers/{id}

Delete a sitemap crawler

Deletes the crawler, cancelling it first when it has not finished, and emits crawler.deleted. No further scans run. The pages it already saved stay as knowledge datasets: remove them with POST /knowledge/remove_from_system if you no longer want them.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: crawler.deleted
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
Success.
objectstringRequired
Alwayssitemap_crawler
idintegerRequired
The crawler id.
deletedbooleanRequired
Always true.
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesIDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X DELETE 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 1,
  "deleted": true
}

/v1/sitemap_crawlers/{id}/pages

List a sitemap crawler's URLs

Returns one row per URL the sitemap lists, newest first and cursor-paginated, with what the last scan did with it: crawled or updated pages are knowledge datasets, unchanged ones were re-fetched with the same content, skipped and failed ones carry the error_code that explains why (excluded by a pattern, blocked by robots.txt, a duplicate, unreachable, …).

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Query parameters

limitintegerOptional
Page size, clamped to 1–100.
Default: 20Minimum: 1Maximum: 100
starting_afterstringOptional
Cursor from a previous page's next_cursor: returns the page after it. A cursor minted for another list, sort or order is 422 INVALID_CURSOR.
ending_beforestringOptional
Cursor from a previous page's previous_cursor: returns the page before it. Wins when both cursors are sent.
created[gte]stringOptional
Only rows created on or after this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.
created[lte]stringOptional
Only rows created on or before this instant: YYYY-MM-DD (whole UTC day) or an RFC 3339 timestamp with a zone. Anything else is 422 INVALID_PARAMETER.

Responses

200
Success.
objectstringRequired
Alwayslist
dataarray of SitemapUrlResourceRequired
has_morebooleanRequired
next_cursorstringnullableRequired
previous_cursorstringnullableRequired
urlstringRequired
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
422
Validation failed. See error.details.
Error codesINVALID_CURSOR INVALID_PARAMETER
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>/pages' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "list",
  "data": [
    {
      "object": "sitemap_url",
      "id": 1,
      "url": "https://shop.example.com/shipping",
      "status": "crawled",
      "error_code": null
    }
  ],
  "has_more": false,
  "next_cursor": null,
  "previous_cursor": null,
  "url": "/v1/sitemap_crawlers/1/pages"
}

/v1/sitemap_crawlers/{id}/statistics

Check a sitemap crawler's outcome

A light read of the crawler's status and failure error_code, without the rest of the crawler. It carries no page counts: count the rows of GET /sitemap_crawlers/{id}/pages by status for those.

Required scope: crawlers:read
Idempotency: Read-only
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Responses

200
Success.
objectstringRequired
Alwayscrawler_statistics
idintegerRequired
The crawler id.
statusstringRequired
pending, running, paused, completed, failed or cancelled.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X GET 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>/statistics' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "crawler_statistics",
  "id": 1,
  "status": "completed",
  "error_code": null
}

/v1/crawlers/validate_url

Check a URL before crawling it

Fetches the URL once, the way POST /crawlers probes base_url, and reports whether it answered (accessible, the HTTP status_code, or an error_code). Use it to tell your user "that site is unreachable" before creating a crawler. It creates nothing, so it needs only crawlers:read, and it has a tight rate limit of its own because each call is a live fetch. A blocked address is 422 on url and never fetched.

Required scope: crawlers:read
Webhook events: None
Rate limits

Request body

urlstringRequired
The public http(s) URL to probe, at most 2048 characters. A private, loopback, link-local or otherwise blocked address is 422 VALIDATION_FAILED on this field and is never fetched.
Format: uriMax length: 2048

Responses

200
Success.
objectstringRequired
Alwaysurl_validation
urlanyRequired
The URL as you sent it.
accessiblebooleanRequired
Whether the URL answered with a page a crawler could start from.
status_codeintegernullableRequired
The HTTP status the URL answered with; null when it could not be reached.
error_codestringnullableRequired
Why the URL could not be reached, or null when the probe succeeded.
Possible valuesnullhttp_errorunreachable
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
422
Validation failed. See error.details.
Error codesVALIDATION_FAILED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/crawlers/validate_url' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "url": "url"
}'

Response

Example 200 response

json

{
  "object": "url_validation",
  "url": "https://example.com/help",
  "accessible": true,
  "status_code": 200,
  "error_code": null
}

/v1/sitemap_crawlers/validate_url

Validate a sitemap URL

Fetches and parses a sitemap before you commit to crawling it, so you can tell your user "that URL is not a sitemap" instead of creating a crawler that fails at once — POST /sitemap_crawlers runs the same check. It creates nothing, so it needs only crawlers:read, and it has a tight rate limit of its own because each call is a live fetch. A blocked address is 422 on sitemap_url and never fetched.

Required scope: crawlers:read
Webhook events: None
Rate limits

Request body

sitemap_urlstringRequired
The public http(s) URL of a sitemap.xml or sitemap index, at most 2048 characters. A private, loopback, link-local or otherwise blocked address is 422 VALIDATION_FAILED on this field and is never fetched.
Format: uriMax length: 2048

Responses

200
Success.
objectstringRequired
Alwayssitemap_validation
urlstringRequired
The sitemap URL as you sent it.
validbooleanRequired
Whether the URL is a sitemap a crawler can use.
typeanyRequired
What the document is: sitemap (a list of pages), sitemap_index (a list of sitemaps) or invalid; null when it could not be fetched.
url_countintegerRequired
How many entries the document lists.
has_lastmodbooleanRequired
Whether entries carry <lastmod> dates, which let later scans skip unchanged pages.
errorsarray of anyRequired
Why the sitemap cannot be used, as readable sentences; empty when valid.
warningsarray of anyRequired
Problems that do not stop a crawl, as readable sentences.
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
422
Validation failed. See error.details.
Error codesVALIDATION_FAILED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/sitemap_crawlers/validate_url' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "sitemap_url": "sitemap_url"
}'

Response

Example 200 response

json

{
  "object": "sitemap_validation",
  "url": "https://example.com/sitemap.xml",
  "valid": true,
  "type": "sitemap",
  "url_count": 1,
  "has_lastmod": true,
  "errors": [],
  "warnings": []
}

/v1/crawlers/queue/cancel

Cancel a crawler of either kind

Cancels one website or sitemap crawler named by crawler_id and crawler_type, exactly as POST /crawlers/{id}/cancel does, and reports its new status. It answers a short status object rather than the crawler. One that already finished is 409 CRAWLER_TERMINAL. Emits no event.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

crawler_id(body)Crawler id
Integer: a website crawler id with crawler_type: url, a sitemap crawler id with crawler_type: sitemap.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- GET /v1/crawlers/queue returns it as data[].id
- POST /v1/crawlers returns it as id
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Request body

crawler_idintegerRequired
The crawler to cancel: a website crawler id with crawler_type: url, a sitemap crawler id with crawler_type: sitemap.
Minimum: 1
crawler_typestringRequired
Which kind crawler_id names: url (a website crawler) or sitemap. Website and sitemap crawlers are numbered separately, so the same id can name one of each.
Possible valuesurlsitemap

Responses

200
Success.
objectstringRequired
The crawler kind: website_crawler or sitemap_crawler.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id.
deletedbooleanRequired
Always false: the crawler is cancelled, not deleted.
statusstringRequired
The crawler's status after the call: cancelled.
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_TERMINAL IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED VALIDATION_FAILED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/crawlers/queue/cancel' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"
  -d '{
  "crawler_id": 1,
  "crawler_type": "crawler_type"
}'

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "deleted": false,
  "status": "cancelled"
}

/v1/crawlers/{id}/pause

Pause a website crawler

Halts a crawler that is running or about to run and returns it as paused; the pages saved so far stay. A paused crawler holds no concurrent-crawl slot. Continue with POST /crawlers/{id}/resume. Any other state is 409 CRAWLER_NOT_RUNNING. Emits no event.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_NOT_RUNNING IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>/pause' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "name": "Help centre (EN)",
  "url": "https://example.com/help",
  "status": "paused",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/crawlers/{id}/resume

Resume a website crawler

Continues a paused crawler and returns it. Resuming takes a concurrent-crawl slot again: at the account's cap it is 429 CRAWLER_LIMIT_REACHED and the crawler stays paused. A crawler that is not paused (cancelled ones included) is 409 CRAWLER_NOT_PAUSED; a paused crawler that was deactivated is 409 CRAWLER_INACTIVE. Emits no event.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_INACTIVE CRAWLER_NOT_PAUSED IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour Retry-After.
Error codesCRAWLER_LIMIT_REACHED RATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>/resume' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "name": "Help centre (EN)",
  "url": "https://example.com/help",
  "status": "pending",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/crawlers/{id}/retry

Re-run a website crawler

Runs a crawler that is not running again now — after a failure, or to pick up changes on the site — and returns it as pending. Pages it saved before are updated in place, not duplicated; the run ends in crawl.completed or crawl.failed. A crawler still running is 409 CRAWLER_RUNNING; a cancelled or deactivated one is 409 CRAWLER_INACTIVE (create a new crawler); a paused one is 409 CRAWL_NOT_RETRYABLE (resume it instead). The run takes a concurrent-crawl slot, so at the account's cap it is 429 CRAWLER_LIMIT_REACHED and the crawler keeps its state.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_INACTIVE CRAWLER_RUNNING CRAWL_NOT_RETRYABLE IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour Retry-After.
Error codesCRAWLER_LIMIT_REACHED RATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>/retry' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "name": "Blog",
  "url": "https://example.com/blog",
  "status": "pending",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/crawlers/{id}/cancel

Cancel a website crawler

Stops a crawler that has not finished, for good, and returns it as cancelled; the pages saved so far stay as knowledge. A cancelled crawler cannot be resumed or re-run. One that already finished (completed, failed or cancelled) is 409 CRAWLER_TERMINAL. Emits no event.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Website crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <CRAWLER_ID>
Get it from
- POST /v1/crawlers returns it as id
- GET /v1/crawlers returns it as data[].id
- GET /v1/crawlers/queue returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The website crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_TERMINAL IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/crawlers/<CRAWLER_ID>/cancel' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "website_crawler",
  "id": 1,
  "name": "Help centre (EN)",
  "url": "https://example.com/help",
  "status": "cancelled",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/sitemap_crawlers/{id}/pause

Pause a sitemap crawler

Halts a crawler whose scan is running or about to run and returns it as paused; the pages saved so far stay, and no scheduled scan starts while it is paused. A paused crawler holds no concurrent-crawl slot. Continue with POST /sitemap_crawlers/{id}/resume. Any other state — a recurring crawler waiting for its next scan included — is 409 CRAWLER_NOT_RUNNING. Emits no event.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_NOT_RUNNING IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour the Retry-After header before retrying.
Error codesRATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>/pause' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 1,
  "name": "Product pages",
  "url": "https://example.com/sitemap.xml",
  "status": "paused",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/sitemap_crawlers/{id}/resume

Resume a sitemap crawler

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_INACTIVE CRAWLER_NOT_PAUSED IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour Retry-After.
Error codesCRAWLER_LIMIT_REACHED RATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>/resume' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 1,
  "name": "Product pages",
  "url": "https://example.com/sitemap.xml",
  "status": "pending",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/sitemap_crawlers/{id}/scan

Scan a sitemap now

Starts a scan of the sitemap within seconds instead of waiting for scan_frequency — the way to refresh a manual crawler — and returns the crawler as pending. The scan picks up new URLs and re-fetches listed pages; unchanged ones are not retrained. It ends in crawl.completed or crawl.failed. A crawler with a scan running is 409 CRAWLER_RUNNING; a cancelled or deactivated one is 409 CRAWLER_INACTIVE; a paused one is 409 CRAWL_NOT_RETRYABLE (resume it instead). The scan takes a concurrent-crawl slot, so at the account's cap it is 429 CRAWLER_LIMIT_REACHED and the crawler keeps its state.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_INACTIVE CRAWLER_RUNNING CRAWL_NOT_RETRYABLE IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour Retry-After.
Error codesCRAWLER_LIMIT_REACHED RATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>/scan' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 1,
  "name": "Product pages",
  "url": "https://example.com/sitemap.xml",
  "status": "pending",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

/v1/sitemap_crawlers/{id}/retry

Re-run a sitemap crawler

Runs a sitemap crawler that is not running again at once — typically after failed — and returns it as pending. It is POST /sitemap_crawlers/{id}/scan without the few seconds' delay, with the same refusals: a crawler with a scan running is 409 CRAWLER_RUNNING, a cancelled or deactivated one 409 CRAWLER_INACTIVE, a paused one 409 CRAWL_NOT_RETRYABLE (resume it instead), and at the account's cap 429 CRAWLER_LIMIT_REACHED with the crawler left as it was.

Required scope: crawlers:write
Idempotency: Accepts Idempotency-Key
Webhook events: None
Rate limits

Before you call

id(path)Sitemap crawler id
Integer. Website and sitemap crawlers are numbered separately, so the same number can name one of each.
Placeholder in the samples: <SITEMAP_CRAWLER_ID>
Get it from
- POST /v1/sitemap_crawlers returns it as id
- GET /v1/sitemap_crawlers returns it as data[].id
- Webhook crawler.created carries it as data.object.id
- crawler.* and crawl.* events carry only {object, id}, not the crawler kind: keep the kind with the id when you create the crawler.

Path parameters

idintegerRequired
The sitemap crawler id.

Header parameters

Idempotency-KeystringOptional
Makes a retry safe. The first 2xx answer is stored for 24 h and replayed for the same key with Idempotency-Replayed: true; a retry while the first request runs is 409 IDEMPOTENCY_IN_FLIGHT; the same key with a different payload is 422 IDEMPOTENCY_KEY_REUSED; a malformed key is 400 INVALID_IDEMPOTENCY_KEY.
Min length: 16Max length: 128Pattern: ^[A-Za-z0-9_-]+$

Responses

200
CrawlerResource
objectstringRequired
The crawler kind: website_crawler (/crawlers) or sitemap_crawler (/sitemap_crawlers). The two kinds are numbered separately.
Possible valuessitemap_crawlerwebsite_crawler
idintegerRequired
The crawler id, unique within its kind.
namestringnullableRequired
Display name.
urlstringnullableRequired
The start URL of a website crawler, or the sitemap URL of a sitemap crawler.
statusstringRequired
pending (queued, or a recurring crawler waiting for its next run), running, paused, completed, failed or cancelled. Reaching completed or failed emits crawl.completed or crawl.failed.
Possible valuespendingrunningpausedcompletedfailedcancelled
chatbot_idstringnullableRequired
The chatbot the collected pages train; null when the crawler only fills a collection.
error_codestringnullableRequired
Why the crawler last failed, as a stable code; null when it has not.
Possible valuesunreachablehttp_errortoo_many_redirectsinvalid_sitemaptoo_largerobots_blockednever_startedknowledge_limit_reachedunsupported_content_encodingcorrupt_content_encodingunsupported_content_typecontent_too_shortnot_knowledge_indexnot_knowledge_mediaexcluded_by_patternduplicatecanonical_duplicatenear_duplicatesoft_404redirect_externalredirect_other_pagestalledextraction_failedremoved_from_knowledge_basechallenge_blockedunknownnull
created_atstringnullableRequired
When the crawler was created (ISO 8601, UTC).
updated_atstringnullableRequired
When the crawler last changed (ISO 8601, UTC).
400
A tenant selector header named a tenant other than the API key's own (ACCOUNT_SELECTOR_NOT_SUPPORTED): an all-digit X-Account-Id other than the key owner's id or, while workspaces.context_enabled is on, an X-Workspace-Id other than the key's workspace (ws_ prefix optional, case-insensitive). A header naming the key's own tenant, or a non-numeric X-Account-Id, is ignored; a key cannot switch tenants, so send neither. A mutation that honours Idempotency-Key also returns INVALID_IDEMPOTENCY_KEY when that header is not 16–128 [A-Za-z0-9_-] characters.
Error codesACCOUNT_SELECTOR_NOT_SUPPORTED INVALID_IDEMPOTENCY_KEY
401
Missing or invalid API key.
Error codesUNAUTHENTICATED
403
The API key is missing a required scope.
Error codesACCOUNT_SUSPENDED INSUFFICIENT_SCOPE
404
The resource does not exist on this account.
Error codesNOT_FOUND
409
The request conflicts with the current resource state.
Error codesCRAWLER_INACTIVE CRAWLER_RUNNING CRAWL_NOT_RETRYABLE IDEMPOTENCY_IN_FLIGHT
422
Validation failed. See error.details.
Error codesIDEMPOTENCY_KEY_REUSED
429
Rate limited. Honour Retry-After.
Error codesCRAWLER_LIMIT_REACHED RATE_LIMITED
500
Unexpected server error. Retry with the same Idempotency-Key.
Error codesSERVER_ERROR

Request

curl

curl -X POST 'https://api.agentency.com/v1/sitemap_crawlers/<SITEMAP_CRAWLER_ID>/retry' \
  -H "Authorization: Bearer <YOUR_KEY>" \
  -H "Content-Type: application/json"

Response

Example 200 response

json

{
  "object": "sitemap_crawler",
  "id": 1,
  "name": "Product pages",
  "url": "https://example.com/sitemap.xml",
  "status": "pending",
  "chatbot_id": 1,
  "error_code": null,
  "created_at": "2026-01-01T00:00:00+00:00",
  "updated_at": "2026-01-01T00:00:00+00:00"
}

About Crawlers

IDs on this page

/v1/crawlers/queue

Query parameters

Responses

See also

Request

Response

/v1/crawlers

Query parameters

Responses

See also

Request

Response

/v1/crawlers

Header parameters

Request body

Responses

See also

Request

Response

/v1/crawlers/{id}

Path parameters

Responses

See also

Request

Response

/v1/crawlers/{id}

Path parameters

Header parameters

Request body

Responses

See also

Request

Response

/v1/crawlers/{id}

Path parameters

Header parameters

Responses

See also

Request

Response

/v1/crawlers/{id}/pages

Path parameters

Query parameters

Responses

See also

Request

Response

/v1/crawlers/{id}/statistics

Path parameters

Responses

See also

Request

Response

/v1/sitemap_crawlers

Query parameters

Responses

See also

Request

Response

/v1/sitemap_crawlers

Header parameters

Request body

Responses

See also

Request

Response

/v1/sitemap_crawlers/{id}

Path parameters

Responses

See also

Request

Response

/v1/sitemap_crawlers/{id}

Path parameters

Header parameters

Request body

Responses

See also