Train a bot

Each source is a dataset: a name, some text, and the chatbot it belongs to. Where the text came from — pasted, uploaded, or crawled — only matters at creation. After extraction they are all the same thing.

Datasets can be grouped into collections, which is useful once you have more than a handful. A collection is organisational; it does not change how retrieval works.

Two fields decide the type of a text or web-page dataset:

A dataset made from an uploaded file needs neither: send file_id and the file's extension sets the type. Datasets you read back can also carry the source_type of the path that made them — upload, url_links, sitemap, google_drive, dropbox or site_info.

The simplest source, and the best one for content you generate yourself — product data, FAQs assembled from a database, policy text.

Upload first, then reference the returned id. Splitting it in two means a failed upload never leaves you with a half-created dataset: a file whose text cannot be read is refused with 422 TEXT_EXTRACTION_FAILED before any dataset exists, and you still hold the file to fix or delete. A file backs one dataset only — naming it in a second POST /v1/knowledge is 409 FILE_ALREADY_ATTACHED.

Accepted: txt, pdf, doc, docx, rtf, odt, images (png, jpg, jpeg, webp) and spreadsheets (xlsx, xls, csv, ods), at most 10 MB each by default. The content must match the extension.

To import a folder, send them together and let one Idempotency-Key cover the batch — up to 20 files and 90 MB, stored all or nothing. The response is a list rather than a single object:

Scanned PDFs and images go through OCR automatically. That takes longer than text extraction, which is worth knowing before you decide your polling interval.

Give it a URL and the page is fetched, cleaned of navigation and boilerplate, and indexed.

Use POST /v1/knowledge/{id}/recrawl to refetch it later — that is the right call when a page changes, rather than creating a second dataset for the same URL.

For more than a page or two, use a crawler. Two kinds:

• Website crawler — follows links from a starting URL, within limits you set.
• Sitemap crawler — reads sitemap.xml and fetches what it lists. More predictable, and the better choice when a sitemap exists.

Check the sitemap before committing to it. This costs nothing and mutates nothing, so it only needs read access:

type is sitemap for a list of pages and sitemap_index for a list of sitemaps. Then create the crawler: it queues its first scan by itself, and later scans follow its scan_frequency. POST /v1/sitemap_crawlers/{id}/scan runs one on demand — the way to refresh a manual crawler. Crawls run in the background and can take a while on a large site — subscribe to crawl.completed and crawl.failed rather than polling for minutes.

Creating a crawler for a URL and chatbot that already has one pending or running returns that crawler (200) instead of a second one. An account runs a limited number of crawls at once (5 website and 10 sitemap crawls by default); one more is 429 CRAWLER_LIMIT_REACHED.

POST /v1/knowledge/import/cloud pulls documents straight from Google Drive or Dropbox. Your side supplies short-lived references — the Google Picker's access_token and Drive file ids, or the Dropbox Chooser's direct links; the platform downloads each file and feeds it through the same pipeline as an upload. One Idempotency-Key covers the whole batch. Files that could not be imported come back in failures with a reason. Imported datasets emit no knowledge.dataset.created; their knowledge.dataset.trained and knowledge.dataset.failed fire as usual.

Every dataset carries embedding_status:

A completed dataset with total_chunks: 0 is not broken: every passage in it repeated knowledge another trained dataset of the same collection already holds, so training skipped it.

Two endpoints read it:

The first is about one source. The second is about the bot: ready is true once the chatbot is active and its training is completed, alongside how many documents and how much knowledge it holds.

• Edit — PATCH /v1/knowledge/{id} to change the name or metadata, or the content of a text dataset (retrained unless you send retrain: false).
• Replace the file — POST /v1/knowledge/{id}/replace_file keeps the dataset and swaps its contents, so anything referencing it stays valid.
• Retry — POST /v1/knowledge/{id}/retry after a transient failure; POST /v1/knowledge/{id}/retry_failed retrains every failed dataset of a chatbot ({id} is the chatbot id there).
• Preview a removal — POST /v1/knowledge/removal_impact counts how many of the ids a removal would act on. Read-only, so a read-scoped key can call it.
• Remove — POST /v1/knowledge/remove_from_chatbot detaches datasets from one bot and keeps them; POST /v1/knowledge/remove_from_system (with confirm_deletion: true) or DELETE /v1/knowledge/{id} deletes them everywhere and emits knowledge.dataset.deleted.

Removal is asynchronous too: a deleted dataset stays readable with is_deleting: true until the background removal finishes, and every change to it meanwhile is 409 DATASET_DELETING.

• Chatting before training finishes. The call succeeds and the answer is ungrounded — sources comes back empty. Check readiness first.
• Re-uploading instead of recrawling. You end up with two datasets saying slightly different things, and retrieval has to pick.
• Polling a failed dataset forever. failed is terminal. Check the source (the dashboard's Knowledge page shows why it failed), fix it, then retry.
• Ignoring the quota. Knowledge is metered per plan, in extracted characters. GET /v1/knowledge/usage shows where you are before a create is refused with 402 KNOWLEDGE_PLAN_SIZE_EXCEEDED.

• Chat — ask questions and read citations.
• Events — the training events worth subscribing to.
• Webhooks — how to receive them.