Question 1

Why did my crawl return almost nothing?

Accepted Answer

Usually the site is rendered by JavaScript, robots.txt disallows those paths, or the start URL redirected. Crawl one static page to tell them apart in seconds.

Question 2

Can Agentency crawl pages behind a login?

Accepted Answer

No. It crawls anonymously and cannot sign in as you. Upload the documents or paste the content as text instead.

Question 3

Which setting actually limits how much is collected?

Accepted Answer

Max pages. Depth only bounds how far link-following reaches — the page count is the real cap, and your knowledge budget can stop it sooner.

Question 4

Does a website crawl repeat itself?

Accepted Answer

No. It runs once and captures the site as it was. Only sitemap mode can rescan on a schedule and skip pages that have not changed.

Question 5

How do I check whether a page will crawl well before I start?

Accepted Answer

Open it in a private window, view the page source, and search for a sentence you can see on screen. If it is not in the source, it will not extract.

Question 6

What are the rescan options and which is the default?

Accepted Answer

Manual, Hourly, Daily, or Weekly. Daily is the default. Manual runs once and never reschedules, which is the setting people forget they chose.

Question 7

When exactly does the next scan happen?

Accepted Answer

The interval is measured from when the previous run finished, not when it started, so a slow crawl drifts later. A background check picks up due scans every few minutes.

Question 8

Will rescanning duplicate all my pages?

Accepted Answer

No. Each page is compared with what was collected before. Identical pages are recorded as unchanged and nothing is written, so a rescan costs almost no capacity.

Question 9

How many URLs can one sitemap crawl collect?

Accepted Answer

Leave Max total URLs empty for no limit, or set a ceiling up to 100,000. Treat that as a safety ceiling — your plan's knowledge cap will stop you far sooner.

Question 10

My sitemap crawl says pending. Is it stuck?

Accepted Answer

Probably not. A scheduled crawl rests in a pending state between scans and does not keep the training banner spinning. That is the healthy waiting state.

Question 11

What is the difference between skipped and failed?

Accepted Answer

Skipped means Agentency chose not to collect the page — a robots rule or too little text. Failed means the fetch was attempted and errored. Different causes, different fixes.

Question 12

My rescan reports mostly 'up to date'. Is it broken?

Accepted Answer

That is success. Those pages are identical to the last scan, so nothing was rewritten and no knowledge capacity was spent. A quiet site should look like that.

Question 13

My sitemap crawl says Scheduled and nothing is happening. Is it stuck?

Accepted Answer

No. Scheduled is the resting state between scans, shown with the next scan time. It is waiting, not working — deleting it would only switch the schedule off.

Question 14

Some pages failed to fetch. Do I have to crawl the whole site again?

Accepted Answer

No. Use Retry failed pages on that row, which re-queues only the failed URLs. Re-running the whole crawl would re-fetch everything that already worked and duplicate it.

Question 15

Does deleting a run from history remove the pages it collected?

Accepted Answer

No. It cancels an in-flight website crawl or deactivates a sitemap schedule. The pages are knowledge sources and stay until you remove them.

Question 16

Should I turn Respect robots.txt off?

Accepted Answer

Only on a site you own, and only when you know which rule is in the way. Fixing robots.txt on your own site is the better answer — it helps search engines too.

Question 17

Will turning it off get me past a login wall?

Accepted Answer

No. It only ignores the site's crawler rules. Logins, paywalls, and non-public addresses are all still out of reach.

Question 18

How do I crawl just one section of a site?

Accepted Answer

Use an include pattern such as https://example.com/docs/* — only matching URLs are fetched. It is the highest-value setting on the crawl form.

Question 19

My staging site refuses everything. Why?

Accepted Answer

Staging sites are commonly published with a rule disallowing every path. Crawl production instead, or upload the content directly.

Question 20

Does recrawling replace the old pages?

Accepted Answer

No — it adds new ones. Run the same website crawl twice and the chatbot has two versions of every page. Remove the old sources first, or use sitemap mode.

Question 21

Can I resume a cancelled crawl?

Accepted Answer

No. Start a new one, and narrow it with include patterns so you are not re-fetching everything you already have.

Question 22

If I switch off a sitemap schedule, do I lose the pages?

Accepted Answer

No. Deactivating stops future scans; the collected pages remain knowledge and the chatbot keeps answering from them.

Question 23

My schedule keeps re-adding pages I deleted.

Accepted Answer

That is what a schedule does — it restores anything still listed in the sitemap. Sitemap mode has no exclude field, so remove those URLs, use a narrower child sitemap, or switch the frequency to Manual.

Question 24

What are the defaults and the maximums?

Accepted Answer

Defaults are depth 5, 100 pages, 1 second delay, 1 page at a time. Maximums are depth 10, 100,000 pages, 60 seconds, and 5 in parallel.

Question 25

How many pages actually fit in my plan?

Accepted Answer

A substantial article extracts to roughly 5-15 KB, so Free's 400 KB is about 30-80 pages. Set your page limit against your plan, not against the form's ceiling.

Question 26

Why did the form change the number I typed?

Accepted Answer

The field clamps to the allowed ceiling. That is the form correcting you before the server rejects the request, not a bug.

Question 27

Should I just raise the page limit when a crawl misses content?

Accepted Answer

Usually not. A bigger crawl spends your budget on tag and pagination pages. An include pattern for the section you want gives a far better chatbot.

Question 28

Can I make a crawl finish faster?

Accepted Answer

Raise Pages in parallel to 5 and lower the delay — but only on a site you own. On someone else's server, a crawler that gets blocked is slower than a polite one.

Question 29

What is the Site information dataset?

Accepted Answer

One knowledge source per crawl, labelled Site information. It stores repeated site prose once — the footer address, phone number, opening hours, and similar contact text — instead of dropping that text as chrome from every page. It appears after a smart crawl finishes when Keep repeated site text as one Site information dataset is on.

Question 30

What does Smart content extraction do?

Accepted Answer

It learns each section's repeated layout (menus, footers, sidebars) and trains each page on that page's own content. The switch is under Advanced options on the crawl form, on by default, and cannot be changed after the crawl has started.

Question 31

Should I turn on Include listing pages?

Accepted Answer

Usually no. Category, tag, and archive pages are mostly links to articles the crawl collects anyway. Leave it off unless a listing page itself carries answers you need. You can turn it on later from the Training report.

Question 32

Where is the training report?

Accepted Answer

On the Knowledge tab, from the active collection card or the crawl history drawer. Open View training report. The drawer shows page outcomes, repeated site text, the Site information status, and the two switches you can still change after the crawl.

Question 33

Why doesn't the chatbot know our phone number after a crawl?

Accepted Answer

That text usually lives in the footer. With Smart content extraction and Site information on, it is stored once in the Site information source — wait until that row is trained, then ask using the words on the footer. If the row is missing, the switch was off or the crawl has not finished.

Website crawling FAQ