Feed your agent the pages that answer questions, not the pages that describe you. Twenty well-written answers outperform two hundred marketing pages, because retrieval rewards specificity and punishes repetition.
Every business we onboard arrives with the same instinct: point the crawler at the whole site and let it sort itself out. It is a reasonable instinct, and it is usually wrong. The knowledge base is not an archive. It is the set of things you want said out loud, to a stranger, without you in the room.
What follows is the audit we run before a single agent goes live. It takes about forty minutes for a typical small business site, and it is the difference between an assistant that sounds informed and one that sounds like a brochure.
What actually changes answer quality
Retrieval works by finding the chunk of text closest in meaning to what the visitor asked. That means the useful unit is not the page, it is the paragraph that contains a fact. A page titled "Our Services" that lists nine services in nine sentences will lose to a page that explains one service in six paragraphs, every time.
Three properties predict whether a page will earn its place:
- It states facts, not positioning. "We deliver within 3 to 5 working days to main centres" is retrievable. "Fast, reliable delivery you can count on" is not.
- It is written the way people ask. If customers say "how long does shipping take", a heading that reads "Fulfilment SLA" costs you the match.
- It is current. Stale facts do more damage than missing ones, because the agent will state them with the same confidence as everything else.
What to leave out
Subtraction is the underrated half of this job. Every page you add competes with every other page for retrieval, so noise is not neutral. It actively displaces good answers.
Dated pages
Press releases, event announcements, and "our new office" posts age badly. They teach the agent to cite things that stopped being true, and because nothing in the text says "this expired", it has no way to know.
Near-duplicates
Landing page variants built for ad campaigns are usually 80% identical. When five chunks say nearly the same thing, retrieval returns five copies of a mediocre answer instead of one good one.
| Content | Index it | Why |
|---|---|---|
| FAQ page | Yes | Already question-shaped, highest hit rate per word |
| Pricing page | Yes | Most-asked question on almost every site |
| Terms, returns, delivery | Yes | Factual, stable, and what people actually need |
| Blog announcements | No | Ages badly, no expiry signal in the text |
| Scanned PDFs | No | No extractable text layer to chunk |
| Team bios | Maybe | Useful for trust questions, noise for everything else |
The pre-launch checklist
Run this before you deploy to a live surface. Each item takes a minute or two, and skipping them is where almost every disappointing first week comes from.
- Pricing is on a page, in text, not only in an image or a downloadable rate card.
- Contact details appear somewhere the crawler can reach without a form submission.
- The five questions from your inbox each have a paragraph that answers them plainly.
- Anything seasonal or dated has been excluded or updated.
- You have asked the agent those five questions yourself, and read the answers as a customer would.
That last one is the only test that matters. Everything above is a way of making it more likely to pass. For the mechanics of adding and excluding sources, see adding sources in the docs.