How indexing works

When you add a source, Lorito turns it into something your agent can search and answer from. That process is called indexing. This page explains the steps in plain terms and what each status on the Knowledge page means.

The pipeline, step by step

Every source goes through the same broad stages, whether it's a website, a PDF, or pasted text.

1. Discover — For a website, Lorito finds the individual pages to read (using the site's sitemap, or the homepage links as a fallback). PDFs and text sources skip this step — the content is already in hand.
2. Read and extract — Lorito visits each page (or opens the PDF, or takes your pasted text) and pulls out the meaningful text, leaving aside navigation, footers, and other page furniture.
3. Split into pieces — The extracted text is broken into smaller, overlapping chunks. Smaller pieces let the agent retrieve the most relevant passage for a question, rather than a whole page.
4. Index — Each piece is converted into a searchable form and stored. This is what "indexed" means: the content is now part of the knowledge base.
5. Retrieve at answer time — When a visitor asks a question, the agent searches the indexed pieces, pulls the most relevant ones, and answers from them. Content that isn't indexed can't be used in an answer.

Source statuses

Each source in your list shows a status that tracks where it is in the pipeline.

• Discovering — Lorito is finding the pages on a website. You'll see this only for website sources, right after you start.
• Indexing — Lorito is reading the content and adding it to the knowledge base. This is normal and may take from seconds to a few minutes depending on size.
• Indexed — The source finished successfully and its content is live in the knowledge base. Your agent can answer from it.
• Partial — The source indexed, but not everything could be included — for example, some pages were unreachable or blocked. The content that did index is usable; the source row tells you what was skipped.
• Error — Something prevented the source from indexing. Common causes are a site blocking automated access, or a PDF with no readable text. Remove the source and try again, or add the content another way.
• Pending — The source is queued and about to start.

Item statuses

Expand a website source to see its individual pages, each with its own status:

• Indexed — The page's content is in the knowledge base and available for answering.
• Scraped — The page was read and is on its way into the index.
• Pending — The page is queued to be read.
• Ignored — The page was intentionally left out. A short note explains why — for example, it had too little content to be useful, was blocked by the site's robots.txt, or you chose to exclude it.
• Error — The page couldn't be read or indexed.

An ignored page isn't a failure — it just isn't contributing content. Pages with very little text are commonly ignored because they'd add noise rather than helpful answers.

What "indexed" means for answering

Your agent can only answer from content that has reached Indexed. While a source is still discovering or indexing, your agent works with whatever is already indexed and improves as more completes. Once a source shows Indexed (or a website shows all its items indexed), its full content is available.

If an answer seems to be missing information, check that the relevant source is indexed and that the specific page or document made it in — the item-level statuses are the quickest way to confirm.

Next steps

• Adding sources — add a website, PDF, or text source.
• Re-scraping a website — refresh a site after it changes.
• Capacity & limits — how indexed content counts against your plan.