Re-scraping a website
Refresh a website source to pick up new and changed pages after the site has been updated.
Websites change — you add pages, update prices, rewrite a policy. Re-scraping reads the site again so your knowledge base reflects the current content rather than what it looked like when you first added it.
When to re-scrape
Re-scrape a website source when:
- You've published new pages you want the agent to know about.
- You've changed existing content — pricing, hours, services, descriptions.
- An earlier run came back Partial or had pages you'd now like to include.
PDF and text sources don't change on their own, so they don't need re-scraping. To update one of those, remove it and add the new version.
How to re-scrape
You re-scrape by running the Add website flow again for the same site. Lorito recognises the site, compares it against what's already indexed, and shows you what's changed before anything is re-indexed.
Start the website flow for the same site. On the Knowledge page, click Add source → Website and enter the same site address, then click Discover pages.
Review the changes. Lorito discovers the site's pages again and shows a summary comparing them to what you already have — how many pages are new, how many were removed from the site, and how many are unchanged. Pages that are already indexed are noted so you can focus on what's actually different.
Adjust the scope (optional). Just as on the first run, you can set a maximum number of pages and exclude paths under Advanced options.
Start indexing. Confirm to index the new and changed pages.
What happens to existing content
Re-scraping is designed to update in place, not to start over:
- Unchanged pages stay as they are — Lorito recognises content that hasn't changed and leaves the existing index untouched, so re-scraping a large site doesn't redo work needlessly.
- New pages are added to the source as new items.
- Changed pages are refreshed so the agent answers from the current version.
Pages an exclusion or page limit leaves out won't be indexed, exactly as on the first run. If you previously excluded a section and want it now, remove that exclusion when you re-scrape.
Removing a source instead
If a site has changed so much that you'd rather rebuild it from scratch — or you no longer want it in the knowledge base — remove the source from its menu in the source list. This deletes all of its indexed content. You can then add the site fresh. Removing a source can't be undone.
Next steps
- How indexing works — what the statuses mean while a re-scrape runs.
- Capacity & limits — keeping within your plan as content grows.