January 9, 2025 · 9 min read
Incremental crawling for RAG: refresh what changed, not the whole internet
A crawl refresh strategy built around canonical URLs, content hashes, and deletion handling.
Reliable retrieval comes from explicit decisions about discovery, rendering, identity, evidence, and failure:not from adding more retries after launch.
A crawl is a changing inventory
Treat each run as an observation of a site, not a permanent dump. Store canonical URL, discovered URL, status, content hash, retrieval time, and crawl run ID with every document. Those fields let you explain the state of the index and distinguish a failed fetch from a page that was actually removed.
Normalize identity before comparing content
Tracking parameters, trailing slashes, locale aliases, and redirects can create multiple records for one page. Honor canonical tags carefully, resolve redirects, and apply deterministic URL normalization. Then hash cleaned semantic content rather than raw HTML, which changes whenever analytics tokens or asset fingerprints rotate.
Refresh by signal
Sitemaps with lastmod values are useful hints, not guarantees. Prioritize high-value and frequently changing sections, use conditional requests when servers support them, and periodically sample supposedly unchanged pages to test your assumptions. A bounded full reconciliation catches orphaned or deleted pages that incremental discovery misses.
Propagate change deliberately
When a hash changes, re-chunk the page and update only its vectors and metadata. When a URL disappears, tombstone it so retrieval stops immediately while an audit record remains. Version chunks by document and run; this makes rollback possible when extraction quality regresses or a site publishes malformed content.
PUT IT INTO PRACTICE
Test retrieval on a real page.
See the cleaned Markdown and source metadata Rounderone returns.
Open playground