February 18, 2025 · 8 min read
Scraping JavaScript sites without turning your pipeline into a browser farm
A practical decision tree for rendering, waiting, extracting, and retrying dynamic pages.
Read articleFIELD NOTES
Building with the live web.
Engineering patterns and hard-won lessons for retrieval systems that need fresh information, clean context, and verifiable sources.
January 30, 2025 · 7 min read
Search architecture for research agents that cite their work
How query planning, source diversity, and evidence tracking turn web search into dependable research.
Read articleJanuary 9, 2025 · 9 min read
Incremental crawling for RAG: refresh what changed, not the whole internet
A crawl refresh strategy built around canonical URLs, content hashes, and deletion handling.
Read article