February 18, 2025 · 8 min read
Scraping JavaScript sites without turning your pipeline into a browser farm
A practical decision tree for rendering, waiting, extracting, and retrying dynamic pages.
Reliable retrieval comes from explicit decisions about discovery, rendering, identity, evidence, and failure:not from adding more retries after launch.
Render only when the page requires it
A headless browser is powerful but expensive. Begin with a normal HTTP fetch and inspect whether meaningful content, JSON-LD, or application state is already present. Escalate to browser rendering when the response is an app shell, when content appears after client-side requests, or when interaction is genuinely required. This split keeps the common path fast and leaves browser capacity for pages that need it.
Wait for evidence, not an arbitrary clock
A fixed five-second sleep is both slow and unreliable. Wait for a selector that proves the target content exists, a relevant network response, or a short network-idle window. Define a hard deadline too: an analytics request that never settles should not hold a worker forever. Rounderone browser actions support selector waits and bounded timeouts for this reason.
Extract the document, not the interface
After rendering, remove cookie banners, navigation, related-content rails, and hidden templates before producing Markdown. Preserve semantic headings, table structure, links, and image alt text. For repeated jobs, use schema extraction for stable fields and keep the cleaned Markdown as evidence when an operator needs to audit a value.
Classify failures before retrying
Timeouts and browser crashes can be retried with backoff. A missing selector may signal a template change and should enter a review queue. Authentication walls and explicit blocks should stop. This distinction prevents retry storms and gives product teams a useful health signal instead of a generic failure percentage.
PUT IT INTO PRACTICE
Test retrieval on a real page.
See the cleaned Markdown and source metadata Rounderone returns.
Open playground