Skip to content
Call wiki

How do you give an agent a whole website to search?

Last updated 2026-08-17 · Confidence: documented — the wget manual, sitemaps.org, WordPress and Firecrawl docs; the grep-a-mirror workflow itself is inferred.

Mirror the site into a local folder once, and questions like “does our site already say this?” or “which pages still reference the old title?” become exhaustive text searches instead of guesses. An agent fetching live pages one at a time — with claude.ai’s fetch restrictions on top — can’t promise it checked everything; grep over local files can, the same localize-first logic that fixes Drive search.

Best source: the CMS or repo behind the site

Section titled “Best source: the CMS or repo behind the site”

Ask whoever runs the site for an export before crawling it. WordPress’s Tools → Export produces one WXR (XML) file containing all posts, pages, and custom post types — a complete corpus in a single download. A static site is even better: its git repo already holds every page as markdown.

Sitemap pull

Section titled “Sitemap pull”

Nearly every site publishes a sitemap at /sitemap.xml — up to 50,000 URLs per file, each with an optional lastmod date. An agent can script the pull: read the sitemap, fetch each URL, save the pages. On refresh, lastmod tells it which pages changed, so it re-fetches only those.

Crawl and mirror

Section titled “Crawl and mirror”

With no sitemap or export, crawl. wget --mirror --convert-links <url> “turns on options suitable for mirroring” — recursive, link-following, timestamped — and rewrites links for local browsing. The output is HTML: still greppable, but noisy. Services like Firecrawl crawl every reachable page and return each as clean markdown instead. The wget manual warns recursive downloads “can overload the remote servers” — add delays, even on your own site.

Whatever the route, a mirror is stale the day after it’s made: re-run the pull on a schedule, or before any consistency sweep. Some sites now also publish llms.txt — curated markdown for agents — worth checking before you build anything.

Sources

Section titled “Sources”