Skip to content
Call wiki

Why does Claude sometimes fail to fetch a web page — or think it fetched one it didn't?

Last updated 2026-08-17 · Confidence: documented — Anthropic’s platform docs, help center, and crawler policy; the Claude Code fetch mechanics are reported from a code teardown, and the fetch-from-your-own-IP claim is tried.

Fetches fail for identifiable reasons — sites that block Claude’s crawler, pages needing JavaScript, private addresses — and when they fail, Claude sometimes answers anyway from search snippets. The reliable check: ask for a verbatim quote from the page.

Why claude.ai can’t fetch some pages

Section titled “Why claude.ai can’t fetch some pages”

Claude fetches with the Claude-User agent, and Anthropic honors robots.txt: “Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query.” The web fetch tool docs add more blockers:

  • Anthropic-side restrictions — private addresses, robots.txt, domain deny-lists (url_not_allowed).
  • Only URLs already in the conversation — pasted by you or returned by search — can be fetched; Claude can’t construct its own.
  • No JavaScript rendering, so dynamically rendered pages come back empty.
  • Results are cached, so a fetch “may not always reflect the latest version.”

Sites with bot detection (Reddit is a documented case) refuse the request outright.

Claude Code: same fetcher, plus an escape hatch

Section titled “Claude Code: same fetcher, plus an escape hatch”

“Claude Code without restrictions” is only half true. Its WebFetch sends the request from your own machine — verified 2026-08-17: an echo service saw the machine’s own IP with agent Claude-User (claude-code/2.1.234) — but still checks a claude.ai deny-list, refuses localhost (tried — a local test server got “Invalid URL”), and a teardown shows a small model digests the page (100KB cap, 15-minute cache) rather than passing it raw. The real difference: Claude Code has a shell, so a blocked fetch can fall back to curl — or mirror the whole site locally. A browser agent covers JavaScript-heavy or logged-in pages.

The false-success mode

Section titled “The false-success mode”

A dated report (2026-04-08, closed as duplicate) shows the trap: WebFetch fails, Claude silently substitutes web-search snippets and presents them as the page. To confirm a real fetch: ask for the page’s first sentence verbatim, or a detail search snippets wouldn’t carry — and treat any pageless-looking summary as unread.

Sources

Section titled “Sources”