docspluginsextensionsweb-fetch

web-fetch

Web page fetching as a capability — renders JavaScript, cleans the result into token-efficient markdown, and writes it to staging for the agent to read. Domain policy and SSRF protection always apply.

⚠ HEADS UP

By default an agent's container runs a locked-down network (sdk-only), so Claude's built-in web-fetch can't reach the open web. This extension fetches host-side instead — the more secure path (every request is domain-policed and SSRF-protected) and the more efficient one (pages come back as cleaned, token-efficient markdown). On a full-network agent the built-in tool can reach out again, so prefer this one either way.

What the agent can do

Fetch a URL — one tool: give it a URL, it renders the page, cleans it, and writes the result to staging, returning the paths for the agent to Read.
Choose the output — token-efficient markdown by default, full markdown, or raw HTML; binary content (PDFs, images) is saved as-is.
Pairs with search — the agent finds URLs with WebSearch, then fetches the specific pages it wants in full.

Configuration

Web-fetch needs no credentials. The one decision is how freely the agent may fetch, and which domains are off-limits.

capabilities.json

json · extensions.web-fetch slice

{
  "extensions": {
    "web-fetch": {
      "enabled": true,
      "fetch_mode": "approval",
      "blocked_domains": ["*.internal.example.com"]
    }
  }
}

Field	Type	Default	Effect
fetch_mode	disabled \| approval \| open	approval	disabled doesn’t register the tool; approval prompts unless the domain is allowlisted; open fetches freely.
allowed_domains	string[]	[]	Domains that skip the prompt under `approval`. Wildcards like `*.example.com`.
blocked_domains	string[]	[]	Always rejected, in every mode. Same wildcard syntax.
allow_query_strings	boolean	true	When false, strips query strings and fragments before fetching.

Tools

web__fetch(url, pipelines?)

Fetch a web page, run it through cleaning pipelines, and write the result to /staging/in/. Returns metadata plus the file paths the agent should Read.

Parameters

url[string]URL to fetch — http or https only.
optional pipelines[string[], default ["crawl4ai"]]Processing pipelines. Options: "crawl4ai" (cleaned markdown), "markdown" (full markdown), "raw" (unprocessed HTML).

Returns

Title: <page title>
URL: <normalized URL>
Content-Type: <media type>
Fetched: <iso timestamp>

Files written:
  /staging/in/fetch_<hash>.<pipeline>.<ext> (<N> tokens|bytes)
  /staging/in/fetch_<hash>.meta.json

Use the Read tool to access these files.

when success

Internal addresses are not allowed.

when SSRF block — hostname matches private ranges (127.*, 10.*, 192.168.*, ::1, etc.)

Domain "HOSTNAME" is blocked.

when hostname in blocked_domains

Only http/https URLs are supported.

when non-http/https protocol

Invalid URL: <url>

when URL parsing failed

Fetch failed: <error>

when network or parsing error

Web-fetch service is not running.

when service spawn failed or crashed

Caveats

Output files live in /staging/in/ as fetch_<sha256(url)[:12]>.<pipeline>.<ext>. Text content uses .html (raw) or .md (crawl4ai/markdown); binary content is base64-encoded and saved with .bin or the detected extension. The meta.json sidecar carries title, description, contentType, fetchedAt, and per-pipeline sizes.

Notes & gotchas

🔒 SECURITY

SSRF protection is always on, in every mode — loopback, link-local, and private-IP targets are refused regardless of the domain lists. Keep fetch_mode at approval with a populated blocked_domains rather than open with an empty list.

💡 TIP

Files in /staging/in/ are ephemeral and clear at conversation end. If the agent should remember a fetched page, it copies the markdown into /memory/.