---
id: "GHSA-3jqg-r5pp-rmp2"
page: "https://infrastack.ai/live/GHSA-3jqg-r5pp-rmp2"
kind: "vulnerability"
aliases: ["GHSA-3jqg-r5pp-rmp2"]
title: "robots.txt enforcement bypass via HTTP redirects in mcp-server-fetch"
severity: "low"
cvss: 3.1
epss: null
exploited: false
status: "open"
category: "package"
products: [{"name": "mcp-server-fetch", "fixed": ["2026.10.24"], "affected": ">= 0.6.1, <= 2026.10.19", "ecosystem": "pypi"}]
ecosystems: ["pypi"]
tags: ["cwe-693", "mcp", "mcp-server-fetch", "package", "python"]
not_affected_if: ["ignore-robots-txt", "trusted-urls-only"]
published_at: "2026-10-25T00:15:37+00:00"
first_seen_at: "2026-11-09T22:53:25+00:00"
updated_at: "2026-11-09T22:58:27+00:00"
sources: [{"source": "ghsa-repo", "url": "https://github.com/modelcontextprotocol/servers/security/advisories/GHSA-3jqg-r5pp-rmp2", "first_seen_at": "2026-11-09T22:53:25+00:00"}]
exploit_sources: []
---

# robots.txt enforcement bypass via HTTP redirects in mcp-server-fetch

Link for people: https://infrastack.ai/live/GHSA-3jqg-r5pp-rmp2 (this .md is for agents)

mcp-server-fetch checks robots.txt only for the URL the model requested, then follows HTTP redirects without checking robots.txt for the redirect targets. A page can redirect the server to a path or host that robots.txt forbids, and the server returns that content to the model.

_Summary, check and fix written by a model from the sources; labelled as such._

## Affected

- pypi:mcp-server-fetch affected >= 0.6.1, <= 2026.10.19 fixed 2026.10.24

## How to check

- Check the installed version: `pip show mcp-server-fetch` and compare against the affected range 0.6.1 to 2026.10.19.
- Look for `mcp-server-fetch` in your Python environments or lockfiles (`uv.lock`, `requirements.txt`) that the fetch tool runs under.

## Fix

- `mcp-server-fetch` `2026.10.24`

## Timeline

- 2026-11-09T22:53:25+00:00 ghsa-repo https://github.com/modelcontextprotocol/servers/security/advisories/GHSA-3jqg-r5pp-rmp2

## Documents

- [advisory] Maintainer's advisory: github.com https://github.com/modelcontextprotocol/servers/security/advisories/GHSA-3jqg-r5pp-rmp2

## From the sources

### Summary

`mcp-server-fetch` checks robots.txt for the URL the model asks for, then lets httpx follow redirects without checking robots.txt again. A page on a host that allows autonomous fetching can redirect to a URL that robots.txt forbids, on another host or on the same one, and the server fetches it and returns the content to the model. This is the default configuration and applies to fetches made through the `fetch` tool, which is where the README says robots.txt is obeyed. The attacker needs the model to fetch a URL they control, or a site that redirects on its own.

### Details

robots.txt is checked once, for the URL passed to the `fetch` tool, in the tool handler ([server.py#L452-L465](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/src/fetch/src/mcp_server_fetch/server.py#L452-L465)):

```python
page_url = str(url)

if not ignore_robots_txt:
    await check_may_autonomously_fetch_url(
        page_url, user_agent_autonomous, proxy_url, allow_private_ips
    )

content, prefix = await fetch_url(
    page_url,
    user_agent_autonomous,
    force_raw=raw,
    proxy_url=proxy_url,
    allow_private_ips=allow_private_ips,
)
```

`check_may_autonomously_fetch_url` ([L247-L293](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/src/fetch/src/mcp_server_fetch/server.py#L247-L293)) downloads `/robots.txt` from that URL's host and calls `can_fetch()` for that one URL. `fetch_url` ([L296-L317](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/src/fetch/src/mcp_server_fetch/server.py#L296-L317)) then fetches it with redirects on:

```python
async with make_client(proxy_url) as client:
    if not allow_private_ips:
        client.event_hooks = {"request": [_refuse_private_destination]}
    try:
        response = await client.get(
            url,
            follow_redirects=True,
            headers={"User-Agent": user_agent},
            timeout=30,
        )
```

httpx follows the 3xx chain inside that `get()`. The only thing that runs per hop is `_refuse_private_destination` (unless `--allow-private-ips` is set), the private-address hook from #5033. Nothing reads robots.txt for a hop's URL, so the server fetches wherever the chain ends. robots.txt belongs to one origin (RFC 9309 section 2.3 gives its URI as `scheme:[//authority]/robots.txt`), which leaves two cases open:

- The requested URL is on a host that allows it (or has no robots.txt) and redirects to another host whose robots.txt says `Disallow: /`, or answers 401/403, which this server treats as "not allowed" ([L274-L277](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/src/fetch/src/mcp_server_fetch/server.py#L274-L277)). The second host's robots.txt is never requested.
- The requested URL redirects within the same host to a path that the host's own robots.txt disallows. The rules were downloaded, but `can_fetch()` only ran for the original path.

The private-address guard still runs on every hop, so this is only about robots.txt.

The `server.py` in the PyPI 2026.10.19 wheel is identical to this commit's. The same script also reproduces on 2026.8.18, 2025.4.7, 2025.1.14 and 0.6.2, with `--allow-private-ips` dropped (it doesn't exist before 2026.10.10) and `isError` for `is_error` (SDK 1.x client). The last three ran with `mcp==1.9.0` and `pydantic==2.10.6`. On 0.6.2 and 2025.1.14 the control calls fail with an unrelated `McpError` constructor error under that SDK and return no content.

### The existing test

`tests/test_protocol.py` pins the current behaviour, inside `test_allow_private_ips_follows_redirect_to_private_address` ([L845-L861](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/src/fetch/tests/test_protocol.py#L845-L861)):

```python
    # robots.txt is only consulted for the original host, never the target.
    assert web.urls() == [ROBOTS, PAGE, metadata, metadata]
```

I read that as a record of what the code does today, in a test whose subject is `--allow-private-ips`, not as a decision that redirects are exempt. The line came in with #4971 (merged 2026-10-04), whose description lists it under "Characterized (pinned, not fixed)" and says each pinned item carries a comment "so the fixing PR knows which assertion to change". There the item is filed under #4838, and #5033, which closed #4838, only dealt with the private-address half.

The README's robots.txt section ([README.md#L168-L172](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/src/fetch/README.md?plain=1#L168-L172)) says:

> By default, the server will obey a websites robots.txt file if the request came from the model (via a tool), but not if the request was user initiated (via a prompt).

and nothing about redirects, while the CAUTION at the top of the same file ([L14](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/src/fetch/README.md?plain=1#L14)) says the private-address guard "checks every redirect hop the same way". SECURITY.md ([L63-L65](https://github.com/modelcontextprotocol/servers/blob/51b6382a3e827aceb6bce430b1208d69b93d95ea/SECURITY.md?plain=1#L63-L65)) names `robots.txt` among the boundaries a server claims to enforce: "A way around a boundary a server claims to enforce (its allowed directories, the client's Roots, or `robots.txt`, for example) is a vulnerability". From the site's side, a model-initiated request for its URL is an autonomous fetch however the request got there. If redirects are meant to be exempt, the README should say so; if not, that assertion is what changes, and the patch updates it.

### Related work

#4838 was fixed by #5033, which registered `_refuse_private_destination` as a request hook so every hop goes through the address guard. robots.txt was not part of that change. Three older open PRs for the same address problem (#4497, #4773, #4803) replace automatic redirects with a manual loop; none of them checks robots.txt inside the loop either. I found no issue, PR or published advisory in this repo about robots.txt and redirects. #5197 is about 5xx responses on robots.txt, a different bug.

### Suggested fix

Check robots.txt in a second request hook, for every request after the first (the handler has already checked the first one), and only when the caller asked for robots.txt enforcement:

```diff
diff --git a/src/fetch/src/mcp_server_fetch/server.py b/src/fetch/src/mcp_server_fetch/server.py
index bdac272..9618251 100644
--- a/src/fetch/src/mcp_server_fetch/server.py
+++ b/src/fetch/src/mcp_server_fetch/server.py
@@ -299,15 +299,31 @@ async def fetch_url(
     force_raw: bool = False,
     proxy_url: str | None = None,
     allow_private_ips: bool = False,
+    check_robots_txt: bool = False,
 ) -> Tuple[str, str]:
     """
     Fetch the URL and return the content in a form ready for the LLM, as well as a prefix string with status information.
     """
     from httpx import HTTPError
 
+    initial_url = str(httpx.URL(url))
+
+    async def _check_redirect_robots(request: httpx.Request) -> None:
+        # The caller has already checked robots.txt for the requested URL;
+        # this checks the URL of every redirect hop after it.
+        if str(request.url) != initial_url:
+            await check_may_autonomously_fetch_url(
+                str(request.url), user_agent, proxy_url, allow_private_ips
+            )
+
     async with make_client(proxy_url) as client:
+        hooks = []
         if not allow_private_ips:
-            client.event_hooks = {"request": [_refuse_private_destination]}
+            hooks.append(_refuse_private_destination)
+        if check_robots_txt:
+            hooks.append(_check_redirect_robots)
+        if hooks:
+           

---

_Free to read, with fair use of about 60 pages a minute. For higher limits or API access: sales@infrastack.ai_