mcp-server-fetch: robots.txt path-canonicalization bypass (e.g. //private fetched while Disallow: /private)

03 · The fix

Upgrade to 2026.11.11

Affected

• Package: mcp-server-fetch (PyPI), under src/fetch
• Tested: 2026.11.4, commit 36380287ed3b15d115b55f546617d723072b1296
• Comparison site: src/fetch/src/mcp_server_fetch/server.py around the Protego can_fetch() call in check_may_autonomously_fetch_url() (about line 532 at that commit). Spot-checks of older revisions since robots support was added show the same non-canonical comparison; we are not claiming every intermediate release.

Reproduction (localhost only)

Uses the project's own FakeWeb harness — no network and no live site. Save as src/fetch/tests/test_repro_robots_bypass.py and run:

cd src/fetch && uv run --frozen pytest tests/test_repro_robots_bypass.py -v

import httpx
import pytest
from mcp.types import CallToolResult
from .harness import FakeWeb, connect

ROBOTS = "https://example.com/robots.txt"
PRIVATE = "the private page"

def page(body):
    return httpx.Response(
        200, content=body.encode(), headers={"content-type": "text/plain"}
    )

def robots():
    return page("User-agent: *\nDisallow: /private")

def text_of(result: CallToolResult) -> str:
    return result.model_dump(by_alias=True, mode="json", exclude_none=True)[
        "content"
    ][0]["text"]

async def test_control_is_refused(web: FakeWeb) -> None:
    """Control: the plain path is refused today, so the rule is enforced."""
    web.add(ROBOTS, robots())
    web.add("https://example.com/private", page(PRIVATE))
    async with connect() as (session, _):
        result = await session.call_tool("fetch", {"url": "https://example.com/private"})
    assert result.is_error is True
    assert PRIVATE not in text_of(result)

@pytest.mark.parametrize(
    "url",
    ["https://example.com//private", "https://example.com/a%2f..%2fprivate"],
)
async def test_alternate_spelling_leaks(web: FakeWeb, url) -> None:
    """These spell the disallowed page differently; the origin still serves it."""
    web.add(ROBOTS, robots())
    web.add(url, page(PRIVATE))
    async with connect() as (session, _):
        result = await session.call_tool("fetch", {"url": url})
    assert result.is_error is True, "the disallowed page was fetched"
    assert PRIVATE not in text_of(result)

async def test_redirect_to_encoded_dots_leaks(web: FakeWeb) -> None:
    """A redirect target is not passed through Pydantic's AnyUrl."""
    web.add(ROBOTS, robots())
    web.add(
        "https://example.com/go",
        httpx.Response(302, headers={"location": "/%2e%2e/private"}),
    )
    web.add("https://example.com/%2e%2e/private", page(PRIVATE))
    async with connect() as (session, _):
        result = await session.call_tool("fetch", {"url": "https://example.com/go"})
    assert result.is_error is True, "the disallowed page was fetched"
    assert PRIVATE not in text_of(result)

On the tested revision the control passes and the three bypass tests fail (AssertionError: the disallowed page was fetched).

We also confirmed against a local Python http.server origin (path resolution comparable to nginx/Apache-style serving): GET /private refused by robots; GET //private and the encoded-dot redirect target returned the private file on the wire. An origin that does not normalize returns 404, so there is no bypass there.

Notes on scope: /%2e%2e/private works through a redirect Location (Pydantic AnyUrl normalizes that form on a direct argument and correctly blocks it). //private works as a direct argument; as a redirect target HTTPX treats scheme-relative //private as host private. Cross-host redirects still re-check the target host's robots.txt, so an attacker-controlled site that 302s to https://victim.example/%2e%2e/private can miss the victim's rule the same way.

Impact assumptions

• Autonomous (model-initiated) fetch with robots enforcement on (default); the intentional prompt-fetch robots exemption is out of scope.
• Attacker influences a URL the agent fetches (directly or via redirect).
• Target publishes a path-specific Disallow, and its origin resolves the alternate spelling to that resource.

Suggested fix

Resolve the request path the way an origin does before matching rules, and require both the literal spelling and the resolved path to be allowed (and). Decode only separator/dot escapes (%2f/%2F/%2e/%2E), collapse duplicate slashes, and remove ./.. per RFC 3986 §5.2.4; leave ordinary escapes such as %20 alone. Derive the wire path with urlsplit/urlunsplit rather than from a URL object that HTTPX/Pydantic may already have normalized. Keep the existing can_fetch(str(url), …) check and add the second check — the change only ever adds refusals.

Happy to share a tested patch + regression file against the project's harness privately if useful.

This issue was identified with AI-assisted static code review and checked against the source before reporting; happy to answer questions or help with a fix.