Proxies for LangChain Applications

Proxies for LangChain Applications

LangChain is the dominant framework for building LLM-powered applications. Many LangChain workflows — document loaders, web research agents, retrieval chains — need to fetch data from external websites. Without proxy infrastructure, these requests hit rate limits, geographic blocks, and anti-bot defenses.

LangChain Components That Need Proxies

• WebBaseLoader: Fetches HTML from URLs for document ingestion
• RecursiveUrlLoader: Crawls entire sites for knowledge base construction
• WebResearchRetriever: Searches the web and fetches results in real-time
• Custom Tools: Agent tools that call external APIs or scrape data

Configuring WebBaseLoader with Proxies

LangChain's WebBaseLoader uses requests under the hood. Pass proxy configuration through the session:

import requests
from langchain_community.document_loaders import WebBaseLoader

def create_proxied_session(username: str, password: str) -> requests.Session:
    """Create a requests session configured with Hex Proxies."""
    session = requests.Session()
    proxy_url = f"http://{username}:{password}@gate.hexproxies.com:8080"
    session.proxies = {"http": proxy_url, "https": proxy_url}
    session.headers.update({
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
    })
    return session

# Use with WebBaseLoader
session = create_proxied_session("YOUR_USER", "YOUR_PASS")
loader = WebBaseLoader(
    web_paths=["https://example.com/page1", "https://example.com/page2"],
    requests_kwargs={"proxies": session.proxies},
)
docs = loader.load()

Custom Proxy-Aware Tool for Agents

Build a LangChain tool that routes all web requests through proxies:

from langchain.tools import tool
import httpx

PROXY_URL = "http://YOUR_USER:YOUR_PASS@gate.hexproxies.com:8080"

@tool
def fetch_webpage(url: str) -> str:
    """Fetch a webpage through proxy infrastructure. Use for any URL that needs scraping."""
    with httpx.Client(proxy=PROXY_URL, timeout=30, follow_redirects=True) as client:
        resp = client.get(url, headers={
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
            "Accept": "text/html,application/xhtml+xml",
        })
        resp.raise_for_status()
        return resp.text[:10000]  # Limit context size for LLM

@tool
def fetch_api_data(url: str) -> str:
    """Fetch JSON data from an API through proxy. Use for structured data endpoints."""
    with httpx.Client(proxy=PROXY_URL, timeout=30) as client:
        resp = client.get(url, headers={"Accept": "application/json"})
        resp.raise_for_status()
        return resp.text[:5000]

Geo-Targeted Research Agent

Build an agent that can research topics from specific geographic perspectives:

from langchain_openai import ChatOpenAI
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_core.prompts import ChatPromptTemplate

def build_geo_proxy(country: str) -> str:
    return f"http://YOUR_USER-country-{country.lower()}:YOUR_PASS@gate.hexproxies.com:8080"

@tool
def fetch_geo_content(url: str, country: str = "US") -> str:
    """Fetch content as seen from a specific country. Useful for regional pricing or localized content."""
    proxy = build_geo_proxy(country)
    with httpx.Client(proxy=proxy, timeout=30) as client:
        resp = client.get(url)
        return resp.text[:8000]

llm = ChatOpenAI(model="gpt-4.1-mini")
tools = [fetch_webpage, fetch_api_data, fetch_geo_content]
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a research assistant with web access through proxies."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])
agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

RecursiveUrlLoader for Knowledge Bases

When building a RAG knowledge base that requires crawling entire documentation sites:

from langchain_community.document_loaders import RecursiveUrlLoader
from bs4 import BeautifulSoup

def bs4_extractor(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    return soup.get_text(separator="\n", strip=True)

loader = RecursiveUrlLoader(
    url="https://docs.example.com",
    max_depth=3,
    extractor=bs4_extractor,
    requests_kwargs={
        "proxies": {
            "http": PROXY_URL,
            "https": PROXY_URL,
        }
    },
)
docs = loader.load()

Performance Tips for LangChain + Proxies

LangChain agents can make many sequential web requests. Use ISP proxies for the lowest latency — their sub-50ms response time keeps agent chains fast. For broad web research that hits many different domains, residential rotating proxies provide the IP diversity needed to avoid blocks.

Hex Proxies' multi-Gbps capacity ensures your LangChain agents are never bottlenecked by proxy throughput.