Why httpx for Proxy-Based Workflows
httpx has become the go-to Python HTTP client for teams that need both synchronous and asynchronous request capabilities in a single library. Unlike the older requests library, httpx natively supports HTTP/2, connection pooling across proxied connections, and async context managers that make it straightforward to build high-throughput scraping pipelines. When paired with residential proxies, httpx excels at maintaining long-lived sessions while rotating exit IPs transparently.
The library's proxy handling is notably more flexible than alternatives. You can configure proxies at the client level for all requests, override them per-request, or use httpx's event hook system to dynamically select proxies based on the target domain. This granularity is critical when you need sticky sessions for checkout flows on one domain while simultaneously rotating IPs for catalog scraping on another.
Complete Configuration Example
import httpxproxy_url = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.hexproxies.com:8080"
transport = httpx.HTTPTransport( proxy=proxy_url, retries=3, verify=True, )
with httpx.Client( transport=transport, timeout=httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=5.0), limits=httpx.Limits(max_connections=20, max_keepalive_connections=10), headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}, follow_redirects=True, ) as client: response = client.get("https://example.com") print(response.status_code, len(response.text))
# Async variant for high-throughput workloads # async with httpx.AsyncClient(transport=transport) as aclient: # tasks = [aclient.get(url) for url in url_list] # results = await asyncio.gather(*tasks) ```
httpx-Specific Proxy Features
httpx separates transport configuration from client configuration, which means you can swap proxy transports without rebuilding your client. The `HTTPTransport` class accepts a `retries` parameter that automatically retries on connection failures at the transport level, before your application-level retry logic ever fires. This two-layer retry approach dramatically improves success rates through residential proxies where occasional connection resets are expected.
Common Pitfalls with httpx
The most frequent mistake is passing proxy configuration using the legacy `proxies` dictionary format. While httpx accepted this in earlier versions, modern httpx (0.24+) requires proxy configuration through the `transport` parameter or the `proxy` shorthand on the client constructor. Using the old format silently ignores your proxy settings, sending requests directly. Another subtle issue: httpx's default connection pool limits are conservative. When running concurrent requests through a proxy gateway, you should explicitly set `max_connections` to match your concurrency target, otherwise httpx will queue requests and artificially throttle throughput.
Advanced: Event Hooks for Dynamic Proxy Selection
httpx supports request and response event hooks. You can use a request hook to log which proxy IP was used, measure latency per-region, or dynamically modify headers based on the target site. Response hooks let you detect soft blocks (200 status but CAPTCHA content) and trigger automatic IP rotation before the next request fires.
Performance Tuning
Enable HTTP/2 by passing `http2=True` to the client constructor. HTTP/2 multiplexing through a single proxy connection reduces handshake overhead significantly when making many requests to the same origin. Combine this with `max_keepalive_connections` tuned to your proxy plan's concurrency limit for optimal throughput.