How to Set Up Rotating Proxies in Python: Complete Tutorial

Python is the most popular language for web scraping and automation, and rotating proxies are essential for any serious data collection project. This tutorial covers four Python HTTP libraries — requests, aiohttp, httpx, and Scrapy — with complete, runnable code examples that include authentication, rotation configuration, error handling, retry logic, and session management. Every example uses the Hex Proxies gateway, but the patterns apply to any proxy service that supports username/password authentication.

By the end of this guide, you will have production-ready proxy integration code for whichever Python library your project uses.

Prerequisites

Before starting, you need:

• Python 3.9 or later installed
• A Hex Proxies account with username and password credentials
• The proxy gateway address: gate.hexproxies.com (port 8080 for HTTP/HTTPS, port 1080 for SOCKS5)

Install the libraries you plan to use:

# Choose one or more
pip install requests
pip install aiohttp
pip install httpx
pip install scrapy

1. Rotating Proxies with Requests

The requests library is Python's most widely used HTTP client. Proxy configuration is straightforward through the proxies parameter.

Basic Rotating Proxy Setup

import requests

PROXY_USER = "YOUR_USERNAME" PROXY_PASS = "YOUR_PASSWORD" PROXY_HOST = "gate.hexproxies.com" PROXY_PORT = 8080

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}" proxies = { "http": proxy_url, "https": proxy_url, }

Each request automatically gets a new IP response = requests.get( "https://httpbin.org/ip", proxies=proxies, timeout=15, ) print(response.json()) # Output: {"origin": "203.0.113.42"} (a different IP each time) ```

Requests with Retry Logic and Error Handling

Production code needs robust error handling. Proxy requests can fail due to authentication errors (407), rate limiting (429), connection timeouts, and temporary gateway errors (502/503).

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import time

PROXY_USER = "YOUR_USERNAME" PROXY_PASS = "YOUR_PASSWORD" PROXY_HOST = "gate.hexproxies.com" PROXY_PORT = 8080

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}" proxies = { "http": proxy_url, "https": proxy_url, }

def create_session_with_retries( max_retries: int = 3, backoff_factor: float = 1.0, status_forcelist: tuple = (429, 500, 502, 503, 504), ) -> requests.Session: """Create a requests session with automatic retry on failure.""" session = requests.Session() session.proxies = proxies

retry_strategy = Retry( total=max_retries, backoff_factor=backoff_factor, status_forcelist=status_forcelist, allowed_methods=["GET", "HEAD", "OPTIONS"], ) adapter = HTTPAdapter(max_retries=retry_strategy) session.mount("http://", adapter) session.mount("https://", adapter) return session

def fetch_with_proxy(url: str, session: requests.Session) -> dict: """Fetch a URL through the proxy with comprehensive error handling.""" try: response = session.get(url, timeout=15) response.raise_for_status() return {"url": url, "status": response.status_code, "data": response.text} except requests.exceptions.ProxyError as e: return {"url": url, "status": 407, "error": f"Proxy auth failed: {e}"} except requests.exceptions.ConnectTimeout: return {"url": url, "status": 0, "error": "Connection timed out"} except requests.exceptions.ReadTimeout: return {"url": url, "status": 0, "error": "Read timed out"} except requests.exceptions.HTTPError as e: return {"url": url, "status": e.response.status_code, "error": str(e)} except requests.exceptions.RequestException as e: return {"url": url, "status": 0, "error": f"Request failed: {e}"}

Usage session = create_session_with_retries() urls = [f"https://example.com/product/{i}" for i in range(1, 11)]

for url in urls: result = fetch_with_proxy(url, session) print(f"{result['url']} -> {result.get('status', 'error')}") time.sleep(1) # polite delay between requests ```

Sticky Sessions with Requests

For workflows requiring the same IP across multiple requests (login flows, checkout processes), append a session ID to your username:

import requests
import uuid

def create_sticky_proxy_session(session_name: str = "") -> requests.Session: """Create a requests session that uses the same proxy IP for all requests.""" sid = session_name or uuid.uuid4().hex[:12] sticky_proxy = ( f"http://{PROXY_USER}-session-{sid}:{PROXY_PASS}" f"@{PROXY_HOST}:{PROXY_PORT}" ) session = requests.Session() session.proxies = {"http": sticky_proxy, "https": sticky_proxy} session.timeout = 30 return session

All requests in this session use the same IP session = create_sticky_proxy_session("login-flow-001") session.get("https://example.com/login") session.post("https://example.com/login", data={"user": "me", "pass": "secret"}) session.get("https://example.com/dashboard") # Same IP as login ```

2. Async Rotating Proxies with aiohttp

For high-throughput scraping, aiohttp provides async HTTP requests that dramatically increase concurrency. Instead of waiting for each response sequentially, you can run dozens of requests simultaneously.

import aiohttp
import asyncio
from aiohttp import BasicAuth

PROXY_USER = "YOUR_USERNAME" PROXY_PASS = "YOUR_PASSWORD" PROXY_URL = "http://gate.hexproxies.com:8080" PROXY_AUTH = BasicAuth(PROXY_USER, PROXY_PASS)

async def fetch( session: aiohttp.ClientSession, url: str, max_retries: int = 3, ) -> dict: """Fetch a URL through the rotating proxy with retry logic.""" for attempt in range(1, max_retries + 1): try: async with session.get( url, proxy=PROXY_URL, proxy_auth=PROXY_AUTH, timeout=aiohttp.ClientTimeout(total=15), ) as response: if response.status == 429: wait = 2 attempt await asyncio.sleep(wait) continue text = await response.text() return {"url": url, "status": response.status, "data": text} except aiohttp.ClientProxyConnectionError: return {"url": url, "status": 407, "error": "Proxy connection failed"} except asyncio.TimeoutError: if attempt < max_retries: await asyncio.sleep(2 attempt) continue return {"url": url, "status": 0, "error": "Timed out after retries"} except aiohttp.ClientError as e: return {"url": url, "status": 0, "error": str(e)} return {"url": url, "status": 429, "error": "Rate limited after retries"}

async def scrape_urls(urls: list[str], concurrency: int = 10) -> list[dict]: """Scrape multiple URLs concurrently through rotating proxies.""" semaphore = asyncio.Semaphore(concurrency) results = []

async def bounded_fetch(url: str) -> dict: async with semaphore: result = await fetch(session, url) await asyncio.sleep(0.5) # polite delay return result

async with aiohttp.ClientSession() as session: tasks = [bounded_fetch(url) for url in urls] results = await asyncio.gather(*tasks) return list(results)

Usage urls = [f"https://example.com/page/{i}" for i in range(1, 101)] results = asyncio.run(scrape_urls(urls, concurrency=10))

success = sum(1 for r in results if r.get("status") == 200) print(f"Success: {success}/{len(results)}") ```

Key aiohttp Advantages

• Concurrency: 10--50 simultaneous requests vs sequential processing with requests.
• Speed: Scrape 1,000 URLs in the time requests handles 50--100.
• Memory: Lower memory footprint per connection than threading alternatives.

3. Rotating Proxies with httpx

httpx is a modern Python HTTP client that supports both sync and async modes, HTTP/2, and first-class proxy support. It is increasingly popular as a replacement for requests.

import httpx
import time

PROXY_USER = "YOUR_USERNAME" PROXY_PASS = "YOUR_PASSWORD" PROXY_URL = f"http://{PROXY_USER}:{PROXY_PASS}@gate.hexproxies.com:8080"

def scrape_with_httpx(urls: list[str]) -> list[dict]: """Scrape URLs using httpx with rotating proxy and retry logic.""" results = [] transport = httpx.HTTPTransport(retries=3)

with httpx.Client( proxy=PROXY_URL, transport=transport, timeout=httpx.Timeout(15.0, connect=10.0), follow_redirects=True, ) as client: for url in urls: try: response = client.get(url) results.append({ "url": url, "status": response.status_code, "data": response.text[:200], }) except httpx.ProxyError as e: results.append({"url": url, "status": 407, "error": str(e)}) except httpx.TimeoutException: results.append({"url": url, "status": 0, "error": "Timeout"}) except httpx.HTTPError as e: results.append({"url": url, "status": 0, "error": str(e)}) time.sleep(1)

return results

Async version async def scrape_async_httpx(urls: list[str]) -> list[dict]: """Async scraping with httpx and rotating proxy.""" results = [] transport = httpx.AsyncHTTPTransport(retries=3)

async with httpx.AsyncClient( proxy=PROXY_URL, transport=transport, timeout=httpx.Timeout(15.0, connect=10.0), follow_redirects=True, ) as client: for url in urls: try: response = await client.get(url) results.append({ "url": url, "status": response.status_code, "data": response.text[:200], }) except httpx.HTTPError as e: results.append({"url": url, "status": 0, "error": str(e)})

return results

Usage urls = [f"https://example.com/item/{i}" for i in range(1, 21)] results = scrape_with_httpx(urls) ```

httpx vs requests vs aiohttp

4. Scrapy Proxy Middleware

Scrapy is Python's leading web scraping framework. Proxy integration works through custom middleware that injects proxy settings into every request.

Basic Proxy Middleware

# myproject/middlewares.py

import logging from scrapy import signals

logger = logging.getLogger(__name__)

class HexProxyMiddleware: """Scrapy middleware that routes all requests through Hex Proxies."""

PROXY_USER = "YOUR_USERNAME" PROXY_PASS = "YOUR_PASSWORD" PROXY_HOST = "gate.hexproxies.com" PROXY_PORT = 8080

@classmethod def from_crawler(cls, crawler): middleware = cls() return middleware

def process_request(self, request, spider): proxy_url = ( f"http://{self.PROXY_USER}:{self.PROXY_PASS}" f"@{self.PROXY_HOST}:{self.PROXY_PORT}" ) request.meta["proxy"] = proxy_url

def process_response(self, request, response, spider): if response.status == 407: logger.error("Proxy authentication failed for %s", request.url) if response.status == 429: logger.warning("Rate limited on %s, will retry", request.url) return request.replace(dont_filter=True) return response

def process_exception(self, request, exception, spider): logger.error("Proxy error on %s: %s", request.url, exception) return request.replace(dont_filter=True) ```

Enable the Middleware in settings.py

# myproject/settings.py

DOWNLOADER_MIDDLEWARES = { "myproject.middlewares.HexProxyMiddleware": 350, # Disable the default HTTP proxy middleware "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None, }

Recommended settings for proxy-based scraping CONCURRENT_REQUESTS = 16 DOWNLOAD_DELAY = 1 DOWNLOAD_TIMEOUT = 15 RETRY_TIMES = 3 RETRY_HTTP_CODES = [429, 500, 502, 503, 504] ```

Sticky Session Middleware for Scrapy

For crawls that require session continuity (paginated results, authenticated scraping):

# myproject/middlewares.py

import uuid

class HexStickySessionMiddleware: """Scrapy middleware with sticky proxy sessions per spider."""

PROXY_USER = "YOUR_USERNAME" PROXY_PASS = "YOUR_PASSWORD" PROXY_HOST = "gate.hexproxies.com" PROXY_PORT = 8080

def __init__(self): self.session_id = uuid.uuid4().hex[:12]

def process_request(self, request, spider): # Use a per-spider sticky session session_id = getattr(spider, "proxy_session_id", self.session_id) proxy_url = ( f"http://{self.PROXY_USER}-session-{session_id}:{self.PROXY_PASS}" f"@{self.PROXY_HOST}:{self.PROXY_PORT}" ) request.meta["proxy"] = proxy_url ```

Error Handling Reference

Every proxy integration should handle these failure modes:

Performance Comparison: Libraries Under Proxy Rotation

Benchmarked scraping 1,000 URLs through Hex Proxies rotating gateway:

Takeaway: For high-volume scraping, use aiohttp or Scrapy. For simple scripts and prototyping, requests or httpx (sync) are easier to debug and maintain.

How Hex Proxies Works with Python

Hex Proxies provides a single gateway endpoint that handles all rotation logic server-side:

• Gateway: gate.hexproxies.com:8080 (HTTP/HTTPS) or gate.hexproxies.com:1080 (SOCKS5)
• Authentication: Username/password via proxy URL or Basic Auth header
• Rotation: Automatic per-request rotation by default. Add -session-ID to username for sticky sessions.
• Geo-targeting: Add -country-US or -city-london to username for location-specific IPs.
• Pool: 10M+ residential IPs across 195 countries.
• Format: All standard Python proxy URL formats are supported. No SDK or custom library required.

Frequently Asked Questions