v1.10.82-f67ee7d
Skip to main content
← Back to Hex Proxies

Python Selenium Proxy Integration

Integrate Hex Proxies into Python Selenium for browser-based automation, testing, and scraping with residential proxy support across all major browsers.

Recommended: Residential Proxies

Why Selenium for Proxy-Based Browser Automation

Selenium remains the industry standard for browser automation that requires full JavaScript rendering, complex user interaction simulation, and cross-browser compatibility. When combined with residential proxies, Selenium enables geo-accurate testing of localized content, CAPTCHA-heavy checkout flows, and single-page applications that lightweight HTTP clients cannot handle. The browser's full networking stack processes cookies, redirects, and JavaScript-initiated requests through the proxy transparently, which is essential for sites that verify behavioral consistency.

Selenium's proxy support varies significantly by browser. Chrome accepts proxy configuration through launch arguments but handles authentication differently than Firefox, which supports proxy auth natively through its profile preferences. Understanding these browser-specific differences is critical to avoiding the most common integration failures.

Complete Configuration Example

import os
import zipfile
import tempfile
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

proxy_host = "gate.hexproxies.com" proxy_port = 8080 proxy_user = os.environ["PROXY_USER"] proxy_pass = os.environ["PROXY_PASS"]

def create_proxy_auth_extension(): manifest = '{"version":"1.0","manifest_version":2,"name":"Proxy","permissions":["proxy","webRequest","webRequestBlocking","<all_urls>"],"background":{"scripts":["bg.js"]}}' bg_js = f"""var config = {{mode:"fixed_servers",rules:{{singleProxy:{{scheme:"http",host:"{proxy_host}",port:{proxy_port}}}}}}}; chrome.proxy.settings.set({{value:config}}); chrome.webRequest.onAuthRequired.addListener(function(d){{return{{authCredentials:{{username:"{proxy_user}",password:"{proxy_pass}"}}}}}},{{urls:["<all_urls>"]}},["blocking"]);""" ext_dir = tempfile.mkdtemp() ext_path = os.path.join(ext_dir, "proxy_auth.zip") with zipfile.ZipFile(ext_path, "w") as zf: zf.writestr("manifest.json", manifest) zf.writestr("bg.js", bg_js) return ext_path

options = Options() options.add_extension(create_proxy_auth_extension()) options.add_argument("--disable-blink-features=AutomationControlled") options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome(options=options) driver.set_page_load_timeout(30) driver.get("https://example.com") print(driver.title) driver.quit() ```

Selenium-Specific Proxy Challenges

The primary challenge with Selenium proxy integration is authentication. Chrome does not support proxy authentication via the `--proxy-server` launch argument alone. You must either use a Chrome extension that intercepts authentication challenges (as shown above) or use Selenium Wire as a drop-in replacement that handles proxy auth transparently. Firefox is simpler: you can set proxy credentials directly in the browser profile preferences.

Common Pitfalls with Selenium Proxies

The extension-based auth approach fails silently in headless Chrome. Chrome extensions are not loaded in headless mode by default. Use `--headless=new` (Chrome 112+) instead of the legacy `--headless` flag, as the new headless mode supports extensions. Alternatively, switch to Selenium Wire which intercepts proxy auth at the network layer regardless of headless mode.

Another frequent issue is WebDriver detection. Sites that employ anti-bot measures check for the `navigator.webdriver` flag that Selenium sets by default. The `--disable-blink-features=AutomationControlled` argument removes this flag, but you should also set a realistic user agent and window size to avoid behavioral fingerprinting.

Session Management for Multi-Step Flows

Residential proxies with sticky sessions are essential for login-based Selenium workflows. Configure a session ID in your proxy username to maintain the same exit IP across page navigations. Without sticky sessions, each new page load might route through a different IP, causing session cookies to be invalidated by the target site's security system.

Resource Optimization

Selenium consumes significant memory per browser instance. When running parallel proxied sessions, limit yourself to 3-5 concurrent browsers per GB of RAM. Disable image loading via Chrome preferences if you only need HTML content, as this reduces both bandwidth through the proxy and memory usage on the client side.

Integration Steps

1

Create a proxy authentication extension for Chrome

Build a minimal Chrome extension that intercepts proxy auth challenges and provides credentials automatically. This is required because Chrome does not support proxy username/password in launch arguments.

2

Configure Chrome options with anti-detection flags

Add the proxy auth extension, disable the webdriver automation flag, and set a realistic window size and user agent to reduce detection risk on protected sites.

3

Set page load timeouts aligned with proxy latency

Residential proxies add 200-500ms of latency per request. Set page_load_timeout to at least 30 seconds to account for this overhead plus the target site render time.

4

Verify proxy routing and test sticky sessions

Navigate to an IP-check site to confirm traffic routes through the proxy. Test a multi-page flow to verify sticky sessions maintain the same exit IP across navigations.

Operational Tips

Keep sessions stable for workflows that depend on consistent identity. For high-volume collection, rotate IPs and reduce concurrency if you see timeouts or 403 responses.

  • Prefer sticky sessions for multi-step flows (auth, checkout, forms).
  • Rotate per request for scale and broad coverage.
  • Use timeouts and retries to handle transient failures.

Frequently Asked Questions

Why does Chrome proxy authentication fail in headless mode?▾

Legacy headless mode (--headless) does not load Chrome extensions. Use --headless=new which supports extensions, or switch to Selenium Wire which handles proxy authentication at the network layer without requiring a browser extension.

How do I use a different proxy IP for each Selenium test?▾

Generate a unique session ID for each test and embed it in the proxy username (e.g., user-session-test1). Each unique session ID routes through a different residential IP. Create a new Chrome extension with the updated credentials for each test instance.

Can I use Firefox instead of Chrome to avoid the extension workaround?▾

Yes. Firefox supports proxy authentication natively through profile preferences. Set network.proxy.type, network.proxy.http, network.proxy.http_port, and the corresponding auth preferences in a Firefox profile. This is simpler than the Chrome extension approach.

Ready to Integrate?

Start using residential proxies with Python Selenium today.

Create AccountView Pricing

Related Resources

Guide

Python Proxy Integration

Code

Python Selenium Proxy

Guide

How to Set Up Rotating Proxies in Python

Code

Python aiohttp Proxy

Use Case

Proxies for Ad Verification

Product

Residential Proxies