Why Typhoeus for Proxy Work

Typhoeus is built on libcurl and its multi interface, which handles parallel HTTP connections at the C level. Where HTTParty and Net::HTTP are single-threaded, Typhoeus's Hydra runs up to 200 concurrent requests through proxies with minimal Ruby thread overhead.

For proxy scraping at scale in Ruby, Typhoeus is the clear winner. A single Hydra can process thousands of proxied requests per minute, with libcurl handling connection pooling, DNS caching, and keep-alive transparently.

Typhoeus-Specific Proxy Features

Single request proxy: `Typhoeus.get(url, proxy: 'http://user:pass@host:port')`. The `proxy` option accepts full URLs with embedded credentials. Typhoeus handles CONNECT tunneling for HTTPS targets automatically.

Hydra parallel requests all share the proxy configuration. Queue requests with `hydra.queue(request)` and run them with `hydra.run`. libcurl multiplexes connections to the proxy, reusing TCP sockets across requests.

Typhoeus exposes libcurl options directly. Set `proxytype: :http` or `proxytype: :socks5` for protocol selection. Use `ssl_verifypeer: true` and `ssl_verifyhost: 2` for secure proxy tunnels.

Common Pitfalls with Typhoeus

Typhoeus requires the libcurl development headers to compile its native extension. On Docker Alpine, install `curl-dev` before bundling. Missing headers cause gem installation failures that look unrelated to Typhoeus.

Hydra's default max_concurrency is 200. Setting it higher than your proxy plan's concurrency limit causes connection failures. Match Hydra concurrency to your plan: `Typhoeus::Hydra.new(max_concurrency: 50)`.

Response callbacks run synchronously between batches. If your callback does heavy processing, it blocks the next batch of proxy requests. Keep callbacks lightweight and defer processing to a background queue.

Advanced Configuration

Use `on_complete` callbacks with status code checking for proxy-aware error handling: `request.on_complete { |response| retry_queue << url if response.code == 429 }`. Build a retry queue that re-queues failed URLs with different proxy sessions.

Set `connecttimeout: 10` and `timeout: 30` per request. Typhoeus passes these to libcurl, which enforces them at the socket level. This is more reliable than Ruby-level timeouts for proxy connections.

Performance Tuning for Typhoeus

libcurl's DNS cache is per-handle. When using Hydra, DNS results are shared across queued requests, reducing DNS lookups through the proxy. For even better performance, set `dns_cache_timeout: 300` to cache DNS results for 5 minutes.

Use `followlocation: false` and handle redirects in your callback if you need to track which proxied redirects occur. Following redirects automatically can mask proxy routing issues.