Why got for Proxy-Based Node.js Workflows
got is the most feature-rich HTTP client in the Node.js ecosystem, offering built-in retry logic, pagination support, response caching, and hook-based request lifecycle management. These features align perfectly with proxy-based workflows where you need to handle transient failures gracefully, paginate through large datasets, and instrument requests for debugging. Unlike node-fetch or axios, got's retry system is proxy-aware: it can distinguish between proxy connection failures and upstream server errors, allowing you to implement different retry strategies for each.
The library's hook system lets you intercept requests at multiple lifecycle points (beforeRequest, beforeRetry, afterResponse) without monkey-patching or wrapping. This is invaluable for proxy integration because you can log proxy performance metrics, rotate IPs on block detection, and modify headers conditionally, all from a clean, declarative configuration.
Complete Configuration Example
import got from 'got';const proxyUrl = \`http://\${process.env.PROXY_USER}:\${process.env.PROXY_PASS}@gate.hexproxies.com:8080\`; const agent = new HttpsProxyAgent(proxyUrl);
const client = got.extend({ agent: { https: agent, http: agent }, timeout: { request: 30000, connect: 10000, response: 20000 }, retry: { limit: 3, statusCodes: [403, 408, 429, 500, 502, 503, 504], methods: ['GET', 'HEAD'], calculateDelay: ({ attemptCount }) => attemptCount * 2000, }, hooks: { beforeRequest: [ (options) => { options.headers['user-agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'; }, ], afterResponse: [ (response) => { console.log(\`[\${response.statusCode}] \${response.url} - \${response.timings.phases.total}ms\`); return response; }, ], }, headers: { 'accept-language': 'en-US,en;q=0.9' }, });
const response = await client.get('https://example.com'); console.log(response.body.substring(0, 200)); ```
got-Specific Proxy Features
got's \`extend\` method creates reusable client instances with preset configurations. This is ideal for proxy setups because you define your proxy agent, timeout, and retry settings once and reuse the extended client across your application. Each \`extend\` call creates a new instance without mutating the original, so you can create specialized clients for different proxy configurations (e.g., one for US-targeted requests and another for EU).
Common Pitfalls with got
The most common mistake is creating a new HttpsProxyAgent for every request. Agent creation involves TCP setup overhead, and for proxy connections this includes the CONNECT tunnel handshake. Always create the agent once and pass it to a got.extend instance. A single agent can handle hundreds of concurrent requests through the proxy gateway.
Another subtle issue is got's default retry behavior. By default, got only retries on network errors, not on HTTP status codes. For proxy-based scraping, you need to explicitly add 403 and 429 to the \`statusCodes\` retry list, otherwise blocks are treated as successful responses and not retried.
Advanced: Pagination Through Proxies
got's built-in pagination support works seamlessly through proxies. Use \`client.paginate\` to automatically follow next-page links while maintaining the same proxy agent configuration. This is particularly useful for API scraping where you need to traverse hundreds of paginated endpoints through rotating residential IPs.
Performance Considerations
got uses the native Node.js HTTP stack under the hood. For maximum throughput through a proxy, increase the global agent's \`maxSockets\` to match your concurrency target. The default value (Infinity in Node.js) is fine for direct connections but can overwhelm a proxy gateway. Set it to your plan's concurrent connection limit for predictable behavior.