Proxies for Kubernetes Deployments

Kubernetes workloads that need to access external services through proxy infrastructure require proper configuration through Secrets, environment injection, and optionally sidecar containers. This guide covers production-ready patterns for integrating Hex Proxies into K8s deployments.

Secret Configuration

Store proxy credentials in a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: hex-proxy-credentials
  namespace: scraping
type: Opaque
stringData:
  PROXY_USER: "YOUR_USERNAME"
  PROXY_PASS: "YOUR_PASSWORD"
  PROXY_URL: "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"

Basic Deployment with Proxy

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-scraper
  namespace: scraping
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-scraper
  template:
    metadata:
      labels:
        app: web-scraper
    spec:
      containers:
        - name: scraper
          image: your-registry/scraper:latest
          env:
            - name: HTTP_PROXY
              valueFrom:
                secretKeyRef:
                  name: hex-proxy-credentials
                  key: PROXY_URL
            - name: HTTPS_PROXY
              valueFrom:
                secretKeyRef:
                  name: hex-proxy-credentials
                  key: PROXY_URL
            - name: NO_PROXY
              value: "localhost,127.0.0.1,.cluster.local,.svc"
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"

Geo-Distributed Workers with ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: proxy-regions
  namespace: scraping
data:
  US_PROXY: "http://YOUR_USER-country-us:YOUR_PASS@gate.hexproxies.com:8080"
  EU_PROXY: "http://YOUR_USER-country-de:YOUR_PASS@gate.hexproxies.com:8080"
  APAC_PROXY: "http://YOUR_USER-country-jp:YOUR_PASS@gate.hexproxies.com:8080"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scraper-us
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: scraper
          image: your-registry/scraper:latest
          envFrom:
            - secretRef:
                name: hex-proxy-credentials
          env:
            - name: HTTP_PROXY
              valueFrom:
                configMapKeyRef:
                  name: proxy-regions
                  key: US_PROXY
            - name: WORKER_REGION
              value: "US"

Sidecar Proxy Pattern

For applications that cannot be configured with HTTP_PROXY environment variables, use a sidecar container that runs a local proxy forwarder:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scraper-with-sidecar
spec:
  template:
    spec:
      containers:
        - name: app
          image: your-registry/app:latest
          env:
            - name: HTTP_PROXY
              value: "http://localhost:3128"
        - name: proxy-sidecar
          image: your-registry/proxy-forwarder:latest
          ports:
            - containerPort: 3128
          env:
            - name: UPSTREAM_PROXY
              valueFrom:
                secretKeyRef:
                  name: hex-proxy-credentials
                  key: PROXY_URL

CronJob for Scheduled Scraping

apiVersion: batch/v1
kind: CronJob
metadata:
  name: scheduled-scraper
  namespace: scraping
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: scraper
              image: your-registry/scraper:latest
              envFrom:
                - secretRef:
                    name: hex-proxy-credentials
              env:
                - name: HTTP_PROXY
                  valueFrom:
                    secretKeyRef:
                      name: hex-proxy-credentials
                      key: PROXY_URL
          restartPolicy: OnFailure

Horizontal Pod Autoscaler

Scale scraping workers based on queue depth:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: scraper-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-scraper
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: redis_queue_length
        target:
          type: AverageValue
          averageValue: "100"

Network Policy

Restrict scraper pods to only communicate with the proxy gateway and internal services:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: scraper-egress
  namespace: scraping
spec:
  podSelector:
    matchLabels:
      app: web-scraper
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - port: 8080
          protocol: TCP
    - to:
        - namespaceSelector:
            matchLabels:
              name: scraping

Production Considerations

Hex Proxies processes 50 billion requests per week across 800TB daily. Our infrastructure scales horizontally just like your Kubernetes cluster. Each pod gets a unique proxy session, ensuring IP diversity across your worker fleet. ISP proxies deliver sub-50ms latency, keeping your K8s workloads responsive.