Proxies for Competitive Intelligence

Proxies for Competitive Intelligence

Competitive intelligence requires consistent, reliable access to competitor websites, pricing pages, product catalogs, and content. Without proxies, your monitoring IPs get blocked within days — leaving you blind to market changes. Proxy infrastructure makes competitive intelligence sustainable at scale.

What to Monitor

Price Monitoring System

from dataclasses import dataclass
from datetime import datetime
import httpx
from bs4 import BeautifulSoup

@dataclass(frozen=True)
class PriceSnapshot:
    competitor: str
    product: str
    price: str
    currency: str
    url: str
    timestamp: str

def monitor_competitor_price(
    url: str,
    competitor: str,
    product: str,
    price_selector: str,
    proxy: str,
) -> PriceSnapshot:
    """Fetch competitor page and extract pricing."""
    with httpx.Client(proxy=proxy, timeout=30) as client:
        resp = client.get(url, headers={
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
            "Accept": "text/html,application/xhtml+xml",
            "Accept-Encoding": "gzip, deflate, br",
        })
        soup = BeautifulSoup(resp.text, "html.parser")
        price_el = soup.select_one(price_selector)
        price_text = price_el.text.strip() if price_el else "not found"

        # Extract currency and amount
        currency = "USD"
        for symbol, code in [("
quot;, "USD"), ("€", "EUR"), ("£", "GBP")]:
            if symbol in price_text:
                currency = code
                break

        return PriceSnapshot(
            competitor=competitor,
            product=product,
            price=price_text,
            currency=currency,
            url=url,
            timestamp=datetime.utcnow().isoformat(),
        )

Content Change Detection

import hashlib
from dataclasses import dataclass

@dataclass(frozen=True)
class ContentChange:
    url: str
    previous_hash: str
    current_hash: str
    changed: bool
    detected_at: str

class ContentMonitor:
    def __init__(self, proxy: str):
        self._proxy = proxy
        self._hashes: dict[str, str] = {}

    def check(self, url: str) -> ContentChange:
        with httpx.Client(proxy=self._proxy, timeout=30) as client:
            resp = client.get(url, headers={
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
                "Accept-Encoding": "gzip, deflate, br",
            })
            current_hash = hashlib.sha256(resp.text.encode()).hexdigest()
            previous_hash = self._hashes.get(url, "")
            changed = previous_hash != "" and previous_hash != current_hash
            self._hashes = {**self._hashes, url: current_hash}
            return ContentChange(
                url=url,
                previous_hash=previous_hash,
                current_hash=current_hash,
                changed=changed,
                detected_at=datetime.utcnow().isoformat(),
            )

Geo-Targeted Competitive Analysis

Monitor how competitors display different pricing or content by region:

def multi_geo_price_check(
    url: str,
    competitor: str,
    price_selector: str,
    username: str,
    password: str,
    countries: list[str],
) -> list[PriceSnapshot]:
    """Check competitor pricing from multiple countries."""
    results = []
    for country in countries:
        proxy = f"http://{username}-country-{country.lower()}:{password}@gate.hexproxies.com:8080"
        snapshot = monitor_competitor_price(url, f"{competitor} ({country})", "product", price_selector, proxy)
        results = [*results, snapshot]
    return results

# Check pricing from 5 countries
snapshots = multi_geo_price_check(
    url="https://competitor.com/pricing",
    competitor="CompetitorX",
    price_selector=".price-amount",
    username="YOUR_USER",
    password="YOUR_PASS",
    countries=["US", "GB", "DE", "JP", "AU"],
)

Scheduling and Alerting

Run monitoring jobs on a schedule and alert on significant changes:

from dataclasses import dataclass

@dataclass(frozen=True)
class Alert:
    type: str
    competitor: str
    message: str
    timestamp: str

def check_price_change(previous: PriceSnapshot, current: PriceSnapshot) -> Alert | None:
    if previous.price != current.price:
        return Alert(
            type="price_change",
            competitor=current.competitor,
            message=f"Price changed from {previous.price} to {current.price}",
            timestamp=current.timestamp,
        )
    return None

Legal and Ethical Considerations

Competitive intelligence through publicly available data is a standard business practice. However: - Only access publicly available pages — do not circumvent authentication - Respect robots.txt directives - Do not overload competitor servers with excessive requests - Use the data for internal analysis only, not redistribution

Hex Proxies operates a residential network, sourced under partner agreements with consent and compensation frameworks, for geographic diversity in multi-market analysis, plus ISP proxies that deliver the sub-50ms latency real-time price monitoring needs.