v1.13.1-4059c5c7
Skip to main content
Web ScrapingTutorial

The Beginner's Guide to Web Scraping with Proxies (Python + Requests)

12 min read

By Hex Proxies Engineering Team

Web scraping is the process of extracting data from websites programmatically. Instead of copying information by hand, you write code that visits pages, reads their content, and saves the data you need. It powers price monitoring, market research, lead generation, academic research, and hundreds of other applications. And if you plan to scrape more than a handful of pages, you need proxies.

This guide starts from zero. You will learn what web scraping is, why proxies matter, and how to build a complete scraping script in Python using the Requests library. By the end, you will have production-ready code that rotates IPs, handles errors, and respects target sites.

Why Proxies Are Essential for Web Scraping

When you scrape a website, every request comes from your computer's IP address. The target site's server sees the same IP making dozens, hundreds, or thousands of requests in a short time. This is not how normal users behave, and sites respond accordingly:

  • IP bans: Your address gets blocked, and all future requests fail.
  • CAPTCHAs: You are forced to solve challenges that break automated scripts.
  • Rate limiting: Your requests get throttled to unusable speeds.
  • Content changes: Some sites serve different (or empty) content to detected scrapers.
Proxies solve this by routing your requests through different IP addresses. Instead of 1,000 requests from one IP, the target site sees 1,000 requests from 1,000 different IPs -- each one looking like a normal user.

For a deeper look at scraping use cases, visit our web scraping guide.

What You Will Build

By the end of this tutorial, you will have a Python script that:

  1. Connects to a proxy gateway
  2. Scrapes product data from a sample website
  3. Rotates IP addresses automatically
  4. Handles errors and retries failed requests
  5. Saves results to a CSV file

Prerequisites

  • Python 3.9 or later installed on your machine
  • Basic Python knowledge (variables, functions, loops)
  • A Hex Proxies account (sign up here)

Install the required libraries

pip install requests beautifulsoup4 lxml
  • requests: Makes HTTP requests (the core of our scraper)
  • beautifulsoup4: Parses HTML and extracts data from page content
  • lxml: Fast HTML parser that BeautifulSoup uses under the hood

Step 1: Your First Request Through a Proxy

Let's start with the simplest possible example -- making a single request through a proxy:

import requests

proxy_url = "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
print(response.json())
# Output: {"origin": "203.x.x.x"}  -- a proxy IP, not your real one

What is happening here:

  1. We construct a proxy URL with your Hex Proxies credentials.
  2. The proxies dictionary tells the Requests library to route traffic through the proxy for both HTTP and HTTPS.
  3. httpbin.org/ip is a test endpoint that returns the IP address it sees -- confirming the proxy is working.
  4. The timeout=15 parameter prevents the script from hanging if the connection is slow.
If you see a proxy IP in the output (not your real IP), the proxy is configured correctly.

For more Python proxy code examples, see our Python Requests integration guide.

Step 2: Understanding Proxy Rotation

With Hex Proxies residential proxies, IP rotation is automatic. Each new request receives a different IP by default. Let's verify:

import requests

proxy_url = "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"
proxies = {"http": proxy_url, "https": proxy_url}

ips = set()
for i in range(5):
    response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
    ip = response.json()["origin"]
    ips.add(ip)
    print(f"Request {i + 1}: {ip}")

print(f"\nUnique IPs used: {len(ips)}")

You should see different IPs for each request. This automatic rotation is what makes residential proxies so effective for scraping -- the target site sees requests from diverse, real residential IPs across different ISPs and locations.

Step 3: Parsing HTML with BeautifulSoup

Now let's scrape actual content. We will parse an HTML page and extract structured data:

import requests
from bs4 import BeautifulSoup

proxy_url = "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"
proxies = {"http": proxy_url, "https": proxy_url}

# Fetch the page
response = requests.get(
    "https://books.toscrape.com/",
    proxies=proxies,
    timeout=15,
    headers={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/120.0.0.0 Safari/537.36"
    },
)

# Parse the HTML
soup = BeautifulSoup(response.text, "lxml")

# Extract book data
books = []
for article in soup.select("article.product_pod"):
    title = article.select_one("h3 a")["title"]
    price = article.select_one(".price_color").text
    rating = article.select_one("p.star-rating")["class"][1]  # e.g., "Three"
    books.append({"title": title, "price": price, "rating": rating})

for book in books[:5]:
    print(f"{book['title']} - {book['price']} ({book['rating']} stars)")

Key points:

  • User-Agent header: Always send a realistic browser User-Agent. Requests without one are immediately flagged as bots.
  • CSS selectors: soup.select() uses CSS selectors to find elements, which is more readable than navigating the DOM manually.
  • Data structure: We store each book as a dictionary, making it easy to export later.

Step 4: Scraping Multiple Pages with Pagination

Most websites spread content across multiple pages. Here is how to handle pagination:

import requests
from bs4 import BeautifulSoup
import time
import random

proxy_url = "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"
proxies = {"http": proxy_url, "https": proxy_url}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/120.0.0.0 Safari/537.36"
}

all_books = []

for page_num in range(1, 6):  # Scrape first 5 pages
    url = f"https://books.toscrape.com/catalogue/page-{page_num}.html"
    print(f"Scraping page {page_num}...")

    response = requests.get(url, proxies=proxies, headers=headers, timeout=15)

    if response.status_code != 200:
        print(f"  Failed with status {response.status_code}, skipping")
        continue

    soup = BeautifulSoup(response.text, "lxml")

    for article in soup.select("article.product_pod"):
        title = article.select_one("h3 a")["title"]
        price = article.select_one(".price_color").text
        all_books.append({"title": title, "price": price})

    # Respectful delay between requests (1-3 seconds)
    delay = random.uniform(1.0, 3.0)
    time.sleep(delay)

print(f"\nTotal books scraped: {len(all_books)}")

The random delay between requests is important. Even with rotating proxies, hammering a site with zero delay between requests is detectable and impolite. A 1-3 second randomized delay mimics human browsing behavior.

Step 5: Robust Error Handling and Retries

Production scrapers need to handle failures gracefully. Network errors, timeouts, and temporary blocks are inevitable:

import requests
from requests.exceptions import RequestException
import time

proxy_url = "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"
proxies = {"http": proxy_url, "https": proxy_url}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/120.0.0.0 Safari/537.36"
}

def fetch_with_retry(url: str, max_retries: int = 3) -> requests.Response | None:
    """Fetch a URL with automatic retries and exponential backoff."""
    for attempt in range(1, max_retries + 1):
        try:
            response = requests.get(
                url,
                proxies=proxies,
                headers=headers,
                timeout=15,
            )
            if response.status_code == 200:
                return response
            if response.status_code == 429:
                # Rate limited -- wait longer
                wait = 2 ** attempt * 5
                print(f"  Rate limited. Waiting {wait}s before retry...")
                time.sleep(wait)
                continue
            if response.status_code >= 500:
                # Server error -- retry
                print(f"  Server error {response.status_code}. Retrying...")
                time.sleep(2 ** attempt)
                continue
            # Client error (4xx other than 429) -- do not retry
            print(f"  Client error {response.status_code}. Skipping.")
            return None

        except RequestException as e:
            print(f"  Attempt {attempt} failed: {e}")
            if attempt < max_retries:
                time.sleep(2 ** attempt)

    print(f"  All {max_retries} attempts failed for {url}")
    return None

This function handles three types of failures:

  • Rate limiting (429): Waits longer between retries because the site is explicitly telling you to slow down.
  • Server errors (5xx): Retries with exponential backoff because these are usually temporary.
  • Client errors (4xx): Does not retry because the request itself is wrong (except for 429).
  • Network errors: Catches connection timeouts, DNS failures, and other transport-level issues.

Step 6: Geo-Targeted Scraping

Many websites serve different content based on the visitor's location. With Hex Proxies, you can target specific countries by appending a parameter to your username:

import requests

def scrape_from_country(url: str, country_code: str) -> str:
    """Scrape a URL using an IP from a specific country."""
    proxy_url = (
        f"http://YOUR_USERNAME-country-{country_code}:"
        f"YOUR_PASSWORD@gate.hexproxies.com:8080"
    )
    proxies = {"http": proxy_url, "https": proxy_url}

    response = requests.get(url, proxies=proxies, timeout=15)
    return response.text

# Compare content from different countries
us_content = scrape_from_country("https://example.com/products", "us")
uk_content = scrape_from_country("https://example.com/products", "gb")
de_content = scrape_from_country("https://example.com/products", "de")

This is invaluable for price comparison, content localization testing, and market research. Hex Proxies supports 93+ country locations with city-level targeting in major markets.

Step 7: Saving Results to CSV

Once you have scraped your data, save it to a CSV file for analysis:

import csv

def save_to_csv(data: list[dict], filename: str) -> None:
    """Save a list of dictionaries to a CSV file."""
    if not data:
        print("No data to save.")
        return

    fieldnames = list(data[0].keys())

    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(data)

    print(f"Saved {len(data)} records to {filename}")


# Usage
save_to_csv(all_books, "scraped_books.csv")

Step 8: Putting It All Together

Here is a complete, production-ready scraper that combines everything we have covered:

"""
Complete web scraper with proxy rotation, error handling, and CSV export.
Uses Hex Proxies for IP rotation.
"""

import csv
import random
import time
from dataclasses import dataclass

import requests
from bs4 import BeautifulSoup
from requests.exceptions import RequestException


@dataclass(frozen=True)
class BookRecord:
    title: str
    price: str
    rating: str
    url: str


PROXY_URL = "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"
PROXIES = {"http": PROXY_URL, "https": PROXY_URL}
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
BASE_URL = "https://books.toscrape.com"
MAX_RETRIES = 3
MAX_PAGES = 50


def fetch_page(url: str) -> str | None:
    """Fetch a single page with retry logic."""
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            response = requests.get(
                url, proxies=PROXIES, headers=HEADERS, timeout=20
            )
            if response.status_code == 200:
                return response.text
            print(f"  Status {response.status_code} on attempt {attempt}")
        except RequestException as e:
            print(f"  Error on attempt {attempt}: {e}")

        if attempt < MAX_RETRIES:
            time.sleep(2 ** attempt)

    return None


def parse_books(html: str) -> list[BookRecord]:
    """Extract book records from a page of HTML."""
    soup = BeautifulSoup(html, "lxml")
    records = []

    for article in soup.select("article.product_pod"):
        title = article.select_one("h3 a")["title"]
        price = article.select_one(".price_color").text.strip()
        rating = article.select_one("p.star-rating")["class"][1]
        relative_url = article.select_one("h3 a")["href"]
        full_url = f"{BASE_URL}/catalogue/{relative_url.lstrip('../')}"

        records.append(
            BookRecord(title=title, price=price, rating=rating, url=full_url)
        )

    return records


def save_results(records: list[BookRecord], filename: str) -> None:
    """Save book records to CSV."""
    if not records:
        print("No records to save.")
        return

    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["Title", "Price", "Rating", "URL"])
        for book in records:
            writer.writerow([book.title, book.price, book.rating, book.url])

    print(f"Saved {len(records)} books to {filename}")


def main() -> None:
    """Scrape all pages and save results."""
    all_books: list[BookRecord] = []

    for page_num in range(1, MAX_PAGES + 1):
        url = f"{BASE_URL}/catalogue/page-{page_num}.html"
        print(f"Scraping page {page_num}...")

        html = fetch_page(url)
        if html is None:
            print(f"  Failed to fetch page {page_num}. Stopping.")
            break

        books = parse_books(html)
        if not books:
            print(f"  No books found on page {page_num}. Reached the end.")
            break

        all_books.extend(books)
        print(f"  Found {len(books)} books (total: {len(all_books)})")

        # Respectful random delay
        time.sleep(random.uniform(1.5, 3.5))

    save_results(all_books, "books_scraped.csv")


if __name__ == "__main__":
    main()

Web Scraping Best Practices

1. Respect robots.txt

Before scraping any site, check its robots.txt file (e.g., https://example.com/robots.txt). While not legally binding in all jurisdictions, respecting these directives shows good faith and reduces your risk of legal issues.

2. Add delays between requests

Even with rotating proxies, do not hammer sites at maximum speed. A 1-3 second delay between requests is the minimum for courteous scraping. For sensitive targets, 3-5 seconds is safer.

3. Use realistic headers

Always set a User-Agent header that matches a real browser. Add Accept, Accept-Language, and other headers that browsers normally send. Missing headers are one of the first things anti-bot systems check.

4. Handle rate limiting (HTTP 429)

When you receive a 429 response, the site is telling you to slow down. Honor this by waiting before retrying, ideally with exponential backoff.

5. Choose the right proxy type

  • Residential proxies ($4.25-$4.75/GB): Best for most scraping tasks. Real residential IPs from diverse ISPs and locations.
  • ISP proxies (per-IP pricing, unlimited bandwidth): Best for speed-critical tasks or when you need a consistent IP. See ISP proxies.

6. Monitor your usage

Track your bandwidth consumption and success rates in the Hex Proxies dashboard. If your success rate drops, it may be time to adjust your delays, headers, or proxy type.

Common Mistakes Beginners Make

  1. No proxy rotation: Using a single proxy (or no proxy) for hundreds of requests guarantees bans.
  2. Missing User-Agent: The default Python Requests User-Agent (python-requests/2.x) is instantly flagged.
  3. No error handling: A single network error crashes the entire script, losing all progress.
  4. Scraping too fast: Zero delay between requests triggers rate limiting on every major site.
  5. Not saving incrementally: If your script crashes on page 47, you lose everything. Save results periodically.
  6. Ignoring response codes: A 403 or 503 response contains no useful data. Always check response.status_code before parsing.

Next Steps

You now have a solid foundation for web scraping with Python and proxies. To level up:

Ready to start scraping?

Get started with Hex Proxies -- 10M+ residential IPs across 150+ countries, automatic rotation, and bandwidth starting at $4.25/GB. Set up your first scraper in under 10 minutes with the code examples above.

Related Resources

Residential Proxies

High-quality residential proxies with rotating IPs from 100+ countries. Perfect for web scraping, data collection, and market research.

Proxies for Web Scraping

Rotating residential proxies engineered for large-scale web scraping with 99%+ success rates.

Best Proxies for Web Scraping in 2026

A data-driven guide to choosing the best proxy for web scraping. Includes testing methodology, success rate benchmarks by site category, code examples in three languages, and a site-category recommendation matrix.

How Many Proxies Do I Need? Sizing Guide by Use Case

A practical sizing guide with formulas, rules of thumb, and use-case-specific proxy calculations for web scraping, social media management, sneaker bots, ad verification, and SEO monitoring.