Getting blocked is the single most frustrating part of web scraping. You build a scraper, it works perfectly for 50 requests, and then every response comes back as a CAPTCHA page, a 403 error, or an empty response. Your IP has been banned.
Anti-bot systems have evolved significantly. In 2026, major sites use a combination of IP reputation scoring, browser fingerprinting, behavioral analysis, and machine learning to identify scrapers. Basic techniques that worked in 2023 are no longer sufficient.
This guide covers 10 techniques that actually work against modern anti-bot systems. Each technique addresses a different detection vector, and the most effective approach combines all of them.
Understanding Why Sites Ban IPs
Before discussing solutions, it helps to understand how bans happen. Modern anti-bot systems track multiple signals:
- Request rate: Too many requests from one IP in a short time
- Request patterns: Identical timing intervals, sequential URL patterns
- IP reputation: Known datacenter IPs, previously flagged addresses
- Browser fingerprint: Missing or inconsistent browser characteristics
- Behavioral signals: No mouse movement, no scrolling, instant page navigation
- TLS fingerprint: Non-browser TLS handshake characteristics
For more on how IP reputation affects scraping, see our glossary entry on IP blacklists.
Technique 1: Use Residential Proxy Rotation
Detection vector addressed: IP reputation + request rate
This is the foundation of any anti-ban strategy. Residential proxies use IP addresses assigned to real home internet subscribers. These IPs have legitimate histories of normal browsing activity, which gives them high trust scores in IP reputation databases.
Rotating residential proxies assign a different IP for each request (or group of requests), distributing your traffic across the entire pool. With Hex Proxies, rotation is automatic:
import requests
# Default rotating mode -- each request gets a new IP
proxy_url = "http://YOUR_USERNAME:YOUR_PASSWORD@gate.hexproxies.com:8080"
proxies = {"http": proxy_url, "https": proxy_url}
for url in urls_to_scrape:
response = requests.get(url, proxies=proxies, timeout=15)
# Each request uses a different residential IPWith 10M+ residential IPs in the Hex Proxies pool, the probability of the same IP being assigned twice in a session is negligible.
Impact: This single technique prevents 60-70% of IP bans on most sites.
Technique 2: Implement Intelligent Rate Limiting
Detection vector addressed: Request rate + request patterns
Even with rotating IPs, sending requests too fast can trigger behavioral detection. Anti-bot systems track request rates across the entire proxy subnet, not just individual IPs.
The key is to make your request timing look human:
import random
import time
def human_delay():
"""Generate a random delay that mimics human browsing."""
# Base delay: 2-5 seconds (normal reading time)
base = random.uniform(2.0, 5.0)
# Occasionally longer pauses (simulating distraction)
if random.random() < 0.1: # 10% chance
base += random.uniform(5.0, 15.0)
return base
for url in urls_to_scrape:
response = requests.get(url, proxies=proxies, timeout=15)
process_response(response)
time.sleep(human_delay())Guidelines for rate limits:
| Site Type | Recommended Delay | Requests/Minute |
|---|---|---|
| E-commerce (Amazon, Walmart) | 3-8 seconds | 8-20 |
| Search engines (Google, Bing) | 5-15 seconds | 4-12 |
| Social media (Instagram, LinkedIn) | 10-30 seconds | 2-6 |
| Small/medium websites | 1-3 seconds | 20-60 |
| APIs with rate limits | Follow the API's documented limits | Varies |
Technique 3: Rotate User-Agent Strings
Detection vector addressed: Browser fingerprint
Every browser sends a User-Agent header that identifies itself. If all your requests use the same User-Agent (or worse, the default python-requests/2.x), you are trivially identifiable as a bot.
Maintain a pool of realistic, current User-Agent strings and rotate them:
import random
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0",
]
def get_headers():
return {
"User-Agent": random.choice(USER_AGENTS),
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
}Important: Include all the standard headers a real browser sends, not just User-Agent. Missing Accept, Accept-Language, or Accept-Encoding headers are obvious bot indicators.
Impact: Eliminates the most basic fingerprinting detection.
Technique 4: Handle CAPTCHAs Gracefully
Detection vector addressed: Behavioral challenge response
CAPTCHAs are a defense mechanism, not an outright ban. When you encounter one, the site is asking you to prove you are human before proceeding. How you handle CAPTCHAs determines whether your scraper recovers or dies.
def is_captcha_page(response):
"""Detect common CAPTCHA indicators."""
captcha_signals = [
"captcha",
"recaptcha",
"hcaptcha",
"challenge",
"verify you are human",
"access denied",
"please complete the security check",
]
content_lower = response.text.lower()
return any(signal in content_lower for signal in captcha_signals)
def scrape_with_captcha_handling(url):
"""Scrape with CAPTCHA detection and IP rotation."""
for attempt in range(5):
response = requests.get(url, proxies=proxies, headers=get_headers(), timeout=15)
if not is_captcha_page(response) and response.status_code == 200:
return response
# CAPTCHA detected -- wait and try with a new IP
print(f"CAPTCHA detected on attempt {attempt + 1}. Rotating IP...")
time.sleep(random.uniform(10, 30))
return None # All attempts triggered CAPTCHAsThe strategy is simple: detect the CAPTCHA, rotate to a fresh IP, and retry after a delay. With a large residential proxy pool, the new IP will not have any CAPTCHA history.
Technique 5: Use Browser-Level Automation for JavaScript Sites
Detection vector addressed: TLS fingerprint + JavaScript execution
Sites that rely on JavaScript-based anti-bot detection (Cloudflare, DataDome, PerimeterX) cannot be scraped with HTTP libraries alone. You need a real browser:
- Playwright: Best for modern sites, supports Chromium/Firefox/WebKit
- Selenium: Older but widely supported
- Puppeteer: Node.js-native, Chromium-only
See our detailed guide on setting up proxies in Playwright for step-by-step instructions.
Technique 6: Respect the Site's Structure
Detection vector addressed: Request patterns
Bots typically crawl sites in predictable patterns: sequential URLs (/page/1, /page/2, /page/3), alphabetical order, or systematic category traversal. Humans do not browse this way.
Randomize your crawl order:
import random
urls = [f"https://example.com/product/{pid}" for pid in product_ids]
random.shuffle(urls) # Randomize order
for url in urls:
response = requests.get(url, proxies=proxies, headers=get_headers(), timeout=15)
process_response(response)
time.sleep(human_delay())Also consider:
- Mix page types: Do not scrape only product pages. Visit category pages, the homepage, and about pages occasionally to simulate natural browsing patterns.
- Follow internal links: Instead of hitting direct URLs, start from a category page and follow links like a real user would.
- Vary session length: Some sessions should be 5 requests, others 50. Humans do not visit exactly 100 pages every time.
Technique 7: Manage Cookies and Sessions Properly
Detection vector addressed: Session tracking
Anti-bot systems use cookies to track behavior over time. If you make 100 requests without any cookies, that is suspicious. If you make 100 requests all sharing the same cookie that was set on the first request, the site can track your entire crawl.
Best practices:
- Accept cookies on the first request of each session
- Maintain cookies within a session (using
requests.Session()) - Start fresh sessions periodically to avoid long-lived tracking
def create_session():
"""Create a new requests session with fresh cookies."""
session = requests.Session()
session.proxies = proxies
session.headers.update(get_headers())
# Visit homepage first to get cookies
session.get("https://example.com/", timeout=15)
return session
# Rotate sessions every 20-50 requests
session = create_session()
request_count = 0
for url in urls:
if request_count > random.randint(20, 50):
session.close()
session = create_session()
request_count = 0
response = session.get(url, timeout=15)
process_response(response)
request_count += 1
time.sleep(human_delay())
session.close()Technique 8: Use the Right Proxy Type for the Target
Detection vector addressed: IP reputation
Not all proxies are equal for all targets:
| Target | Recommended Proxy Type | Why |
|---|---|---|
| E-commerce sites | Residential rotating | Diverse IPs from real ISPs avoid pattern detection |
| Social media platforms | ISP (dedicated) | Consistent IP per account prevents "suspicious login" flags |
| Search engines | Residential rotating | SERPs heavily penalize known proxy subnets |
| Small business sites | Either | Lower anti-bot investment means less detection risk |
| Sites with Cloudflare/DataDome | Residential + browser | Strongest combination against advanced protection |
Technique 9: Monitor and Adapt in Real-Time
Detection vector addressed: All vectors (continuous improvement)
Static scrapers get banned. Adaptive scrapers survive. Build monitoring into your scraper:
from dataclasses import dataclass, field
@dataclass
class ScrapeMetrics:
total_requests: int = 0
successful: int = 0
captchas: int = 0
blocks: int = 0
errors: int = 0
@property
def success_rate(self) -> float:
if self.total_requests == 0:
return 0.0
return self.successful / self.total_requests
@property
def block_rate(self) -> float:
if self.total_requests == 0:
return 0.0
return (self.captchas + self.blocks) / self.total_requests
metrics = ScrapeMetrics()
def adaptive_scrape(url: str) -> requests.Response | None:
metrics.total_requests += 1
response = requests.get(url, proxies=proxies, headers=get_headers(), timeout=15)
if response.status_code == 200 and not is_captcha_page(response):
metrics.successful += 1
return response
elif is_captcha_page(response):
metrics.captchas += 1
elif response.status_code in (403, 429):
metrics.blocks += 1
else:
metrics.errors += 1
# Adapt behavior based on metrics
if metrics.block_rate > 0.3:
print("High block rate detected. Increasing delays...")
time.sleep(random.uniform(30, 60))
elif metrics.block_rate > 0.1:
print("Moderate block rate. Adding extra delay...")
time.sleep(random.uniform(10, 20))
return NoneKey metrics to monitor:
- Success rate: Should stay above 95%. Below that, adjust your strategy.
- CAPTCHA rate: Above 5% means your fingerprint or rate needs adjustment.
- Block rate (403/429): Above 3% indicates IP reputation issues or rate limit violations.
Technique 10: Respect robots.txt and Crawl-Delay
Detection vector addressed: Legal and ethical compliance
This is not just about ethics (though it matters). Many anti-bot systems check whether scrapers respect robots.txt. If a site explicitly disallows scraping certain paths and you scrape them anyway, the site may apply more aggressive blocking.
from urllib.robotparser import RobotFileParser
def can_scrape(url: str, user_agent: str = "*") -> bool:
"""Check if a URL is allowed by the site's robots.txt."""
from urllib.parse import urlparse
parsed = urlparse(url)
robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
parser = RobotFileParser()
parser.set_url(robots_url)
parser.read()
return parser.can_fetch(user_agent, url)Respecting robots.txt also provides legal protection. In many jurisdictions, courts consider robots.txt compliance when evaluating the legality of scraping activities.
Combining All 10 Techniques: The Anti-Ban Stack
Here is how these techniques layer together for maximum effectiveness:
| Layer | Technique | Purpose |
|---|---|---|
| 1 (Foundation) | Residential proxy rotation | IP diversity and trust |
| 2 (Timing) | Intelligent rate limiting | Human-like request patterns |
| 3 (Identity) | User-Agent rotation + full headers | Authentic browser fingerprint |
| 4 (Recovery) | CAPTCHA detection and retry | Graceful failure handling |
| 5 (Engine) | Browser automation (for JS sites) | Real browser fingerprint |
| 6 (Behavior) | Randomized crawl order | Non-predictable patterns |
| 7 (State) | Session and cookie management | Realistic browsing state |
| 8 (Strategy) | Right proxy type for target | Optimized IP reputation |
| 9 (Adaptation) | Real-time monitoring | Continuous strategy adjustment |
| 10 (Compliance) | robots.txt respect | Legal protection and reduced targeting |
What to Do When You Get Banned Despite Everything
Sometimes bans happen. Here is the recovery playbook:
- Stop immediately: Do not continue sending requests from the same configuration.
- Analyze what changed: Did the site update their anti-bot? Did your request pattern change?
- Switch proxy type: If residential is getting blocked, try ISP. If a specific country pool is exhausted, try a different geo.
- Reduce concurrency: Cut your request rate by 50% and scale back up gradually.
- Review your fingerprint: Check that all headers, cookies, and browser settings are current and consistent.
Ready to Scrape Without Bans?
The most effective anti-ban strategy starts with the right proxy infrastructure. Hex Proxies provides 10M+ residential IPs across 150+ countries with automatic rotation, sticky sessions, and geo-targeting built in.
View pricing plans -- residential proxies from $4.25/GB with unlimited rotation and connections. ISP proxies with dedicated IPs and unlimited bandwidth for session-critical tasks.
For the conceptual foundation behind these tactics, see How to Avoid IP Bans When Web Scraping.