Web Scraping with Rotating Proxies

A Python web scraper with rotating proxies solves the problem every scraper eventually hits: your requests all come from one IP, that IP gets rate-limited or blocked, and your collection stops. This guide puts a scraper on a VPS, routes it through rotating residential proxies, and adds the retry and backoff logic that separates a scraper that runs for weeks from one that dies overnight.

Using an AI coding agent? There's a ready-made prompt at the end of this guide. Copy that instead of this article.

Before you start: scrape responsibly

This part isn't boilerplate. Most scrapers that get blocked deserved it, and the practices that keep you unblocked are the same ones that keep you on the right side of the line.

• Check robots.txt and the site's terms. They tell you what the operator considers acceptable. Ignoring them is how you end up in a legal conversation.
• Public data only. Anything behind a login is governed by the terms you accepted to get that login.
• Don't collect personal data unless you have a lawful basis and have thought properly about GDPR or your local equivalent. "It was publicly visible" is not a lawful basis.
• Rate-limit yourself. A polite scraper making one request every few seconds is nearly invisible and rarely blocked. A scraper hammering a site as fast as it can is a denial-of-service problem with a friendlier name.
• Identify yourself with a real User-Agent, and give the operator a way to contact you if you're running anything substantial.

Rotating proxies exist so that legitimate collection isn't throttled by crude per-IP limits. They are not a way to ignore a site that has clearly told you to stop.

What you'll need

• A PrivateByte VPS. The Flare plan ($5.99/mo: 1 vCPU, 2 GB RAM, 25 GB SSD) is plenty. A scraper spends nearly all its time waiting on the network, not computing.
• A PrivateByte proxy API key from the Proxies section of the dashboard, with some balance on it. Proxies are pay-as-you-go per GB. There's no subscription and no minimum.
• Python basics. You should be comfortable reading a for loop.
• About 25 minutes.

Step 1: Deploy your VPS

In the dashboard, open the store, choose Flare, pick Ubuntu 24.04, and deploy. Ready in under 60 seconds.

Running the scraper on a VPS rather than your laptop matters for the same reason it does for a bot: it keeps running when you close the lid, and it has a stable connection that doesn't drop when you move between networks.

Step 2: Connect over SSH

ssh root@YOUR_SERVER_IP

Step 3: Set up Python

sudo apt update && sudo apt upgrade -y
sudo apt install -y python3 python3-pip python3-venv git
mkdir -p ~/scraper && cd ~/scraper
python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 python-dotenv

The virtual environment keeps your project's packages separate from the system Python, which stops a future apt upgrade breaking your scraper.

Step 4: Get your proxy API key

In the dashboard, open Proxies. Your API key is on that page. It looks like pb_ followed by a long hex string. Copy it.

Store it as an environment variable rather than putting it in your code:

nano ~/scraper/.env

PROXY_KEY=pb_your_key_here

chmod 600 ~/scraper/.env

Anyone with that key can spend your proxy balance. Treat it like a card number, never commit it, never paste it into an issue tracker.

Step 5: Understand the proxy endpoint

The gateway takes standard HTTP proxy authentication. Your API key is the username and the tier is the password:

http://API_KEY:[email protected]:3128

There's a SOCKS5 listener on port 1080 with the same credentials, if your stack prefers it.

The tiers, and what each is for:

All five sit on the same endpoint and draw from one balance, the tier is a per-request choice, not a separate product or a separate key.

One important gotcha. If you send a tier name the gateway doesn't recognise, it does not return an error. It falls back to your account's registered tier and proceeds. So a typo means you're billed for one tier while believing you tested another, silently. Copy the keywords from the table exactly. Note that cleanproxy_geo uses an underscore.

To pin a country, append _country-XX to the tier, with an uppercase two-letter code:

http://API_KEY:[email protected]:3128

Country routing is included on every tier at no extra cost.

Step 6: Write the scraper

nano ~/scraper/scrape.py

import os
import time
import random
import requests
from bs4 import BeautifulSoup
from dotenv import load_dotenv

load_dotenv()

PROXY_KEY = os.getenv("PROXY_KEY")
if not PROXY_KEY:
    raise SystemExit("PROXY_KEY is not set, check your .env file")

TIER = "standard"          # see the tier table; exact keywords only
COUNTRY = ""               # e.g. "GB" to pin a country, or "" for any

ENDPOINT = "proxy.privatebyte.com:3128"
USER_AGENT = "Mozilla/5.0 (compatible; MyScraper/1.0; +https://example.com/bot)"


def proxy_url() -> str:
    password = f"{TIER}_country-{COUNTRY}" if COUNTRY else TIER
    return f"http://{PROXY_KEY}:{password}@{ENDPOINT}"


def fetch(url: str, attempts: int = 4) -> str | None:
    """Fetch a URL through the proxy, retrying with backoff.

    Each attempt gets a new exit IP, because the gateway rotates per
    connection. So a retry is not just 'try again'. It is 'try again
    from somewhere else', which is what makes retries worth doing.
    """
    proxies = {"http": proxy_url(), "https": proxy_url()}

    for attempt in range(1, attempts + 1):
        try:
            response = requests.get(
                url,
                proxies=proxies,
                headers={"User-Agent": USER_AGENT},
                timeout=30,
            )

            if response.status_code == 200:
                return response.text

            # 429 and 5xx are worth retrying. 404 and 403 usually are not:
            # retrying a 403 twenty times just spends your balance.
            if response.status_code in (429, 500, 502, 503, 504):
                print(f"  attempt {attempt}: HTTP {response.status_code}, retrying")
            else:
                print(f"  attempt {attempt}: HTTP {response.status_code}, giving up")
                return None

        except requests.RequestException as exc:
            print(f"  attempt {attempt}: {type(exc).__name__}, retrying")

        # Exponential backoff with jitter. The jitter matters: without it,
        # every retry in a parallel scraper fires at the same instant.
        time.sleep((2 ** attempt) + random.uniform(0, 1))

    return None


def main() -> None:
    urls = [
        "https://example.com/page-1",
        "https://example.com/page-2",
    ]

    for url in urls:
        print(f"fetching {url}")
        html = fetch(url)

        if html is None:
            print("  failed, moving on")
            continue

        soup = BeautifulSoup(html, "html.parser")
        title = soup.title.string.strip() if soup.title else "(no title)"
        print(f"  title: {title}")

        # Be polite. This single line is the difference between a scraper
        # that runs for months and one that gets the whole range blocked.
        time.sleep(random.uniform(2, 5))


if __name__ == "__main__":
    main()

Two design choices worth understanding rather than copying blindly.

Retries get a new IP for free. The gateway rotates the exit IP per connection, so attempt two comes from a different address than attempt one. That's why a retry is worth making at all, retrying from the same blocked IP would just fail identically.

Not every failure is worth retrying. A 429 means "slow down" and will likely succeed later. A 403 usually means something about the request itself is wrong, and retrying it twenty times spends your balance to learn nothing. The code distinguishes the two.

Verify it works

First, prove the proxy is being used at all, and prove it's rotating:

cd ~/scraper && source .venv/bin/activate
python3 - <<'EOF'
import os, requests
from dotenv import load_dotenv
load_dotenv()
key = os.getenv("PROXY_KEY")
p = f"http://{key}:[email protected]:3128"
for i in range(5):
    ip = requests.get("https://api.ipify.org",
                      proxies={"http": p, "https": p}, timeout=30).text
    print(f"request {i+1}: {ip}")
EOF

You should see five different IP addresses. That single output confirms three things at once: your key works, traffic is going through the gateway, and rotation is happening.

Now the control, which is the part most people skip. Run the same request without the proxy:

curl -s https://api.ipify.org; echo

That should print your server's own IP, and it must be different from all five above. If it matches, your requests were never going through the proxy, and a scraper that silently bypasses its proxy looks like it's working right up until the target blocks your server.

Then run the scraper itself:

python3 scrape.py

Finally, check the Proxies page in your dashboard. Usage should have gone up by a small amount, and the per-tier breakdown should show it on the tier you actually intended. That's the check that catches a mistyped tier keyword.

Troubleshooting

407 Proxy Authentication Required. The API key is wrong, or the .env didn't load. Check for a trailing space or a stray quote in .env, and confirm the key starts with pb_.

Every request returns the same IP. You're probably reusing a single requests.Session, which holds one connection open. Rotation happens per connection. Either create a fresh session per request, or accept that a session is sticky by design, which is sometimes exactly what you want.

Usage shows on a different tier than you expected. A mistyped tier keyword. The gateway doesn't error on an unknown tier: it falls back to your registered one. Check the spelling against the table above, cleanproxy_geo with an underscore is the usual casualty.

Everything returns 403 no matter which tier you use. The target is likely behind an active anti-bot system, Cloudflare Turnstile, DataDome and similar. Raw HTTP proxies don't defeat those, ours or anyone's, and no amount of tier-switching will change it. That's a different class of tool; don't burn balance discovering it.

Requests are slow. Residential proxies route through real consumer connections, so 1–3 seconds per request is normal and not a fault. If throughput matters more than IP quality, reduce per-request overhead rather than expecting residential latency to improve.

Balance disappearing faster than expected. Check the per-tier breakdown in the dashboard. The usual cause is a script left on Mobile at $10/GB after a test, or retrying non-retryable errors in a loop. Both show up immediately as spend concentrated on one tier.

Do it with an AI agent

If you'd rather hand this to Claude Code, Cursor, or another coding agent, don't paste the article at it. Articles are written for humans, and agents skim the warnings and lose the ordering. Copy this instead, and run it from your own machine with your agent able to SSH out.

You are helping me set up a Python web scraper on a fresh Ubuntu 24.04 VPS, routed
through PrivateByte rotating residential proxies.

FILL IN BEFORE YOU START:
- SERVER_IP   = <your VPS IP from the PrivateByte dashboard>
- TARGET_URLS = <the URLs I want to scrape>
- TIER        = standard | cleanproxy | cleanproxy_geo | sticky_clean | mobile
- COUNTRY     = <two-letter uppercase code, or leave empty for any>

WHAT TO DO:
1. SSH to root@SERVER_IP. Confirm it's Ubuntu 24.04 before you change anything.
2. Install python3, python3-pip, python3-venv and git. Create ~/scraper, make a
   venv inside it, and install requests, beautifulsoup4 and python-dotenv.
3. Create ~/scraper/.env containing PROXY_KEY. ASK ME for the key, do not guess
   it and do not look for it anywhere else. chmod 600 the file.
4. Write ~/scraper/scrape.py that:
   - reads PROXY_KEY from .env and exits clearly if it is missing
   - builds the proxy URL as http://KEY:[email protected]:3128
     (if COUNTRY is set, the password becomes TIER_country-COUNTRY)
   - fetches each URL with a 30s timeout and a real User-Agent
   - retries ONLY on 429 and 5xx, with exponential backoff plus random jitter
   - does NOT retry 403 or 404. Those waste my balance and prove nothing
   - sleeps a random 2-5 seconds between URLs
   - parses the page with BeautifulSoup and prints what it found

RULES:
- Never print, echo, cat or log the proxy API key. It is a live billable
  credential. To confirm it is set, check the variable is non-empty. Never check
  what it contains.
- Use the tier keyword EXACTLY as I gave it. An unknown tier name does not
  error. The gateway silently falls back to my account's registered tier, so a
  typo means
  I get billed for one tier while thinking I tested another.
- Do not raise the request rate, remove the sleep, or add concurrency unless I ask.
  Politeness is deliberate here, not an oversight.
- Do not scrape anything I did not list in TARGET_URLS.
- Nothing destructive. If ~/scraper already exists, stop and ask.

VERIFY, AND SHOW ME THE OUTPUT OF EACH:
- Make 5 requests to https://api.ipify.org through the proxy -> five DIFFERENT
  IP addresses, proving rotation works
- Run "curl -s https://api.ipify.org" with NO proxy -> the server's own IP, and it
  must differ from all five above. This is the control: without it, a scraper that
  silently bypasses the proxy looks identical to one that works.
- Run scrape.py against TARGET_URLS -> it completes and prints parsed output
- Tell me to check the Proxies page in my dashboard and confirm usage landed on
  the tier I chose, not a fallback tier

Do not tell me a step succeeded without showing the command output that proves it.
If a verification fails, stop and report the actual error. Do not retry silently
and do not improvise a workaround.

The control in that verification block is the part worth carrying into your own work. "Five different IPs" alone doesn't prove the proxy is being used. It proves something returned five values. Pairing it with an unproxied request that must return a different IP is what makes the test able to fail.

Deploy your VPS

A scraper is a good VPS workload for the same reason a bot is: it needs to keep running when your laptop doesn't, and it wants a connection that doesn't move.

The Flare plan is the most cost-effective option for this, since a scraper spends nearly all its time waiting on the network rather than using CPU.

Proxies are separate and pay-as-you-go from $3/GB, with no subscription, no minimum and no monthly reset. All five tiers share one balance and one endpoint, so you can switch per request without provisioning anything new.

Common questions

Do I need proxies to scrape? Not always. For small volumes against a site that doesn't mind, your server's own IP is fine. You need rotating proxies once per-IP rate limits become the thing stopping you, which usually arrives sooner than people expect.

Which tier should I start with? standard. Move up only when you have evidence you need to. If a target is rejecting you specifically on IP reputation, try cleanproxy; if it's a mobile-only platform, mobile. Starting on an expensive tier "to be safe" just spends more for a result the cheap tier would have given you.

Will this get past Cloudflare? No, and be sceptical of anyone claiming otherwise about raw HTTP proxies. Active anti-bot systems like Cloudflare Turnstile and DataDome challenge the browser itself, not the IP. A proxy changes where your request comes from, not what it looks like. Those targets need a different class of tool.

Is web scraping legal? It depends on what you collect, where you are, and what the site's terms say. Scraping publicly available, non-personal data is broadly accepted in many jurisdictions; collecting personal data, bypassing a login, or ignoring an explicit prohibition is a very different question. If anything you're planning gives you pause, that's worth resolving before you build it, not after.