How to Scrape a Website Without Getting Blocked (2026 Guide)

Last Updated: June 2026 · Written by DigiMetrics Hub Team · 8 min read

If your scraper worked for ten minutes and then started getting 403s, CAPTCHAs, or empty pages, you are not alone. Modern websites use Cloudflare, Akamai, and DataDome to spot bots in milliseconds. The good news: you can scrape almost any public site reliably if you behave like a real visitor. Here is the 2026 playbook.

Why Websites Block Scrapers in the First Place

Sites block bots to protect servers, prevent price scraping by competitors, defend against credential stuffing, and stay compliant with terms of service. The block does not mean the data is illegal to access — it means your traffic pattern looked suspicious.

• Too many requests in a short window from one IP.
• Missing or default User-Agent (python-requests/2.x is a giveaway).
• No referer, no cookies, no Accept-Language header.
• Same request order on every page — humans click around randomly.
• Headless browser fingerprints like navigator.webdriver = true.

Step 1 — Throttle Your Requests

Speed is the single biggest reason scrapers get blocked. A real user reads a page for at least a few seconds before clicking. Add a randomized delay between 1 and 4 seconds. For large jobs, schedule them across hours instead of slamming the site in five minutes.

Step 2 — Send Realistic Headers

Browsers send a rich set of headers on every request. Match them. At minimum, include User-Agent, Accept, Accept-Language, Accept-Encoding, and a plausible Referer that matches the navigation path.

1. Open the target site in Chrome DevTools → Network tab.
2. Copy the request as cURL.
3. Paste those exact headers into your scraper.
4. Rotate User-Agent strings every few requests.

Step 3 — Rotate IPs With Proxies

One IP making thousands of requests is the clearest bot signal there is. Use a proxy pool so each request (or each session) comes from a different address.

• Datacenter proxies — cheapest, blocked the fastest.
• Residential proxies — real consumer IPs, hardest to detect.
• Mobile proxies — best for Instagram, TikTok, and other mobile-first apps.
• Tor — free, but slow and increasingly fingerprinted.

Step 4 — Respect robots.txt and Rate Limits

Fetching /robots.txt before scraping is both ethical and practical. Disallowed paths are often the ones with the most aggressive bot detection, so honoring them keeps you out of trouble and off the block list.

Step 5 — Handle JavaScript-Rendered Pages

If the data only appears after the page runs JavaScript (React, Vue, Next.js sites), a plain HTTP request returns an empty shell. Use a headless browser:

• Playwright — modern, cross-browser, great for stealth scraping.
• Puppeteer — Chrome-only, huge ecosystem, easy stealth plugins.
• Selenium — older, slower, still works for legacy stacks.

Step 6 — Solve or Avoid CAPTCHAs

If you are hitting reCAPTCHA or hCaptcha, you have already been flagged. Solving services like 2Captcha and Anti-Captcha cost roughly $1–3 per 1,000 solves. The better long-term fix is to look less like a bot in the first place — slower requests, better headers, residential IPs.

Step 7 — Cache Aggressively

Most scraping jobs re-fetch pages that have not changed. Cache responses locally with a hash of the URL and an ETag or Last-Modified check. You scrape less, get blocked less, and finish faster.

Tools That Make Stealth Scraping Easier

• Playwright + playwright-stealth.
• puppeteer-extra with the stealth plugin.
• Scrapy + scrapy-rotating-proxies.
• ScraperAPI, ScrapingBee, ZenRows — managed scraping with proxy + CAPTCHA solving.
• Crawlee (Apify) — open-source, modern, batteries included.

Common Mistakes That Get You Blocked

• Scraping from a single static IP at full speed.
• Leaving the default Python or Node User-Agent string.
• Hitting the same URL pattern in perfect order.
• Ignoring HTTP 429 (rate limit) responses — back off immediately.
• Running headless Chrome with no stealth patches.

Frequently Asked Questions

How many requests per second is safe?

Aim for 1–2 requests per second per IP for most sites. Large sites like Amazon or Google can take more, but new or small sites often rate-limit at 1 request every few seconds.

Should I use a VPN or a proxy for scraping?

A VPN gives you one new IP — fine for testing, useless at scale. Proxy pools give you hundreds or thousands of rotating IPs, which is what you actually need to scrape without getting blocked.

Does using a headless browser slow scraping down?

Yes — by 5–10x compared to plain HTTP requests. Only use a headless browser for pages that genuinely require JavaScript rendering. For everything else, plain requests with good headers are faster and cheaper.