Most scrapers start life as a for loop with requests.get() and some BeautifulSoup on top, and honestly that's fine for a couple hundred URLs.
The trouble starts when the list grows to 200,000 product pages and the same script needs a whole night to finish, mostly sitting idle while servers take their time responding. Two changes fix that: switching to async requests, and routing traffic through rotating proxies so a faster scraper doesn't get blocked after its first few hundred hits.
Where the Time Actually Goes
A typical request needs 300 to 800 ms to come back, and requests blocks the thread for all of it. At 500 ms a page, 1,000 pages is a little over eight minutes, and the CPU is busy for maybe five seconds of that.
Threads? They help, sort of. They also eat memory, and somewhere past a few hundred of them the GIL becomes a real nuisance.
Scrapy avoids most of this because it runs on Twisted, but for a 300-line job it's a lot of framework to learn and configure.
Going Async
asyncio became properly usable with Python 3.5, when async and await showed up in 2015. A single event loop sends requests and handles each response whenever it lands, so one process can keep hundreds of connections open with no thread per connection.
Since requests has no async support, the HTTP client has to change too. Most people go with aiohttp or httpx, and if you haven't picked yet, this roundup of the best python async http client options compares them in more detail than there's room for here.
With 50 requests in flight, those 1,000 pages take roughly 10 seconds instead of eight minutes.
The approach is older than Python's async support, by the way. Back in 1999 the C10k problem asked how one server could handle 10,000 connections at once, and event-driven I/O was the answer then too; the official asyncio documentation explains how tasks, queues and semaphores work today and is worth a read.
Speed Is How You Get Blocked
No cap on concurrency is a quick way to get banned. 2,000 requests to one domain within a second will usually earn you CAPTCHAs, blocks, or a stack of HTTP 429s (the "Too Many Requests" code from RFC 6585).
An asyncio.Semaphore handles this with very little code. Wrap each fetch in async with sem: and set a limit of 10 to 50 per domain, using the lower end on smaller sites.
Connection pools need attention too. aiohttp's TCPConnector allows 100 open connections by default and httpx allows 100 total with 20 kept alive; you can raise both, but from a single IP that just speeds up the ban.
For retries, check the Retry-After header first. If there isn't one, back off exponentially (1 second, 2, then 4) and take the proxy that got the 429 out of rotation for a few minutes.
Adding Rotating Proxies to the Mix
Async solves the speed problem, but every request still comes from the same IP, which is the main thing rate limiters track. Spreading requests across a pool keeps each address under the limit.
aiohttp accepts a proxy= argument on each call, so rotating is easy there. httpx sets proxies per client, so the common workaround is a small pool of clients with one proxy each.
Datacenter IPs are cheap and fast and work well for large catalogs, or anything else where the target isn't too picky. Residential IPs cost more per gigabyte, but they get past the stricter bot checks you'll find on a lot of retail and travel sites.
Sessions trip up plenty of teams. A scraper that logs in or goes through paginated search results should keep the same IP for the whole sequence (a "sticky" session) and only rotate between jobs, otherwise you end up with half-loaded pagination and random logouts that take a while to debug.
What Comes Next
JavaScript-heavy sites are usually the next problem. aiohttp and httpx can't render React or Vue pages, which is why Playwright, which has its own async API, keeps showing up in pipelines that used to be plain HTTP.
The basics stay the same though: cap concurrency per domain and back off when a server pushes back. With Cloudflare and Akamai tightening their bot detection every year, getting that part right is only going to matter more.
