The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google
In my experience, these two use detection algorithms that could best be described as "pattern-based."
Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or forced to fill out a CAPTCHA.
For example, you can hit Google Search once an hour via a painfully obvious scraper no problem. But, even if you take control of a Chrome browser via selenium, and write your scraper to do everything exactly like humans down to typing in the searches and moving the mouse around and clicking results, Google is ridiculously good at identifying bot vs human traffic patterns.
I think the algorithm builds a "normal usage profile" for a combination of IP,cookie/user,device and sets a threshold, activity above that threshold gets flagged.
I can trigger Google's Captcha simply by using Chrome's Incognito Mode plus a SSH-based SOCKS proxy. No automation at all. Which is probably against the "normal usage profile" build for data Google gets from me despite Incognito Mode. Well, for starters, dropping Cookies all the time is probably an indicator. Not sure if they can somehow see my SOCKS tunnel - I hope they can't, but am not basing my security on that.
I'm assuming that your SOCKS tunnel endpoint is on a VPS, or some hosted solution, if that's the case, this probably explains it since those IPs are not the source of typical "user" web traffic. I get the same when I use a VPN.
This type of protection is also aligned with what I think of as the most legitimate concern, which is resource exhaustion. I don't understand why some site operators object to scrapers even when they are well-behaved, and collecting public information. I think if the load is low, let scrapers scrape.
I once scraped Google Search a few thousand times, in a few seconds, a few times, just for fun, and in a school (1000+ people).
It was quite funny when everyone suddenly saw a captcha.
Then a month ago or so I scraped Google+ for some statistics, maybe 1-3k times in 2-3 minutes and nothing happened.
The point is that my node.js scraper was far from perfect. Just a fake Chrome user agent, nothing else.
If a large organization with a static IP has a lot of users, Google is clever enough to figure out that it is large and has a high average number of requests by e.g. seeing that many people use their distinct Google Accounts.
In my experience, these two use detection algorithms that could best be described as "pattern-based."
Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or forced to fill out a CAPTCHA.
For example, you can hit Google Search once an hour via a painfully obvious scraper no problem. But, even if you take control of a Chrome browser via selenium, and write your scraper to do everything exactly like humans down to typing in the searches and moving the mouse around and clicking results, Google is ridiculously good at identifying bot vs human traffic patterns.
I think the algorithm builds a "normal usage profile" for a combination of IP,cookie/user,device and sets a threshold, activity above that threshold gets flagged.