20 Percent of All ‘Ok’ Web Responses AI Gets Are Not Okay

PUBLISHER ARTICLE PREVIEW

From the original article

My recent experiments in AI-assisted web-scraping have brought home to me the growing extent to which the web is becoming ‘locked-down’. The motivating event for this retrenchment, of course, is the shocking growth in AI web-scraping, whose uncompromising demand for new data can end up taking domains offline.

This is a good excuse for domains to do what many of them have wanted to do for a long time: make their content accessible only when the user is logged in (though naturally, Googlebot typically gets a free pass). Facebook completed its gradual adoption of this policy in 2023, and Twitter/X did likewise the same year.

Reddit will adopt it later this month.

In this way, compliance with any country-specific age-limitation enforcement becomes a little easier for the domain, since the user is already registered. More importantly, the user’s data can be saved, exploited for local analytics, and – local legislation permitting – sold to data brokers and AI companies.

Elsewhere, measures squarely aimed at defending against bots (rather than enrolling users) have risen more than 58% year-on-year, according to Cloudflare; and all that escalating bot activity naturally encourages websites to scrutinize VPN and proxy addresses, potentially denying legitimate users access, together with any ‘malicious’ traffic.

The anecdatal aspect of this is that I seem to come up against ‘blocking’ responses or challenges more and more, to the point where I may even abandon a site that begins loading them; and since CAPTCHA -walls have been found to deter users and potential customers, one can imagine the pressure that causes domain owners to adopt them in spite of that downside.

All this new and increased friction is partly logistical (because not all domains can withstand the onslaught) but also often a kind of negotiating tactic to see if AI companies will offer a single or collective form of remuneration for all that data they so desperately crave, and which has been free for so long, until now.

Since AI crawlers have to work at scale, they tend to use ‘headless’ instances of Chrome under the Playwright framework.

What’s wanted is the ‘200’ (‘OK’) client response, even if one has to hand over a pile of money to a residential proxy farm, in concert with other diverse non-human sneaky tricks, in order to obtain it.

There’s nothing better than the 200 response; which is to say, there is no client response code with a higher level of positive signal. Neither is it an ambiguous response; it’s never ‘nearly okay’, ‘going to be okay later’, or ‘getting more okay’; it’s ‘okay!’. If it’s not okay, it’s not a 200 response.