Cloudflare yesterday announced the private beta of “pay per crawl”.
Also announced was the option of using HTTP Message Signatures, over existing verification mechanisms for verified bot operators.
At the same time they also published their findings on a new metric called crawl-to-refer ratio as detailed in a separate post here.
What do community members think about this pay-per-crawl initiative?
I don’t operate a website being exploited by AI bots. So, I’ll never have that initial rush of relief or excitement when the current crisis ends.
But, I can’t help but notice that no one is saying you can go back to doing what you care about.
Instead, you can choose one of the paths that big business has laid out for you. And, there is no other way possible.
It just seems like one part of society created the problem. And, the same part of society solves the problem. Nothing was decided democratically. And, no one has the power to do anything other than what they want.
How long until a select few realize that more gatekeeping and more rent seeking were bad ideas?
How long until everyone says this is the way it always was?
Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content.
It’s an interesting development but do feel the same distrust that Mike says about gatekeepers.
They do have something like ~20% of Internet traffic flowing through them so may have some clout.
It would be interesting to see how they define what is an AI bot. They clearly have a handle on a dozen or so players but what about the likes of Brave? No user agent, charges for AI content on their API.
Hopefully it does not hinder Mojeek’s information discovery.