One Weird Trick For Fighting Search Engine Bots

I heard Colin mention in his talk on OSSYM23 that Mojeek (like every search engine) is struggling with bot traffic. I’ve discovered an incredibly stupid technique that’s unreasonably good at filtering spam queries, not all of them but catches a bunch of stuff that otherwise gets through e.g. Cloudflare.

I don’t want to spell it out in a public forum like this, the walls have eyes; but I’ve been sharing it with other search engines and many have seen success in filtering out some of the bots with the heuristic. Direct message me and I’ll share the finding.

6 Likes

Very nice of you to offer to share this with other independent search engines.

1 Like

Thanks @vlofgren for offering to help, and getting in touch with the details. Also thanks for your talk at OSSYM23. It’s great to have you here, and as a fellow builder of alternative search engines.