good article - this sums it up…
you can’t serve up search results if you can’t crawl the web
and…
The internet should not be allowed to discriminate between search engine crawlers based on who they are
my personal attitude towards this differs from Mojeek - in my view, if it’s publicly available, then it ought to be indexed, period
robots.txt is a really dumb way of controlling anything - there are better and far more effective ways of controlling access and punishing abusers
i also get the ethics issue of Mojeek however - they have a reputation to protect - but again, in the end, “you can’t serve up search results if you can’t crawl the web”
that said, you can’t risk getting all your IP’s blocked either, hence my idea of having others do it for you when need be - i think it’s a potential way to remain ethical in a highly unethical environment