Hey @all, I’m not going to give out too much detail about exactly what it is targeted so as to not bias assessments of it, but there is a new algorithm up on the Evaluation Page - Algorithm Evaluation which is testing a slightly-more semantic approach when this is likely to be beneficial.
For anyone not aware of this tool, it presents results for a new testing algorithm up against the current one for a given query, allowing you to vote on which set of results is better.
This change is something we’re very keen to get tested. Often results will be very similar, but you can mark those as “Even”. If you thinks one is much better then please mark it as such. What we really need to know is does it make any or many searches worse, or is this change mostly better.
i get that, but on the flip side, it seems very difficult to test if one doesn’t know what’s being tested - the several queries i tried turned out to be essentially even
i think it would help if some general information were provided regarding what is being tested; are you testing “phrase search accuracy” simple keyword searches, numbers, “quoted” words vs. unquoted words, hyphenated-words, etc.
I have always been very interested in semantic search. I’ve been chipping away at my own solutions for it, in code and on paper. I’m excited to hear that Mojeek is experimenting with it!
Read and received, I guess more semantic and we’re looking for anything which seems massively amiss as this is near the final stage… that’s not a large amount more information, but I can think on it next time one of these comes up to see if there’s a better way of informing whilst keeping our tests somewhat unbiased
I’ll admit here again that I am not being too helpful. I guess one thing which could be of use is repeating a search you can recall that did something positive. We’re reasonably confident the change makes things better on average, but the real thing we want to know is if we’ve produced anything that we haven’t seen yet and that is a lot worse.
IF any of those exist then it’s a good thing to dig into.
