TESTING: Relevancy

Testing
1

Hi Everyone,

We’ve been working on a new “algorithm” and we’d really value your help to test the results.

It’s actually about the pages that get to the algorithm stage, rather than the algorithm itself, with semantic search results introduced into the mix so longer queries, questions etc return better results.

Our tests show a dramatic improvement in a lot of cases with no real downsides, and plenty of scope for further improvement. We’d therefore appreciate your help testing, specifically, the overall relevancy of results.

There are two ways to get involved:

  1. Using Algorithm Evaluation shows two sets of results, side-by-side, with one set from the current algorithm and one set of results from the new update. Where results are the same or are both as relevant (or irrelevant), it still helps to vote them the same.

  2. You can also test using the URL parameter ‘ac=test’. If testing this way, please provide whatever feedback you can via any method you prefer (e.g. on here, pm, email or feedback form).

Oh, and expect some quirks.

We hope you enjoy testing and thanks in advance for your help!

Team Mojeek

3 Likes
2

Would it be possible to share the queries where improvements were seen? I tried a dozen queries and didn’t see any difference in the algo evaluation page.

1 Like
3

Yes, of course, please find below…

The improvements can return more relevant results but we haven’t fully implemented the update. With your help, we want to make sure that only relevant results are added, rather than semantically similar but irrelevant results, as there’s a lot of scope for continued improvement.

Here are some examples from OpenEvals/SimpleQA · Datasets at Hugging Face :

  • In the last episode of the first season of Happy Valley, in what type of mechanized vehicle does Catherine find her grandson Ryan along with his father Tommy?

The original results are off topic. The second result in the new algorithm provides the answer and the results are, in general, more relevant.

  • On what day, month, and year was Noel Turner, a Maltese footballer, born?

The original results are off topic. The new algo provides only one extra result at the top, which provides the answer.

  • What was Jitendra Kumar Maheshwari’s position just before being appointed as a judge of the Supreme Court of India?

The original results are irrelevant. The new algo provides the answer in the top result and they are, in general, more relevant.

Hope that helps and thanks for your help with testing.