Hi @mike, just regarding the comment about being kicked out of other experiments, if you comma separate your FFID cookie values, you can have multiple tests enabled. For maps and summary, you’d need the value of the FFID cookie to be eiSoh7ii,B9FX3DK3.
Hope this helps!
Cheers,
Ash
This is interesting as, using the same script, I get doubled numbers in the right-hand side.
Vivaldi, I’m guessing? If you make any amendments to it (excluding the Summary tab) please let me know. As mentioned before it’s very useful for me.
I was doing research with the experimental Mojeek Summary feature and noticed a few problems that I want to share.
The issues all have one thing in common: A human cannot verify the information that the Summary is providing.
One problem I noticed was that the Summary referenced tag feeds. The obvious problem is information in a feed cannot be verifiable on a permanent basis because the page constantly changes. And the feed will only ever present generalized summaries of the articles which is not as good as drawing from the articles directly.
example.com/tags/some-topic
Another problem was that the Mojeek Summary can reference deleted pages. I’m assuming this is an inherent latency either in the Mojeek index itself or in the relationship between the index and the RAG API. Again, if the page being summarized is deleted, I can never verify the information in the Summary.
The Summary can refer to information behind a paywall. While there is nothing inherently wrong with a paywall, there is a perverse incentive here. The crawler can peek at paywalled content and summarize that. But, a person would have to pay to verify that information. Most people would just depend on the free summary in that case.
I also noticed a relatively minor issue but one which is likely to affect Mojeek users disproportionately. One of the referenced URLs relied on JavaScript to properly redirect the web browser to the article. While most people could reach the article in that circumstance, some Mojeek users have JavaScript turned off. And they might not be able to easily access the underlying source because of their preference.
I’m sure I’m getting some of the technical details wrong here. But I think my conclusions are valid.
These circumstances create claims which will show in the Mojeek Summary but which can be difficult or impossible to verify.
While these large language model problems might not be specific to Mojeek, formally launching a retrieval-augmented generation feature would mean that Mojeek is endorsing and adopting these problems as their own. At that point, you would be reserving some fraction of your resources to dealing with RAG issues. And you might not be able to work on more innovative features.
Hey Mike, thanks for this and sending in some complementary detail, in the order as listed.
Tag Feeds: Tag pages ending up high up the page is a ranking issue, so it’s one that we’ll look into (to see if there’s something which is causing these kinds of pages to end up higher than they should be).
Deleted Sources: We have some work being carried out currently in order to improve refresh rates, which should eventually fix this kind of issue, which crops up also outside of Summary.
Paywall Peeking: We can only get content from behind a paywall if the site in question has allowed us, otherwise we’ll see the same thing that a non-paying user would see. We’re going to look at this properly just to make sure nothing strange is happening behind the scenes.
Requires Javascript: Currently this tool requires JS as standard; this is not just limited to Summary as the summaries themselves are based off of the results, which do not change order based upon if Summary is included or not.
Summary is, and will be pitched as, a way that someone can see a summary of what comes from the top-ranked results.
We won’t be pitching this as an answer creation process, but rather a way that someone can quickly grab bits from the results pages before looking further.
I just recently encountered one of the issues you stated, the tag feeds. I noticed that these problems are also shared by snippets, where the phrases/sentences displayed cannot be seen in the article.
I agree, and I’m quite worried about it. On issues in Mojeek, I often get replies stating in one way or another that they haven’t been resolved due to Mojeek’s limited resources (stemming problems, malfunctioning date operators, jumbling search terms, poor quality snippets, bare-bones map, to name a few), yet here Mojeek launching another defective feature. I fear that Mojeek may be biting off more than they can chew…
These are a bit more complex than just limited resources problems, and some of them are possibly me not explaining the things themselves well enough.
Stemming: we fix issues of stemming on a case-by-case basis, from the feedback we get we wouldn’t say it’s a big problem; they are bundled and then sorted out in batches. If you have examples sitting then please send them our way.
Date Operators: As mentioned in this it’s a really thorny problem to solve at scale, for now the date operators work as described on the page, and are not included in a UI element - because if they were individuals would assume they world differently.
Jumbling Search Terms: can you give some examples? I don’t completely follow what this means.
Poor Quality Snippets: is this a reference to here / can you elaborate upon it?
Bare Bones Maps: currently still a feature that’s in testing, we’ve noted your previous suggestions.
Stemming. First, the problem is due to it being fixed on a case-to-case basis, which I think is the wrong approach, due to the following reasons:
With these, a better solution would be to give the control to users, like through checkboxes or search operators that I mentioned before.
Second, it’s breaking the main functionality of a search engine, as the stemming issues frequently pollutes the results with off-topic ones. Moreover, it breaks expectations on the touted lexical search feature, as stemming issues have a similar effect to second guessing the search terms done by other search engines, and the lack of any methods to circumvent this problem makes it quite infuriating. Due to the aforementioned reasons, this should be dealt with a sense of urgency.
Jumbling Search Terms. I mean this one, where Mojeek tend to jumble the order of the search terms, rather than prioritizing results with the correct order. As it also reduces the quality of results, there’s an urgent need for this issue to be resolved.
Poor Quality Snippets. Yes, that one. Although it’s not a core feature, 2 years seem to be too long for it to remain unresolved. Moreover, as I have explained there, quotations are superior to hallucinations, so I find it a waste that Mojeek invested on imitating the latter but not fixing the former.
Bare Bones Maps. Yeah, it’s not ready for prime time… but isn’t the summarizer in the same situation? Adding hallucinations to off-topic results is a recipe for disaster. I just find it puzzling that with such a critical issue, summarizer was promoted from an experiment to a feature.
I hope that you don’t take offense on my response. I really believe in Mojeek’s cause in becoming a privacy-preserving independent search engine, but I think that resolving issues with its basic functionality should be prioritized over experimental ones.