ah, needs JS, that’s why it wasn’t working for me - i thought Moj had a rather rigid anti-JS stance
1 Like
Nope this is pre-trained and there’s no continuous learning.
If possible then we’ll always try and push out non-js first, but the simplest quick way of doing this was with js. It’s an experiment at the moment and the intention is to improve as time goes by - part of that would be offering non-js.
Non-js would require the whole response to be complete, so this would be slower and we’ll need to flag this; other than that it’s quite easy to do.
1 Like
Yes this is a function of getting it out there quickly, it requires a full-page reload but we’re aware and it’s on the list.
1 Like
my initial impression after using it isn’t good - it seems there’s little difference between the Moj summary and the Wikipedia summary
the text appears to be sourced only from mainstream sources and ‘mainstream’ does not imply accuracy and in many cases is simply disinformation or misinformation
in some cases the summary reads more like an advertisement rather than general information (“covid vaccine” for ex.)
i think it would be immensely more helpful if countering POVs were included - while sourcing reliable information from non-mainstream sources may be difficult, ignoring it will, without a doubt, lead people in a very wrong direction some of the time, especially with regard to politically sensitive topics, healthcare, etc.
if Mojeek is unbiased, then so too should the summary be IMO
i can provide examples if desired, but they are political hot potatoes
I haven’t tested this summarizing feature, but I can share some thoughts regarding this feature on the practical perspective, which I hope you may find helpful in determining the direction of this project.
On my experience with Google and Brave Search, one thing AI is particularly bad at is being infoboxes. Hallucination is not actually unique to AI, but while search engines easily reveal off-topic results in a simple glance, AI summarizers create or string together wrong results that can sound plausible to someone who lacks deep knowledge about a certain topic (which, unfortunately, is often the reason someone is searching in the first place). So although a number of people love its convenience, it being muddled by too many unrecognizable false positives makes it terribly unreliable to me that I frequently just ignore AI-made infoboxes.
On the other hand, I noticed that AI excels in seeding ideas. The times when I’m having difficulty thinking of keywords due to lack of knowledge on a subject, or queries in question form that are hard to distill into keywords, AI can come handy in giving me keywords that I can use, or websites where I can start the search.
With that, I’ll mostly agree with @mike’s idea of a separate URL, like what you did with the RAG search. The referenced websites being listed along with the response make it easier to check the quality of those websites’ contents, and to spot hallucinations by comparing their contents to the response. Moreover, the related questions section helps in narrowing down vague searches.
Thank you @Videonas, @mike, @itsMe, @gnome, @snatchlightning for all this very interesting and useful feedback. Some of your questions will be answered in a blog post we will likely publish next week. At this point, I wanted to clarify some possible misunderstandings about how the summariser works. Firstly the summariser is not so different from the RAG protoype at labs.mojeek.com. The summaries on both are the pulling the same information from the search results; actually from the snippets of the top 8 ranked search results. The differences are more in the UI; both are showing the search results (only 8 in the case of labs.mojeek.com) with those results cited in the “Summary”/“Answer”.
When search results give you links to less mainstream sources, the same will be true in both these AI answers. Here is an example:
https://www.mojeek.com/search?q=What+do+we+know+about+UAPs%3F&mal=1
You may note that when you hover over a citation in the summary the corresponding results on the vertical search links is highlighted:
Here is the same result in RAG, labs.mojeek.com, with the main differences being here the addition of suggested “Related” queries and a limit of 8 results aka “Sources”:
On Google Gemini and Bard Copilot you get the following, evidently with more mainstream sources:
3 Likes
exactly, and given the focus of Mojeek, i’m not sure it’s a great fit in general, even if it is somewhat helpful some or perhaps most of the time (depending on the query)
i’ll give you folks a very good example of why the so-called “AI” the public has access to is not artificial intelligence: query RAG for “holocaust”, then ask it to “cite scientific evidence of homicidal gas chambers during world war 2”
the contradictions are obvious and this is one of the many reasons why “AI” can be quite dangerous (manipulation of public perception), yet the public is likely to view it as at least somewhat authoritative
i can provide examples all day long where these primitive chat-bots are factually and scientifically incorrect regarding many subjects because the information used to train them is also often largely and factually incorrect, but i only know this because i used search engines to navigate the weeds myself and find the necessary books, papers and other materials rather than relying on very flawed “AI” which, like Wikipedia in many cases, i see as a detour to critical thinking
“AI”, at this time, can easily amplify incorrectness and given the human biases in the algorithms, i think this is likely to only get worse as those algorithms are “adjusted” for accuracy
1 Like
It’s good to see there are a variety of views on this, and this thread shows that anything like this has to be dealt with sensitively and cater to all of the varied opinions of people using Mojeek.
Just to reinforce something up top, this is an experiment, not an indication of a change of direction for Mojeek, or any kind of change to the fabric of what we do.
Mojeek is still a keyword-based search engine (with some semantic elements) that is working hard to provide the world’s alternative in search. This is something that people will be able to use if they want to.
4 Likes
If you have the cookie and you visit a results page you should now see a Summary tab. Clicking this will take you to a page where the query is inputted into Summary and the resulting output will generate below the search box and tabs.
This has been built in order to allow easy display of and access to the sources (on the right-hand side).
As with the rest of the functionality, testing and feedback is very much welcomed and encouraged.
3 Likes
Thank you. This new layout addresses my concerns.
It looks like my user script hijacked the formatting. 
@Josh This kicks you out of other experiments like Maps. In the future, will there be a way to participate in multiple experiments? Or, are you willing to release Maps?
2 Likes
Hi @mike, just regarding the comment about being kicked out of other experiments, if you comma separate your FFID cookie values, you can have multiple tests enabled. For maps and summary, you’d need the value of the FFID cookie to be eiSoh7ii,B9FX3DK3.
Hope this helps!
Cheers,
Ash
4 Likes
This is interesting as, using the same script, I get doubled numbers in the right-hand side.
Vivaldi, I’m guessing? If you make any amendments to it (excluding the Summary tab) please let me know. As mentioned before it’s very useful for me.
I was doing research with the experimental Mojeek Summary feature and noticed a few problems that I want to share.
The issues all have one thing in common: A human cannot verify the information that the Summary is providing.
Tag Feeds
One problem I noticed was that the Summary referenced tag feeds. The obvious problem is information in a feed cannot be verifiable on a permanent basis because the page constantly changes. And the feed will only ever present generalized summaries of the articles which is not as good as drawing from the articles directly.
example.com/tags/some-topic
Deleted Sources
Another problem was that the Mojeek Summary can reference deleted pages. I’m assuming this is an inherent latency either in the Mojeek index itself or in the relationship between the index and the RAG API. Again, if the page being summarized is deleted, I can never verify the information in the Summary.
Paywall Peeking
The Summary can refer to information behind a paywall. While there is nothing inherently wrong with a paywall, there is a perverse incentive here. The crawler can peek at paywalled content and summarize that. But, a person would have to pay to verify that information. Most people would just depend on the free summary in that case.
Requires JavaScript
I also noticed a relatively minor issue but one which is likely to affect Mojeek users disproportionately. One of the referenced URLs relied on JavaScript to properly redirect the web browser to the article. While most people could reach the article in that circumstance, some Mojeek users have JavaScript turned off. And they might not be able to easily access the underlying source because of their preference.
I’m sure I’m getting some of the technical details wrong here. But I think my conclusions are valid.
These circumstances create claims which will show in the Mojeek Summary but which can be difficult or impossible to verify.
While these large language model problems might not be specific to Mojeek, formally launching a retrieval-augmented generation feature would mean that Mojeek is endorsing and adopting these problems as their own. At that point, you would be reserving some fraction of your resources to dealing with RAG issues. And you might not be able to work on more innovative features.
3 Likes
Hey Mike, thanks for this and sending in some complementary detail, in the order as listed.
Tag Feeds: Tag pages ending up high up the page is a ranking issue, so it’s one that we’ll look into (to see if there’s something which is causing these kinds of pages to end up higher than they should be).
Deleted Sources: We have some work being carried out currently in order to improve refresh rates, which should eventually fix this kind of issue, which crops up also outside of Summary.
Paywall Peeking: We can only get content from behind a paywall if the site in question has allowed us, otherwise we’ll see the same thing that a non-paying user would see. We’re going to look at this properly just to make sure nothing strange is happening behind the scenes.
Requires Javascript: Currently this tool requires JS as standard; this is not just limited to Summary as the summaries themselves are based off of the results, which do not change order based upon if Summary is included or not.
Summary is, and will be pitched as, a way that someone can see a summary of what comes from the top-ranked results.
We won’t be pitching this as an answer creation process, but rather a way that someone can quickly grab bits from the results pages before looking further.
1 Like
I just recently encountered one of the issues you stated, the tag feeds. I noticed that these problems are also shared by snippets, where the phrases/sentences displayed cannot be seen in the article.
I agree, and I’m quite worried about it. On issues in Mojeek, I often get replies stating in one way or another that they haven’t been resolved due to Mojeek’s limited resources (stemming problems, malfunctioning date operators, jumbling search terms, poor quality snippets, bare-bones map, to name a few), yet here Mojeek launching another defective feature. I fear that Mojeek may be biting off more than they can chew…
These are a bit more complex than just limited resources problems, and some of them are possibly me not explaining the things themselves well enough.
Stemming: we fix issues of stemming on a case-by-case basis, from the feedback we get we wouldn’t say it’s a big problem; they are bundled and then sorted out in batches. If you have examples sitting then please send them our way.
Date Operators: As mentioned in this it’s a really thorny problem to solve at scale, for now the date operators work as described on the page, and are not included in a UI element - because if they were individuals would assume they world differently.
Jumbling Search Terms: can you give some examples? I don’t completely follow what this means.
Poor Quality Snippets: is this a reference to here / can you elaborate upon it?
Bare Bones Maps: currently still a feature that’s in testing, we’ve noted your previous suggestions.
1 Like
Stemming. First, the problem is due to it being fixed on a case-to-case basis, which I think is the wrong approach, due to the following reasons:
- too many cases — seems like a losing battle of whack-a-mole with new names and words being produced every day
- too many subjective cases — ex. if I’m searching how to quickly do something, having results with both “quick” and “quickly” would be ok, but not if I’m searching for a title or a brand that specifically includes the word “quickly”
With these, a better solution would be to give the control to users, like through checkboxes or search operators that I mentioned before.
Second, it’s breaking the main functionality of a search engine, as the stemming issues frequently pollutes the results with off-topic ones. Moreover, it breaks expectations on the touted lexical search feature, as stemming issues have a similar effect to second guessing the search terms done by other search engines, and the lack of any methods to circumvent this problem makes it quite infuriating. Due to the aforementioned reasons, this should be dealt with a sense of urgency.
Jumbling Search Terms. I mean this one, where Mojeek tend to jumble the order of the search terms, rather than prioritizing results with the correct order. As it also reduces the quality of results, there’s an urgent need for this issue to be resolved.
Poor Quality Snippets. Yes, that one. Although it’s not a core feature, 2 years seem to be too long for it to remain unresolved. Moreover, as I have explained there, quotations are superior to hallucinations, so I find it a waste that Mojeek invested on imitating the latter but not fixing the former.
Bare Bones Maps. Yeah, it’s not ready for prime time… but isn’t the summarizer in the same situation? Adding hallucinations to off-topic results is a recipe for disaster. I just find it puzzling that with such a critical issue, summarizer was promoted from an experiment to a feature.
I hope that you don’t take offense on my response. I really believe in Mojeek’s cause in becoming a privacy-preserving independent search engine, but I think that resolving issues with its basic functionality should be prioritized over experimental ones.