I haven’t done extensive testing and I was holding off commenting before I’d used it for more than half an hour, but I’ll just comment some of my thoughts now.
I think this feature is a good idea.
I’ve had trouble navigating Mojeek results because while the page I want is usually there, it’s usually not ranked at the top. The summary is a quick way of getting the answers I want and identifying the pages that have those answers.
I like that it’s on the side and not at the top.
I like that you need to click it the first time to summarize it.
I’ve observed that when you click it, the whole page reloads instead of just the Summary box. It didn’t do this initially. I’m not sure what the reason is, because the Summary feature doesn’t work without Javascript (which is fine, by the way). Is it just so you can share the page with the Summary attached?
Anyway, I think it’s a great feature. Obviously, users need to use it responsibly, but this will go a long way in satisfying the appetite for Instant Answers. You get the helpfulness of Google’s widgets without the entire SERP being overcrowded by widgets.
The only thing I’m unsure about is this:
For now, my only question is, does the model use the queries as training data?
Also @Josh, the cookie expired on the same day I added it. I needed to delete the date field for the summary to come back.
Nope this is pre-trained and there’s no continuous learning.
If possible then we’ll always try and push out non-js first, but the simplest quick way of doing this was with js. It’s an experiment at the moment and the intention is to improve as time goes by - part of that would be offering non-js.
Non-js would require the whole response to be complete, so this would be slower and we’ll need to flag this; other than that it’s quite easy to do.
my initial impression after using it isn’t good - it seems there’s little difference between the Moj summary and the Wikipedia summary
the text appears to be sourced only from mainstream sources and ‘mainstream’ does not imply accuracy and in many cases is simply disinformation or misinformation
in some cases the summary reads more like an advertisement rather than general information (“covid vaccine” for ex.)
i think it would be immensely more helpful if countering POVs were included - while sourcing reliable information from non-mainstream sources may be difficult, ignoring it will, without a doubt, lead people in a very wrong direction some of the time, especially with regard to politically sensitive topics, healthcare, etc.
if Mojeek is unbiased, then so too should the summary be IMO
i can provide examples if desired, but they are political hot potatoes
I haven’t tested this summarizing feature, but I can share some thoughts regarding this feature on the practical perspective, which I hope you may find helpful in determining the direction of this project.
On my experience with Google and Brave Search, one thing AI is particularly bad at is being infoboxes. Hallucination is not actually unique to AI, but while search engines easily reveal off-topic results in a simple glance, AI summarizers create or string together wrong results that can sound plausible to someone who lacks deep knowledge about a certain topic (which, unfortunately, is often the reason someone is searching in the first place). So although a number of people love its convenience, it being muddled by too many unrecognizable false positives makes it terribly unreliable to me that I frequently just ignore AI-made infoboxes.
On the other hand, I noticed that AI excels in seeding ideas. The times when I’m having difficulty thinking of keywords due to lack of knowledge on a subject, or queries in question form that are hard to distill into keywords, AI can come handy in giving me keywords that I can use, or websites where I can start the search.
With that, I’ll mostly agree with @mike’s idea of a separate URL, like what you did with the RAG search. The referenced websites being listed along with the response make it easier to check the quality of those websites’ contents, and to spot hallucinations by comparing their contents to the response. Moreover, the related questions section helps in narrowing down vague searches.
Thank you @Videonas, @mike, @itsMe, @gnome, @snatchlightning for all this very interesting and useful feedback. Some of your questions will be answered in a blog post we will likely publish next week. At this point, I wanted to clarify some possible misunderstandings about how the summariser works. Firstly the summariser is not so different from the RAG protoype at labs.mojeek.com. The summaries on both are the pulling the same information from the search results; actually from the snippets of the top 8 ranked search results. The differences are more in the UI; both are showing the search results (only 8 in the case of labs.mojeek.com) with those results cited in the “Summary”/“Answer”.
When search results give you links to less mainstream sources, the same will be true in both these AI answers. Here is an example: https://www.mojeek.com/search?q=What+do+we+know+about+UAPs%3F&mal=1
You may note that when you hover over a citation in the summary the corresponding results on the vertical search links is highlighted:
Here is the same result in RAG, labs.mojeek.com, with the main differences being here the addition of suggested “Related” queries and a limit of 8 results aka “Sources”:
exactly, and given the focus of Mojeek, i’m not sure it’s a great fit in general, even if it is somewhat helpful some or perhaps most of the time (depending on the query)
the contradictions are obvious and this is one of the many reasons why “AI” can be quite dangerous (manipulation of public perception), yet the public is likely to view it as at least somewhat authoritative
i can provide examples all day long where these primitive chat-bots are factually and scientifically incorrect regarding many subjects because the information used to train them is also often largely and factually incorrect, but i only know this because i used search engines to navigate the weeds myself and find the necessary books, papers and other materials rather than relying on very flawed “AI” which, like Wikipedia in many cases, i see as a detour to critical thinking
“AI”, at this time, can easily amplify incorrectness and given the human biases in the algorithms, i think this is likely to only get worse as those algorithms are “adjusted” for accuracy
It’s good to see there are a variety of views on this, and this thread shows that anything like this has to be dealt with sensitively and cater to all of the varied opinions of people using Mojeek.
Just to reinforce something up top, this is an experiment, not an indication of a change of direction for Mojeek, or any kind of change to the fabric of what we do.
Mojeek is still a keyword-based search engine (with some semantic elements) that is working hard to provide the world’s alternative in search. This is something that people will be able to use if they want to.
If you have the cookie and you visit a results page you should now see a Summary tab. Clicking this will take you to a page where the query is inputted into Summary and the resulting output will generate below the search box and tabs.
This has been built in order to allow easy display of and access to the sources (on the right-hand side).
@Josh This kicks you out of other experiments like Maps. In the future, will there be a way to participate in multiple experiments? Or, are you willing to release Maps?
Hi @mike, just regarding the comment about being kicked out of other experiments, if you comma separate your FFID cookie values, you can have multiple tests enabled. For maps and summary, you’d need the value of the FFID cookie to be eiSoh7ii,B9FX3DK3.
I was doing research with the experimental Mojeek Summary feature and noticed a few problems that I want to share.
The issues all have one thing in common: A human cannot verify the information that the Summary is providing.
Tag Feeds
One problem I noticed was that the Summary referenced tag feeds. The obvious problem is information in a feed cannot be verifiable on a permanent basis because the page constantly changes. And the feed will only ever present generalized summaries of the articles which is not as good as drawing from the articles directly.
example.com/tags/some-topic
Deleted Sources
Another problem was that the Mojeek Summary can reference deleted pages. I’m assuming this is an inherent latency either in the Mojeek index itself or in the relationship between the index and the RAG API. Again, if the page being summarized is deleted, I can never verify the information in the Summary.
Paywall Peeking
The Summary can refer to information behind a paywall. While there is nothing inherently wrong with a paywall, there is a perverse incentive here. The crawler can peek at paywalled content and summarize that. But, a person would have to pay to verify that information. Most people would just depend on the free summary in that case.
Requires JavaScript
I also noticed a relatively minor issue but one which is likely to affect Mojeek users disproportionately. One of the referenced URLs relied on JavaScript to properly redirect the web browser to the article. While most people could reach the article in that circumstance, some Mojeek users have JavaScript turned off. And they might not be able to easily access the underlying source because of their preference.
I’m sure I’m getting some of the technical details wrong here. But I think my conclusions are valid.
These circumstances create claims which will show in the Mojeek Summary but which can be difficult or impossible to verify.
While these large language model problems might not be specific to Mojeek, formally launching a retrieval-augmented generation feature would mean that Mojeek is endorsing and adopting these problems as their own. At that point, you would be reserving some fraction of your resources to dealing with RAG issues. And you might not be able to work on more innovative features.
Hey Mike, thanks for this and sending in some complementary detail, in the order as listed.
Tag Feeds: Tag pages ending up high up the page is a ranking issue, so it’s one that we’ll look into (to see if there’s something which is causing these kinds of pages to end up higher than they should be).
Deleted Sources: We have some work being carried out currently in order to improve refresh rates, which should eventually fix this kind of issue, which crops up also outside of Summary.
Paywall Peeking: We can only get content from behind a paywall if the site in question has allowed us, otherwise we’ll see the same thing that a non-paying user would see. We’re going to look at this properly just to make sure nothing strange is happening behind the scenes.
Requires Javascript: Currently this tool requires JS as standard; this is not just limited to Summary as the summaries themselves are based off of the results, which do not change order based upon if Summary is included or not.
Summary is, and will be pitched as, a way that someone can see a summary of what comes from the top-ranked results.
We won’t be pitching this as an answer creation process, but rather a way that someone can quickly grab bits from the results pages before looking further.
I just recently encountered one of the issues you stated, the tag feeds. I noticed that these problems are also shared by snippets, where the phrases/sentences displayed cannot be seen in the article.
I agree, and I’m quite worried about it. On issues in Mojeek, I often get replies stating in one way or another that they haven’t been resolved due to Mojeek’s limited resources (stemming problems, malfunctioning date operators, jumbling search terms, poor quality snippets, bare-bones map, to name a few), yet here Mojeek launching another defective feature. I fear that Mojeek may be biting off more than they can chew…
These are a bit more complex than just limited resources problems, and some of them are possibly me not explaining the things themselves well enough.
Stemming: we fix issues of stemming on a case-by-case basis, from the feedback we get we wouldn’t say it’s a big problem; they are bundled and then sorted out in batches. If you have examples sitting then please send them our way.
Date Operators: As mentioned in this it’s a really thorny problem to solve at scale, for now the date operators work as described on the page, and are not included in a UI element - because if they were individuals would assume they world differently.
Jumbling Search Terms: can you give some examples? I don’t completely follow what this means.
Poor Quality Snippets: is this a reference to here / can you elaborate upon it?
Bare Bones Maps: currently still a feature that’s in testing, we’ve noted your previous suggestions.
