Some Mini-Survey Feedback

[bbbhltz]

Out of personal interest, I made a quick survey the other week and asked some people — mostly on Mastodon, but the survey was shared on LinkedIn and Reddit — to filled it out.

There were only 43 responses. Lesson learned, I guess, not many people recognise what Framaforms is so I suppose it does look like a risky click. The results aren’t much for drawing conclusions, but since Mojeek was one of the options I decided I could share this here as well.

The survey was a Likert-ish questionnaire where respondents were asked to grade search engines. They were given the following options:

• S - Top Search Engine(s)
• A - Excellent Second Choice(s)
• B - Mediocre / Not Worthless
• C - Only in an emergency
• D - Utter Garbage
• F - Never heard of / used

(In hindsight, it is also clear that I should have stuck with typical five-level Likert scale.)

I have already posted the following charts on Mastodon, but I will share them here as well. The first shows the number of votes the search engines received — omitting the “F-Tier” votes. The second shows only the “F-Tier” votes. The third shows a “weighted” score [1].

Today I had 10 minutes to kill, so I also looked at just the the people who answered “Don’t know / Don’t use” for Mojeek and produced another chart looking at the respondents’ opinions on the “winners” from the third chart above (plus Bing).

Normally I would sit back and write up a little report on this sort of thing, but it is such a small sample size I am not so inclined. I might give it to my students to do for homework. Who knows. A quick look says something, though.

• Bing is not popular among the Mastodon/Reddit/LinkedIn crowd
• Qwant and searX lack users
• DDG is popular
• Several members of the “google sucks” brigade came out to vote
• Neither Qwant nor Mojeek received votes ranking them as “Garbage” (possible due to lack of users)

I am using Mojeek in class as an example for a sequence on CSR/ESG, so I will have another chart or two to share someday, along with some detailed feedback.

---

My Feedback (so far)

My own feedback after a little over two weeks of use is similar to other comments made here in the forums. There are searches that return old results, incorrect results, imprecise results, I cannot find my own blog with it (that is not such a big deal), and so on. Mostly, it reminds my of searching when I first started using the web in 1996/97 — patience is required.

The other minor gripe I have, because I have a computer with a smaller screen, is that I would like a preference to change the sizing of the fonts.

---

[1]: S=10 / A=8 / B=6 / C=1 / D=-6 (no points added or subtracted for F)

[Josh]

Ridiculously useful data @bbbhltz, as you said only 43 responses, but the question you’re asking there is hard to squeeze into a Mastodon poll or similar, so a sharper tool had to be used.

On:

@bbbhltz

The other minor gripe I have, because I have a computer with a smaller screen, is that I would like a preference to change the sizing of the fonts.

I’ve raised that just now internally so I’ll see if there’s something we can do.

Correct me if I’m wrong here but it looks like there’s a bit of an awareness gap and a bit of a utility one to close; both problems we’ve been working on and will continue to plug away with. I’d be very interested in whatever comes off the back of the CSR/ESG work.

[bbbhltz]

@Josh

it looks like there’s a bit of an awareness gap and a bit of a utility one

I think that is a good way to put it. I also don’t know if a sharp increase in awareness about different search engines would be very beneficial to any of the lesser-known options. Why do I say that?

I watch my students search for things. Today my students had an activity on a pharmaceutical company and they were stuck on the term “river blindness” but out of 20 students, all armed with Macbooks — and keep in mind that these students must pass competitive exams to attend the school — none of them realised it was a disease despite that fact that the first hit on Google, and DDG, and Mojeek, and probably the others, tells you that it is a disease.

If what they want is not in the first five hits, and the title doesn’t have the answer they are looking for, they give up. Many people are just so used to the results from whatever they use. In the case of river blindness, the students did not expect to see the word “Onchocerciasis” (the real name of the disease) in the title so they didn’t click on it!

Enough ranting about that.

The CSR/ESG thing will have around 60 answers with some specific questions. But, everyone answering it is roughly the same age with the same interests, attends the same school, etc. It is not at all a diverse sample.

[Josh]

@bbbhltz

It is not at all a diverse sample.

Reminds me of pretty much every study from the Econ papers in my undergrad - we tested 30 male students at Harvard etc.

Even if not diverse I’m sure it’ll provide something of interest. I don’t know how you sort out that river blindness, an engine would be my goto, but I guess that’s pretty obvious if you work at a search engine

[bbbhltz]

I suppose I could create a second topic, but I feel like that is unnecessary. Some parts of what follows are off-topic, but the whole thing is feedback…

Context

I teach at an international business school. Every class I teach is required to have a sequence on Corporate Social Responsibility (or something adjacent). Much of the time, we focus on the bad companies, looking at the dirtiest of dirt — think Chiquita/United Fruit Co… I wanted to spend at least one lesson looking at what companies do right, but not focus on just the environmental aspect. Weeks ago, before discovering the existence of the Mojeek Discourse Community, I had planned a very short (10-minute) activity. I needed my students to use a product that represented the ethical counterpart to a product that they all had some familiarity with. At first I planned on using DuckDuckGo, but then by using Privacy Browser on my phone I discovered Mojeek. It was the perfect choice because my students all come to class armed with MacBooks and iPhones.

Presentation

Before giving them a survey to fill out, they were subjected to a short introduction.

Slides

The survey took an average of under 10 minutes to complete.

Findings

The survey was presented to 75 students over the course of 3 weeks. Somehow, because my students don’t pay attention and some of them don’t know how to access their school email (no joke) and when they do the links do not open for reasons or they don’t don’t realise they can also access their email via webmail, I received 51 answers.

They were presented with a form of “web quest” and asked to report their findings via the survey.

1. Search for your name on Mojeek. Does an account related to you show up on the first page of results?

One student shares a name with more than on pornographic actresses…

2. Search for a brand or company. Does the brand or company’s homepage show up on the first page of results?

2.1 What company did you search for?

Looking at the companies they searched for, it appears the students were confused. Some searched for products rather than brands, and they incorrectly reported reported no in some instances:

• Gucci — Can be found on Mojeek; first hit for me.
• Airdrone — I think this is a product rather than a brand? Or they spelled something wrong?
• Netflix — First hit.
• Nestlé — First hit.
• Dior — First hit.
• Apple — First hit.

As far as I can tell, all brands were located on Mojeek.

3. Search for a recipe. Did you find a relevant result on the first page?

4. Search for a how to article. Did you find a relevant result on the first page?

4.1 What did you search for?

Here are the searches (sic) that were deemed to return irrelevant results:

• How to send an forwarded email ? — there are some good results on the first page.
• How to repair a lightbulb — I believe they were thinking of how to change a lightbulb
• comment faire un tabouret ? (French for how to make a stool) — First hit looks pretty good.
• how to create an nft
• how to hunt a mouse
• How to work at home? — This one returns a conversions calculator.
• How to progress in mathematics? (searched for twice)
• How to be rich
• how to improve their life
• How to use Tor — Proposes several tutorials.
• How to learn japanese
• How to find a will to live
• How to upgrade windows

5. Search for the name of your hometown BUT spell the name incorrectly. Did Mojeek find what you were looking for?

I didn’t ask for hometowns, privacy and all that…

6. Can you easily figure out how to limit the search results to a single country? Like France?

Students no longer learn how to use tools, it is assumed they know how to do things like this.

6.1 If you wanted to find all mentions of the word “France” on the website “cbc.ca” excluding results mentioning “Paris”, what would you type?

Five students found the solution. I am quite sure that one of them shared the answer. It confused them. Using advanced search options or operators in the search field is not something they have ever learned to do.

Opinions

They were asked their opinion on the following statements:

1. You would use Mojeek
2. You would recommend Mojeek
3. You like their idea
4. You think the company has good values
5. You don’t understand why something like this exists when most people use Google
6. You like the presentation of the site
7. You found the site fast and responsive

This is where the ideas about CSR/ESG and reputation came into the picture. The students have generally favourable opinions about the missions and values, which is what I was hoping to see. The opinions also show a hesitation to use or recommend the service despite a majority finding the site to be fast and responsive.

Keeping in mind that they used it for less than 10 minutes, the only conclusion that can be drawn is that first impressions matter.

Write a single sentence comment about your initial impressions of Mojeek

Positive/Mixed	Negative
a lot of things to improve but good idea	I don’t understand the point
Great but need time to adapt	A bit “has-been”
The search engine promotes good values but seems to be a work in progress	It’s useless because not precise enough and Google already exist.
i like the application because it doesnt keep our information	I don’t understand what is the real goal of this website, cause’ Google is more efficient and more known.
the green is nice	Meh!
It’s just a habit to adopt in order to use Mojeek every day	It looks like an old search engine created in the 90s but not really. good discovery
It is cool but not really efficient	that is not very efficient
good	It doesn’t seem as powerful as Google
good web site but google is more practical	kind of has-been
Nice to use but I will continue to use safari	It is easy to use.
It’s a site that seems useful but I don’t see the interest in comparison with Google	I don’t see the interest
I think that they have good intentions but I will continue to use Google	I think the concept is interesting but the interface is quite obsolete
great	the platform is very basic
GOOD	Not competitive enough against Google
The site is fine and the concept is great, the only thing is I have the impression that the results are less interesting	The site looks like all the others searching sites
I am agreeably surprised	The site needs more colour
Neutral website / Need to get a better design to attract more people	The site is not attractive
I find it really intuitive and the design is nice	disgusting
fast but useless	Work in progress
I think the idea is great but as many other CSR project, it is hard to put in place	Not very precise, lack of rigour
fast	It is less pleasant than Google because the results appear only with lists of articles, while on Google there are images that appear for example: it is much easier to identify the sites on Google
The idea is good and the values are great but the page is not attractive
It is very fast and precise. The absence of ads is pleasant because we have direct access to the results of our research without it being biased.
It’s user-friendly, and fast
What I like the most is the values of this search engine
Nice

I highlighted my favourite comment because it supports the idea that some people just use the default and never consider other options. It also underlines that there are some members of the so-called generation of digital natives (or digital residents) that are, in fact, not that knowledgable of digital tools.

Some of the negative comments are about the design of the page. Comments about being unattractive or lacking in colour are expected. This recent blog post from Filteroff that made the rounds talks about this issue as well:

many [users] do not like “millennial minimalism.”

What search engine do you normally use?

Thoughts

My personal conclusion from this is influenced by the fact that I know these students. I have been watching the decline over the years. I have no doubt that some of them are proficient users of search engines, but I know for a fact that the majority just don’t have the reflex to click on a link, or open it in another tab to investigate later. This is a public that has multiple browsers open at the same time, all with multiple tabs, some of them duplicates of Facebook in order to maintain multiple conversations at the same time.

Asking them to use a different search engine was not a nice thing to do. Their expressions showed it. They wanted their Google fix after this activity.

[Josh]

@bbbhltz, thanks a lot for this, it’s the kind of research that is quite difficult, expensive, and/or time intensive for us to undertake, especially as I would guess the bulk of your students hadn’t seen/heard of Mojeek before. We ran a primary/interviews-based research piece from late 2020 to late 2021 and a very large slice of the people involved know Mojeek, so very different.

Slide 7 says:

Mojeek checks many of the CSR boxes

I’m interested: when it comes to the teaching of a course/the academic side of stuff which boxes do we not currently tick and/or shout out about?

The questions themselves are great, we bang on about feedback quite a lot because it’s such a great and organic way of finding how we’re falling down in places. The examples in 2. and 4. are of great assistance when it comes to that.

• How to work at home? — This one returns a conversions calculator.

This has been raised so we’ll chase down why that’s triggering. Also the results on the page aren’t great either so it’s one for the to solve pile in more than one respect. We also undertook a couple of hometown-spelt-wrong searches and found that there was at least one case in which the Did you mean: didn’t trigger, so another thing to look at.

6. is unexpected for me, removing the 15.7% (8 if my numbers are right) there who didn’t understand, there’s a slim majority who gave a straight yes. This being said, more is always better so it’s something to look at.

From Opinions downwards is something that I’m going to dig into a bit deeper, but for me is the most useful thing you can possibly chuck at me. First-time users giving honest first-time opinions; it’s good too see that the values resonate even if there is room for improvement elsewhere. Forgive me if I come back to you asking more in time.

The confusion between a browser and a search engine crops up every now and again, I guess the ubiquity of tech means that it’s not necessary to really learn about it before using.

Asking them to use a different search engine was not a nice thing to do. Their expressions showed it. They wanted their Google fix after this activity.

That’s another valuable insight there, especially as you know them so pick up on this easier. The experience has to be delightful enough to counteract the pain, something to think on also.

I said it at the top, but invaluable, thanks once again for sharing!

[bbbhltz]

@Josh

I’m interested: when it comes to the teaching of a course/the academic side of stuff which boxes do we not currently tick and/or shout out about?

I didn’t want to cover everything when I presented this. Also, I never planned on sharing it outside of the class. But, since I am the only one doing this activity I have no problem doing so. For me, the ethical aspects were what made me want to use it when I first discovered it and the fact that it does check tick the boxes, all of them, I think. I tend to teach the subject by first talking about the classic aspects: the 3 Ps.

And then move on to talking about reputation.

I feel that as long as the product is improving, which Mojeek does with each page indexed and each minor tweak, and if a company keeps its promises to uphold the values, then all boxes are ticked.

If I had discovered this, or any service like it, a little earlier in the school year, I could have collected hundreds of responses.

I am glad you find this useful, though. It is very rare that something I do in class finds an audience beyond my students .

[mike]

I’m not familiar with business school. If a student says, “I don’t understand the point,” or “Google already exist[s],” are they ever challenged on their specific feedback and required to develop their thinking so they can empathize and educate themselves?

It seems like this kind of skill would be prioritized.

[bbbhltz]

@mike for this activity I did not get so deep. We do push them to think and develop critical thinking but are confronted with a challenge during their first year: they are all very similar. Out of these students, maybe two would be “out of the box open to new things” thinkers, the rest are less so. They all come from the same socio-economic background, are well off, attended the best high schools and best prep schools, and think the same way. They have all just spent two years preparing for an interview to get into a business school.

In their second year, though, they are required to do a 6-month internship in a foreign country and/or several months of study in a foreign university. This experience is a light bulb moment for them and usually opens their eyes to the idea that there is more than one way to do something.

For the moment, they still think the answer to all business problems is social media and that Canva is the best way to make any type of document. They still get their parents to phone the school to excuse them for absences!

If this had been a graded exercise, their answers would have been judged in some way.

[Colin]

@bbbhltz I’d like also add my big thanks for this. It’s a well designed study and very interesting for us to see. Not an easy crowd to do it for and all the more important to develop their critical thinking skills. Overall I would say, this is very encouraging for us. With 4% (2 people out of 50) said they would use Mojeek and 3 would recommend if that was scaled up, we would have a huge user base. Of course, the small sample, doesn’t mean this, although as you suggested the numbers of those prepared to think “out of the box” can be higher in with other audiences. We are held back by many factors, obviously awareness and mixed search quality. Both are improving. The brazen tactics of Google, Micrsoft and Apple and their co-opetition which keeps out anything but Google or Bing(-lite) search engines in their browsers/devices is the biggest throttle on wider usage. Anyway, we battle on, and we’ll take on board this feedback, thank you.

[bbbhltz]

@Josh I just found another search that triggers a conversion calculator:

How to win solitaire

[Josh]

Cheers for that one, I’ll get it looked into

[bbbhltz]

OK, not actually such a great find. Any search that is “how to w*” seems to propose a conversion.

[samleh]

Just dropping a few thoughts…

I’m personally slightly split between B and C options. If it’s something popular I would say B. With my specialty searches looking up for something specific Mojeek is strongly in C and almost in D category.

I like UI, speed, independence and ideology, but there are some rough bumps to deal with and index size is what it is. I’ve very got similar feelings about Matrix / Element, but that’s not the topic for this discussion.

DuckDuckGo is clearly better, but for the specialty searches I usually use Google because DuckDuckGo is also very weak compared to Google if looking something like string in source code or some special configuration parameters of some obscure servers software or something similar.

searX is not a search engine itself, and I’ve had usually very bad experiences with shared instances. Qwant is bit mixed case as well, I would currently rank it below DDG. Other Bing fronts are mostly useless, I dont’ get the point of those / advantage.

And finally Yandex is actually surprisingly good, or at least let’s say that it’s better than most probably would expect. Yet of course it’s clearly worse for English specialty searches than Google.

[Zed]

Hi @bbbhltz

I don’t know why I have only just discovered this thread/topic. I suspect that I have not yet ‘got to grips’ with Mojeek Discourse. Anyway, I’d like to say that I find your survey very interesting. But, I’m in no position to act on your valuable findings. Has a plan been drawn up to implement any changes?

[Zed]

I am very confused. I submitted the post immediately above on 19 Sept 2022 but the post is dated 18 Jul. Can anyone explain?

Thanks in advance.

[Josh]

We fed this research provided by @bbbhltz into our plans at the point it was provided, extremely valuable and the kind of thing which doesn’t just pop into your lap.

@Zed

I am very confused. I submitted the post immediately above on 19 Sept 2022 but the post is dated 18 Jul. Can anyone explain?

Thanks in advance.

I don’t see this:

@samleh’s post was number 14 which was on that date but yours is registering correct here.

[bbbhltz]

I think I noticed the same thing as you

@Zed

I submitted the post immediately above on 19 Sept 2022 but the post is dated 18 Jul.

And I just had it happen again on another thread where I was notified of a new post but when I clicked the “new post” was from several months ago.