TESTING: Substack Search

Testing
#1

Hello @all, we’ve got something new to share for testing, currently via a cookie. I’ll explain the functionality first, then put the method for how to try it out.

What this is

This functionality is a first version of a blog search, with the intention being that we’ll eventually incorporate a whole range of different writers across many platforms into this separate tab. Much like /news pulls together news sources, this would allow for a search just across a range of different independent writers, bloggers etc.

For the moment, due to the ease of implementing it as a v1, this only incorporates Substack. The range of options in this one website, ease of adding in feeds, and coverage we already had in the index, made it a simple one to rig up.

This intention of broadening out the search is why it has been given /blogs to sit on.

How to test it out

Much like with Focus, you can access this functionality with an FFID cookie using the value H73R2j3Y on path / like so:

scren
scren2980×1474 282 KB

Browser methods are in the Focus How To Set Up - changing the value to H73R2j3Y, obviously.

After that, the tab should pop up on a refresh and you should be all good to go, clustering is as you’d expect:

image
image1515×1000 103 KB

Thoughts, bugs, anything really is much appreciated ahead of this going out. :pray:

4 Likes
#2

This is a great idea for a feature! I’m often on the lookout for new and interesting (or old and venerable) blogs. I’ll have to think of some topics to search for later, but from my first attempt, I didn’t get any Substack results: https://www.mojeek.com/search?q=search+engines&fmt=sst&sst=1

image
image967×936 144 KB

So, count that as a bug? I’ve added the FFID cookie.

#3

It definitely counts as something to investigate, as that is just regular search results. Are you okay with giving me your cookieless link so we can see if there’s something pushing or pulling it away from working as expected? DM is all good if so

#4

It works now that I’ve revisited it. Could you clarify what my cookieless link is? Then I’ll be able to decide how I might provide it to you :slight_smile:

1 Like
#5

Oh of course, sorry, assumptions; in Search Settings - Mojeek on the right there is a box which populates with a link you can use instead of saving settings in a cookie, it’s a quick way also of being able to see what has been set by someone, and then from there check if anything is affecting anything else.

This being said, if it works now, all good to leave it unless this happens again.

#6

Ah yes, I understand now. I knew I’d heard that term before.

Here’s mine if it’s useful to you: https://www.mojeek.com/?theme=light&t=40&tn=5&date=1&cdate=1&si=3&autocomp=0

Tangentially, I think the cookieless link would be much more useful to users if you served a different OpenSearch document depending on what query parameters appear in the URL. That way, users could set Mojeek as the default search engine, and it will always have particular settings without setting cookies.

This is most useful for desktop Firefox users, because Chromium-based browsers allow you to edit the search engine link directly, while Firefox doesn’t (except in obscure ways). One has to wonder what the point of an open standard is if every browser handles it differently, and not even consistently across platforms…

1 Like
#7

adding a comment here to bump this to the top of the forum, testing/feedback much appreciated if anyone is looking to try something out :pray:

actually scratch that, pinned :laughing:

#8
#9

Interesting feature! I like. Hopefully you can add more sources soon.

Differentiation between blogs and other sites

How are you planning to differentiate between blogs, news sites and just other websites (like corporate etc.)? If you’re talking about blogs, I think about WordPress (professional malformation). But there are many different types of websites using WordPress. So I’m curious as to how you’ll distinguish blogs from other sites?

Feeds

Another idea from here: Are you using RSS feeds for this? If so, then perhaps you could make that available as well (with a link or an icon next to the search results)? This way, a user can easily subscribe to that feed with an RSS reader. Or you could cater to the wishes to see the entire feed (with a link or button to a page with all articles of the feed in chronological, or user definable, order)?

1 Like
#10

Currently an open question; it will likely keep to independents as much as possible. Whereas /news has larger organisations/media, this would probably pull together smaller outlets and individual writers.

Part and part, Substack here is being used as the additions are all quite easy (they all have feeds at /feed) and because of the number of independent writers on it, but also because our index already covers Substack pretty well - there are RSS feeds in this tool as well.

Noted, thanks for the suggestion :pray:

#11

Regarding heuristics to detect blogs, here are some positive signals:

  • Website technology fingerprinting. There are tons and tons of open-source OSINT tools to do this, for reference. If you detect that a website uses, say, Ghost or Blogger, you can increase its “probably a blog” score.
  • Feed detection. Websites with RSS/Atom/WebSub/JSON feeds are more likely to be blogs.
  • Open Graph “article” metadata.
  • schema.org metadata indicating a “BlogPosting” or “Article” type. Blogger uses Microdata syntax for this; WordPress often uses JSON-LD.
  • Microformats (legacy microformats and/or microformats2).
  • Whether it triggers “reader mode” heuristics in a reader mode implementation. Readability and DOM-Distiller have several open-source implementations in multiple programming languages.
  • Existing directories. Search My Site and Brave’s “tech blogs” goggle are two public directories of blogs used by blog-focused search engines, which you could use as a starting point.

The best engine in this space is Marginalia. It might be worth reaching out to see if Viktor wants to share.

#12

Thanks for these, for the moment we’re looking for feedback on the functionality as it stands, so all and any would be appreciated, including anything that looks off :pray: