For Science

RSS-Librarian has worked quite well and attracted about 150 users to the main instance that regularly add a lot of articles, as well as several custom forks. I have also noticed several instances that have been modified and inserted into custom pages. The main instance is running stable with the default configuration and custom XSLT stylesheets for previewing the feeds.

During my time using RSS-Librarian I started craving some additional features, particularly around custom extraction paths for websites that have some form of API access, such as Peertube, Youtube, Odysee/LBRY, and Arxiv. Especially since the last big waves to migrate to federated services happened (Mastodon and LBRY as an example) there is need to identify an instance of a service. How does RSS-Librarian detect it is reading data from a Mastodon instance hosted on some arbitrary URL? It became clear there needs to be some extra code to parse metadata from the page before making a decision what type of extraction is needed.

Other pages such as Arxiv provide links to PDFs of publications, but some of them also have a HTML version of the publication and so instead you might want to scrape this content rather than just the abstract. How to deal with scenarios where the URL needs to be redirected?

RSS-Librarian is a read-it-later service primarily, but I often find myself storing video URLs in it, i.e. there is nothing to read here. Could we simply embed the video file directly, so I can stay within my reader?

Version Two

RSS-Librarian is a read-it-later service for RSS purists. You can store articles from the web in your own personal RSS/Atom feed and use your favorite feed-reader software to read your stored articles later. RSS-Librarian uses no database and works without accounts.

I am happy to announce that RSS-Librarian has now arrived at Version Two, a much improved state compared to the last release. Below are the the changes since Version One.

1. Custom Extraction: An optional file called custom_extractor.php with an entry function called function custom_extractor($url) can be placed next to librarian.php to implement custom extraction paths. RSS-Librarian is kept simple on purpose, but if you need to extract a website to get additional data out, or connect to a custom API, or even redirect content extraction, then this is for you.
2. Improved URL fetching: The function was simplified and is reused across the rest of the codebase. Additionally, it now handles cookies and redirections that would previously fail a fetch.
3. Fivefilters dependency removal: When RSS-Librarian started, I wanted an easy way to deploy it, with a single PHP file sitting on a webserver and no other dependencies. However, the readability and meta info extraction would rely on an external service from FiveFilters, making it very centralized. I have now removed this dependency entirely, and instead rely on a naive extraction path with meta tag information from OpenGraph if readability.php 4.1.0 has not been installed.
4. OpenGraph support: Many pages feature an OpenGraph header which contains meta information about the current page, giving you a minimal set of data to extract into an RSS feed. This is now used to augment potentially missing items from previous extraction paths. In addition, Twitter cards and old-school meta tags will be used as fallbacks.
5. Enclosures: Many feed-readers support enclosures, a subelement in the feed that can directly link to a media file such as a video or an audio file. It allows the feed-reader to directly pre-fetch a media file for offline replay.
6. Embeddings: When adding an MP4, an image, a PDF or a audio file, RSS-Librarian would try to parse the binary and then horribly fail. I have added HTTP header peeking to determine the mimetype of the URL, and in case of media files or documents embed and enclose it in a feed item for proper display.
7. Versioning: Even though the releases follow a simple tagging scheme VersionOne/Two/Three..., users might run custom checkouts instead of following releases. To better identify those versions, a date plus hash scheme is used to identify the file roughly.
8. Custom cookies: RSS-Librarian does not have access to (pay-)walled content, such as paid-for articles on Substack, Patreon or your favorite news-outlet. In the configuration you can now supply a cookies_file pointing to a text file of extracted cookies which RSS-Librarian can use to log in on your behalf and get access to articles locked behind an account.
9. Additional instance info: For debugging purposes, the instance will report if it has readability.php or custom extraction installed.
10. Bugfixes: A slew of fixes went into this release addressing general stability, XSS prevention, encoding issues, sanitizing inputs and outputs better and an upgrade to readability.php 4.1.0.

Consume it later

My next goal is to make RSS-Librarian much more than a read-it-later service, that instead allows you to consume-it-later. With the enclosure support, I have started opening tickets on Capy Reader and NetNewsWire - the readers I personally use - to support enclosure caching. Some other readers, for example Vienna RSS, already support enclosure downloading, though not automatic.

The idea is that a reader has a user setting to allow certain articles - for instance all starred items - to fetch and cache enclosures in the app automatically, and embed viewers automatically for them in the feed viewer. This would allow the user to not just view text articles offline, but videos, audio files and any other attachments as well. The main RSS-Librarian instance currently features a custom extraction path that, for Youtube links, will look up if the same video has been mirrored on Odysee/LBRY. If so, it will attach the raw MP4 to the article. To view the video, one doesn't even have to open Youtube anymore and wade through loads of ads, and with pre-caching enclosures those videos would be watchable when offline!

What's next?

I'm pretty happy with Version Two. The RSS-Librarian has been rather stable since Version One and attracted new users quickly that are clearly into reading a lot of articles, but also add a lot of video or non-article content.

There are a couple of ideas I am working on right now:

• More custom paths: Ideally I'd like to open up for contributions using the new custom extraction mechanism, similar to gallery-dl, yt-dlp and RSS-Bridge that accept custom extraction paths. Currently the main instance is running custom extraction for Odysee, Youtube, Twitter, Bluesky, Arxiv, ACM Digital Library and Eurographics Digital Library.
• Document extraction: Pandoc is the Swiss army knife of document conversions, and combined with Markitdown it can cross-compile almost anything to HTML5 too, which would allow RSS-Librarian to extract document text from other document formats. Particularly for reading papers this would be great, and the original document can be attached using the enclosures that have just been added.
• Media RSS: Next to enclosures, RSS features the ability to enhance the frozen standard with extensions. One such extension is Media RSS. This extensions provides much broader meta information about a media file that gets enclosed with the article.
• Naive extraction: RSS-Librarian no longer connects to FiveFilters and uses a simple copy of a page <body> as the content, paired with OpenGraph and other meta-data extraction for the title and author. I intend to insert simple mechanisms to get rid of control elements and ads to get the simplest bang-for-the-buck content improvement.

If you have any additional ideas please open a ticket and let me know, or even better, open a PR.