Tobias Alexander Franke

Agent OPML

Same procedure as last year

We do not need to debate the merits of AI; this has been discussed to death on the web and reminds me of one of the many programming language wars I was involved in in the early 2000s, where a single language was somehow the solution to world hunger and memory allocation bugs.

I wanted to talk about a part of my day to day work. I need to evaluate a lot of publications - blog posts, articles from various hardware vendors, Github releases, papers, conference programs … - to judge if they match our business use case or can be feasibly integrated into a prototype. Naturally, I have a lot of constraints around what I can and cannot do: Something must be open source and MIT or equivalently licensed, must meet certain hardware requirements, must be using a subset of GPU APIs, must be part of a certain graphics domain etc.

I read a lot of sources every day that publish new and interesting stuff, like Jendrik Illner’s most excellent Graphics Programming Weekly blog, individual bloggers, conference programs and more.

As an example, a single SIGGRAPH conference can sport around 100 publications that I need to review. Each publication comes with a session, title, an abstract and a full PDF. Naturally, there’s a progression how I get to the juicy parts: Discard sessions I don’t care for first, search for interesting sounding titles, review their abstract and eventually keep the most interesting matches to read later in full.

I am subscribed to several conferences, around 250 personal blogs and several newsletters of companies, all in the computer graphics domain, all of them crunching out several articles a day. This takes an insane amount of time off from my day, every day, and the process is almost completely monotonous. Clanker-like one might say.

So here’s my AI use-case, and how my obsession with RSS feeds directly into all of this.

Pipeline Overview

Agent OPML is a local LLM-based agent (NanoClaw, Pi, OpenCode, OpenClaw, whatever) that uses a tool to fetch publications from various sources and serializes all new entries into a fresh JSON file. Each entry looks like this:

{
    "url": "...",
    "title": "...",
    "authors": "...",
    "summary": "...",
}

Once the data is fetched, the agent’s task, described in a skill, is to analyze each entry. The agent first analyzes the summary (which in many cases is the entire publication pulled via RSS), and if this is unsatisfactory - because it is just a shortened RSS <description>, is metadata about the publication but linking to it or is a URL of a video - go to the original url and try to fetch more using extra tooling. For instance in case of Arxiv, there may be a HTML version of the paper, and if not there may be a PDF instead, in which case the agent proceeds to download the file and converts it to Markdown for processing.

The skill describes what is interesting, what sounds interesting, etc. Each entry is analyzed for its relevance and goals, and more importantly if it fits within certain constraints. If all of these criteria are met, create a mini-summary and rating why this was deemed useful. The report is produced as a Markdown document, with annotations into a JSON file. Both are combined and attached to a new feed file that is served on some httpd.

The outcome is this: A single feed I can subscribe to which updates me daily with those publications I truly want, and the pipeline/algorithm to judge all of it entirely under my control.

Routing data to the agent

The input for the agent is a stream or feed of new publications that is updated regularly.

To do this, we will use an OPML file. For the uninitated: An OPML is a XML file pointing to multiple URLs of feeds. A feed in turn is a structured file containing blog posts or any other updates from a webpage, and those come in three major formats: Atom, RSS 2.0 or JSON Feed. Most importantly however is that everything in those files is machine-readable. Usually OPML files and feeds are used with a feed-reader such as NetNewsWire and are a simple way to store all your subscriptions.

We are going to use the OPML instead to let the agent automatically fetch new content from webpages for automagic analysis. The OPML should capture every feed of information that you’d have to wade through manually, but want the agent to do it for you.

To use my interests as an example, the OPML contains the following types of feed subscriptions:

The last two here are worth elaborating on:

The OPML is used when running a script called fetch_publications.py, which goes through every subscription one by one, fetches the feeds and then combines all new entries it has not seen since the last run into a JSON file that is then used by the agent for analysis.

Setting up Agent OPML

Prerequisites

This setup includes a bunch of things:

Start by setting up a Raspberry Pi with Raspbian, and create a separate non-sudo account for the agent. If you want to run this on your local machine, you can skip this step. Then simply run the following steps.

On your sudo account:

$ sudo apt install pandoc cython3

On the agent account:

$ curl -fsSL https://opencode.ai/install | bash
$ pip -m venv ~/.venv
$ . ~/.venv/bin/activate
$ pip install markitdown feedparser

Code snippets

fetch_publications.py

This mini feed-aggregator fetch_publications.py is a simple Python script: Loop through a list of subscriptions from an OPML given via argument --opml and fetch the feeds, parse feed entries, reformat them into JSON dictionaries and use the URLs as IDs to remove duplicates. The outputs are written to a file given by --output that contains the extracted data. A temporary file called feed_state.json is kept in the same output directory to mark already visited articles.

#!/usr/bin/env python3

import json
import hashlib
import requests
import feedparser
import argparse
import xml.etree.ElementTree as ET

from fake_useragent import UserAgent
from typing import List, Dict, Any, Optional
from pathlib import Path

def load_opml(opml_path: str) -> List[str]:
    tree = ET.parse(opml_path)
    root = tree.getroot()
    urls = []
    for outline in root.findall(".//outline"):
        url = outline.attrib.get("xmlUrl") or outline.attrib.get("url")
        if url:
            urls.append(url)
    return urls

def load_state(path: str) -> Dict[str, List[str]]:
    if not Path(path).exists():
        return {}
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)

def save_state(path: str, state: Dict[str, List[str]]):
    with open(path, "w", encoding="utf-8") as f:
        json.dump(state, f, ensure_ascii=False, indent=2, default=st)

def entry_id(e: Dict[str, Any]) -> str:
    candidate = e.get("id") or e.get("guid") or e.get("link") or ""
    if not candidate:
        composite = (e.get("title","") + e.get("summary","") + e.get("published","")).encode("utf-8")
        return hashlib.sha256(composite).hexdigest()
    return candidate

def normalize_entry(e: Dict[str, Any], feed_url: str) -> Dict[str, Any]:
    return {
        "title": e.get("title"),
        "url": e.get("link"),
        "summary": e.get("summary"),
        "authors": [a.get("name") for a in e.get("authors", [])] if e.get("authors") else [],
    }

def fetch_feed(url: str, timeout=20) -> Dict[str, Any]:
    headers = { "User-Agent": UserAgent().random }
    resp = requests.get(url, headers=headers, timeout=timeout)
    
    try:
        resp.raise_for_status()
    except:
        # retry without user agent
        requests.get(url, headers={}, timeout=timeout)

    resp.raise_for_status()
    
    return feedparser.parse(resp.content)

def load_existing_entries(path: str) -> List[Dict[str, Any]]:
    if not Path(path).exists():
        return []
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)

def save_entries(path: str, entries: List[Dict[str, Any]]):
    with open(path, "w", encoding="utf-8") as f:
        json.dump(entries, f, ensure_ascii=False, indent=2, default=str)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("-i", "--opml")
    parser.add_argument("-o", "--output")

    param = parser.parse_args()

    if (not param.opml or not param.output):
        print("Parameters missing")
        return

    feeds = load_opml(param.opml)
    if not feeds:
        print("No feeds found in OPML.")
        return

    state_file = Path(param.output).with_name("feed_state.json")
    state = load_state(state_file)

    new_entries: List[Dict[str, Any]] = []

    for feed_url in feeds:
        try:
            parsed = fetch_feed(feed_url)
        except Exception as ex:
            print(f"Failed to fetch {feed_url}: {ex}")
            continue

        seen_for_feed = set(state.get(feed_url, []))
        new_ids = []
        for e in parsed.entries:
            eid = entry_id(e)
            if eid in seen_for_feed:
                continue
            ne = normalize_entry(e, feed_url)
            new_entries.append(ne)
            new_ids.append(eid)
            seen_for_feed.add(eid)

        state.setdefault(feed_url, [])
        state[feed_url].extend(new_ids)

    if new_entries:
        save_entries(param.output, new_entries)
    else:
        print("No new entries found.")

    save_state(state_file, state)
    print(f"Collected {len(new_entries)} new entries")

if __name__ == "__main__":
    main()

Re-running the script should only produce a file if any new entries have been found since the last run.

convert-pdf-to-md.sh

OpenCode has a directory ~/.config/opencode/tools which contains small scripts the agent can use to do something without requiring it to go through the process manually, thus saving tokens.

This tool uses markitdown to convert a PDF to Markdown. This is helpful when the agent downloads a PDF and needs to process/read it. The easiest way to do that is to simply convert it to a structured text file.

#!/bin/sh

INPUT=$1
OUTPUT="${INPUT%.*}"

markitdown $INPUT > $OUTPUT.md

The Filter Skills

This is the heart of the analysis and filtering process. These files describe what the agent should do, structured into these three general steps:

  1. Read the combined entries that fetch_publications.py spits out
  2. Analyse each entry if it’s worth your time
  3. Create a report of what passed the filter

First is the filter-publications skill, which generically describes how to parse through the JSON file with all the new entries and how to collect them together. Save it as ~/.config/opencode/skills/filter-publications/SKILL.md.

---
name: filter-publications
description: Given a list of new publications, find those relevant to our business use-case and produce a Markdown report on the agents findings.
---

# When to use this skill

Use this skill when the user wants the agent to evaluate and filter publications such as papers, blog posts, announcements, conference proceedings and so on.

Rank each publication by whether its idea, approximation, pipeline, or result could realistically transfer to the users specified use-case.

Be conservative and evidence-grounded. Clearly separate what the publication actually measured from what the agent infers as possible goals that can be reached with high relevance. Do not invent performance estimates, deployment claims, or results that are not supported by the inspected sources.

Use these three steps outlined below:
1. Read Input
2. Analyze Publications
3. Create Output

# Read Input

The agent is given one JSON file by the user (most often called `new_entries.json`). It contains a list of publications, where each entry has: a `title`, a `url`, a `summary`, and a list of `authors`. 

If the file the agent was given by the user is empty or does not exist, immediately halt and do not progress any further! Do not read any other json files or try to get data from somewhere else.

Each entry represents one new publication that was just discovered. The agent will need to analyze each one independently for its merits. Try to read the `summary` first. The summary may or may not contain the full publication. If it seems too short, try to open the URL and read the contents from there instead. If the `summary` is a short message from a social media page and contains a URL, follow that URL, especially if it contains a full program of a scientific conference.

Execute the [Analyze Publications](#analyze-publications) forstep described in the next section for all entries.

# Analyze Publications

## Required behavior

1. Read one publication entry source from the supplied JSON list.
2. Extract or read the best available information from the `summary` field.
3. Summarize the publication's core idea, evidence, and claimed results.
4. Evaluate the publication using the `review-publication` skill and produce an assessment with a clear verdict.

Report missing abstracts, inaccessible PDFs, broken links, paywalls, failed document extraction, low-quality extracted text, missing project pages, missing code, or insufficient evidence.

## Execution strategy

Do not treat a preliminary title/session-based inference as a confirmed analysis. If only title, session, program metadata, or DOI was inspected, label the evidence as **Limited** and make the score conservative.

Before finalizing top recommendations, do a detailed inspection of the strongest candidates. A publication should not appear as a final **Validated Top Recommendations** with a high score unless the agent has inspected substantive evidence such as an abstract, project page, PDF/text extraction, code repository, results, or limitations. If a publication cannot be analyzed in detail because the PDF is too large, paywalled, inaccessible, or time-limited, list it as a **Potential Top Candidates** rather than presenting it as fully validated.

These recommendations should not be homogeneous. Prefer a diverse shortlist that covers different topics that match the *Relevance* and *Goals* sections of the `review-publication` skill.

Prefer lightweight sources first: program page, DOI or publisher pages, abstracts, project pages, code repositories, videos, and supplemental descriptions. Do not download very large PDFs or supplemental files unless necessary. If a large file is skipped, report that limitation.

# Create Output

Create a subdirectory called `output`. All files the agent generates, even intermediate files, need to be written into this directory!

## Markdown report

Create a file called `report.md`. The file must start with a frontmatter. Use this exact style, with no blank line before the closing delimiter:

```yaml
---
title: Daily Report
author: Agent OPML
date: \today
documentclass: report
colorlinks: true
urlcolor: Maroon
linkcolor: Maroon
---
```

Do not modify this frontmatter. The report must only use Markdown headings, bullet points, and tables. Do not use `---` after the frontmatter, and no HTML tags. Structure the report into the following sections and no other headings:

1. **Overview**: A summary - as a list - of the overall statistics of the analysis, how many entries the agent processed, the themes the agent discovered and what the agent discarded. The list can be structured by source or category.
2. **Validated Top Recommendations**: Publications supported by substantive evidence. Use the the output of the `review-publication` skill for each publication.
3. **Potential Top Candidates**: Publications that look promising from title/session/metadata but lack enough evidence for a confident recommendation. Use the the output of the `review-publication` skill for each publication.
4. **Overall Verdict**: A summary of the agents findings and the papers that were recommended.

## JSON list

Create a file called `report.json` which follows the [Markdown report](#markdown-report), and use the JSON outputs from `review-publication` to populate the `validated_top_recommendations` and `potential_top_candidates` lists.

```json
{
  "overview": "{OVERVIEW}",
  "validated_top_recommendations": [],
  "potential_top_candidates": [],
  "overall_verdict": "{OVERALL_VERDICT}"
}
```

In this file you will need adjust and modify the # Analyze Publications section and/or provide a customized skill review-publication that judges and summarizes each individual entry the filtering process goes through.

Next is the review-publication skill that will guide the agent to judge an individual publication. Save it as ~/.config/opencode/skills/review-publication/SKILL.md.

---
name: review-publication
description: Judge a publication based on relevance, constraints and goals, and describes how to score the publication
---

# When to use this skill

The user asks the agent to produce a summary or judgement of a publication.

# Preprocessing

- If the publication is a social media post ...
- If the publication is a video, first ...
- ...

# Relevance

[Summarize here what is considered a relevant publication to you.]

# Constraints

[Describe all constraints you have and whether they can be circumvented.]

# Goals

[Describe the goals you want to achieve by using or implementing something described in a publication.]

# Final Scores

Use a 1 to 10 relevance score. The score should reflect the how well the publications fits the items in [Relevance](#relevance), [Constraints](#constraints) and [Goals](#goals).

...

Generate the following **Scores**:

- **Overall score**: 1 to 10.
- ...

# Output

- **Summary**: Title, authors, source, and core idea.
- **Evidence**: Abstract, PDF, project page, code, video, supplemental, or pasted metadata.
- **Measurements**: 
- ...

Note that some fields are being referenced in the filter-publications skill that drives it, most importantly the Goals, Constraints and Relevance sections. Filling these sections should already give you a working set to get a summary of the inputs.

create_feed.sh

This script reads the Markdown report from the agent, converts it into HTML and creates a feed file ready to deploy to a host where feed readers can subscribe to it. It only adds one entry. This could benefit from some polishing later.

#!/bin/sh

REPORT=$(pandoc -f markdown -t html $1)
DATE=$(date +%Y-%m-%d)
UPDATE=$(date -R)
GUID=$(echo $REPORT | sha256sum | awk '{print $1}')

cat > "$2" <<EOF
<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
  <title>Publication Filter Feed</title>
  <generator>Agent OPML</generator>
  <lastBuildDate>$UPDATE</lastBuildDate>
  <item>
    <author>Agent OPML</author>
    <title>Daily Report - $DATE</title>
    <description><![CDATA[ $REPORT ]]></description>
    <pubDate>$UPDATE</pubDate>
    <guid>$GUID</guid>
</item>
</channel>
</rss>
EOF

The Cronjob

A small script run.sh to implement the pipeline: Fetch new publications, run OpenCode to filter those and upload the resulting feed item.

#!/bin/sh

FEED_FILE="feeds.opml"
OUTPUT_DIR="./output"
JSON_FILE="$OUTPUT_DIR/new_entries.json"
ARCHIVE_DIR="$OUTPUT_DIR/$(date +%s)"

. ~/.venv/bin/activate

cd ~/ai-paper-filter

# create output dir
mkdir -p $OUTPUT_DIR

# fetch publications
python fetch_publications.py -i $FEED_FILE -o $JSON_FILE

# generate filtered report
if [ -f $JSON_FILE ]; then
    ~/.opencode/bin/opencode run "Use filter-publications on $JSON_FILE"

    # upload new feed file
    sh create_feed.sh $OUTPUT_DIR/report.md $OUTPUT_DIR/report.xml    
    curl -T $OUTPUT_DIR/report.xml "ftp://USER:PASS@MYFTP/PATH/report.xml"

    # archive everything
    mkdir -p $ARCHIVE_DIR
    mv $OUTPUT_DIR/report* $ARCHIVE_DIR

    rm $JSON_FILE
fi

Put this into a Cron job by running crontab -e on your target machine:

0 8 * * * ~/run.sh > log.txt

This will execute the pipeline every day at 8:00 in the morning and produce a new report. If fetch_publications.py does not produce any new entries, the pipeline will not run. If it does however, the findings will be uploaded as a feed file to some FTP after the agent finished execution.

Future ideas

More Extraction Tooling

The agent has to read or interpret content, and in some cases it boils down to the extraction mechanism behind it all: Scrape a URL and convert it to Markdown. Through my work on RSS-Librarian I’ve already found ways to deal with text extraction. For non-raw-text document formats Markitdown and Pandoc already do a great job. I want to find ways however to also address videos (maybe generate dubs automatically) and presentations (either XLSX or PDF), which are harder to parse because of missing structure tags.

Recommendation system for RSS categories

A second thing I plan to test is my own recommendation algorithm for video subscriptions from Youtube/Odysee/Vimeo/PeerTube/etc. A general complaint from those who never grew up on RSS is that subscribing to video channels can quickly drown out good videos in a lot of nonsense (shorts mostly, but also irrelevant videos from channels that veer off topic), whereas the Youtube recommendation algorithm can easily get sidetracked if you click on one wrong video and suddenly everything recommended is just funny slop videos.

I am testing a skill that ingests various video platform feeds (I continue to be amazed that these still exist on Youtube at all), sorts out feed items based on their description and the subtitles downloaded by Markitdown, and then repackages what is left into a new RSS feed that I subscribe to. This feed has one nice property: It is not a discovery feed, meaning the algorithm can never steer from its original goal of giving me only that what I subscribe to. It merely removes entries rather and recommends the highlights in the data stream.

Agentic RSS

Taking the same idea further I can imagine expanding this recommendation algorithm to the entire subscription library: One could subscribe - very liberally - to RSS feeds and end up with thousands of items per day, but the agent runs through them and instead of generating a report, decides whether to keep an item or not, then attaches the ones that were kept to a filtered feed. The agent can have arbitrary many input feeds, but you only ever subscribe to the output feed that has the sorted-out content based on custom rules.

I currently run several digests similar to the paper filter already, but I’d like them to act as pure filters on feed files by abstracting some of the scripting logic more.

Conclusion

This pipeline is not perfect, but has reduced the time I spend on reviewing new publications (especially around SIGGRAPH each year) significantly. Instead of mindlessly eyeballing loads of paper titles and spending hours discarding irrelevant stuff I can concentrate on the things that matter to me the most. Sure the agent might miss something, but so do I when I stare at thousands of unread items in my feed-reader. All I keep doing now is adding publications I find on random sources to a RSS-Librarian feed and adjust the OPML every now and then.

The setup is minimal, and the algorithm behind it all is not some opaque logic from a subscription service, but something I fully control end-to-end. This particular use-case might appeal to you as well: Wading through tons of items for sorting is a common task almost everyone will run into. You may write a paper and need to find new related publications to yours in a sea of articles. You may have too many RSS items in your feed reader. You may need to setup a more in-depth filtering mechanism than a simple keyword search.

Whatever it is, if your task boils down to a for loop with some slightly more complex word-bingo search inside a bunch of feeds, and you’ve set up this process as well, I’m interested in your story. Write a blog post about it perhaps?

 2026-05-30
 AI RSS
 Comments