STAR-NT: Spatiotemporal Acceleration of Real-Time Neural Transparency Rendering

Grigoris Tsopouridis et al. and Tobias Alexander Franke Proceedings of Computer Graphics International 2026

Abstract

Neural order-independent transparency delivers high-quality rendering of overlapping transparent surfaces, but its geometry passes and network input generation remain costly, particularly on mobile and legacy hardware. We present a spatiotemporal acceleration framework that exploits spatial and temporal coherence to reduce this overhead while preserving visual quality. Spatially, we use adaptive quadtree-based screen-space subdivision to scale geometry pass resolution according to local color variance. Temporally, selected frames reuse the previous transparency result through depth-based reprojection instead of full rendering. Together, these optimizations reduce rendering cost and integrate efficiently into existing real-time rendering pipelines.

Preview

Supplemental Video

Links

• Paper
• Arxiv
• Arxiv HTML
• CGRG
• CGI 2026
• Video
• Bibtex

Same procedure as last year

We do not need to debate the merits of AI; this has been discussed to death on the web and reminds me of one of the many programming language wars I was involved in in the early 2000s, where a single language was somehow the solution to world hunger and memory allocation bugs.

I wanted to talk about a part of my day to day work. I need to evaluate a lot of publications - blog posts, articles from various hardware vendors, Github releases, papers, conference programs … - to judge if they match our business use case or can be feasibly integrated into a prototype. Naturally, I have a lot of constraints around what I can and cannot do: Something must be open source and MIT or equivalently licensed, must meet certain hardware requirements, must be using a subset of GPU APIs, must be part of a certain graphics domain etc.

I read a lot of sources every day that publish new and interesting stuff, like Jendrik Illner's most excellent Graphics Programming Weekly blog, individual bloggers, conference programs and more.

As an example, a single SIGGRAPH conference can sport around 100 publications that I need to review. Each publication comes with a session, title, an abstract and a full PDF. Naturally, there's a progression how I get to the juicy parts: Discard sessions I don't care for first, search for interesting sounding titles, review their abstract and eventually keep the most interesting matches to read later in full.

I am subscribed to several conferences, around 250 personal blogs and several newsletters of companies, all in the computer graphics domain, all of them crunching out several articles a day. This takes an insane amount of time off from my day, every day, and the process is almost completely monotonous. Clanker-like one might say.

So here's my AI use-case, and how my obsession with RSS feeds directly into all of this.

Pipeline Overview

Agent OPML is a local LLM-based agent (NanoClaw, Pi, OpenCode, OpenClaw, whatever) that uses a tool to fetch publications from various sources and serializes all new entries into a fresh JSON file. Each entry looks like this:

{
    "url": "...",
    "title": "...",
    "authors": "...",
    "summary": "...",
}

Once the data is fetched, the agent's task, described in a skill, is to analyze each entry. The agent first analyzes the summary (which in many cases is the entire publication pulled via RSS), and if this is unsatisfactory - because it is just a shortened RSS <description>, is metadata about the publication but linking to it or is a URL of a video - go to the original url and try to fetch more using extra tooling. For instance in case of Arxiv, there may be a HTML version of the paper, and if not there may be a PDF instead, in which case the agent proceeds to download the file and converts it to Markdown for processing.

The skill describes what is interesting, what sounds interesting, etc. Each entry is analyzed for its relevance and goals, and more importantly if it fits within certain constraints. If all of these criteria are met, create a mini-summary and rating why this was deemed useful. The report is produced as a Markdown document, with annotations into a JSON file. Both are combined and attached to a new feed file that is served on some httpd.

The outcome is this: A single feed I can subscribe to which updates me daily with those publications I truly want, and the pipeline/algorithm to judge all of it entirely under my control.

Routing data to the agent

The input for the agent is a stream or feed of new publications that is updated regularly.

To do this, we will use an OPML file. For the uninitated: An OPML is a XML file pointing to multiple URLs of feeds. A feed in turn is a structured file containing blog posts or any other updates from a webpage, and those come in three major formats: Atom, RSS 2.0 or JSON Feed. Most importantly however is that everything in those files is machine-readable. Usually OPML files and feeds are used with a feed-reader such as NetNewsWire and are a simple way to store all your subscriptions.

We are going to use the OPML instead to let the agent automatically fetch new content from webpages for automagic analysis. The OPML should capture every feed of information that you'd have to wade through manually, but want the agent to do it for you.

To use my interests as an example, the OPML contains the following types of feed subscriptions:

• GPU Vendor: ARM, NVIDIA, Qualcomm, AMD, …
• Graphics API: DirectX blog, Khronos blog, Apple Metal, SLANG, …
• Game Engines: Godot, Unreal, Unity, O3DE, Unigine, OGRE, Stride, …
• Tools: Renderoc, Pix, NSIGHT, …
• Research Blogs: Activision, EA Seed
• Paper Feeds: Arxiv CS section, The Journal of Graphics Techniques (JGCT), Siggraph Digital Library entries, …
• Author Feeds: DBLP and Google Scholar, individual blogs
• Github Commits: Microsoft hlsl-spec, Gigi, …
• Conference Social Media: GPC, GDC, Siggraph, EGSR, EG, I3D, …
• Extras: One feed set up using RSS-Librarian

The last two here are worth elaborating on:

• Conference social media feed might sound odd at first: These are subscriptions from Mastodon, Bluesky and Twitter (through Nitter) that will have a special rule in the skill to catch announcements of a program URL for an upcoming conference. This way, the agent can grab more publications before they might appear anywhere else.
• The Extras feed serves the outlier case: If some article source is not captured by the OPML, i.e. some interesting but completely random blog post that appears somewhere you cannot subscribe to or that is otherwise irrelevant, then this individual article can be added to the Extras feed manually.

The OPML is used when running a script called fetch_publications.py, which goes through every subscription one by one, fetches the feeds and then combines all new entries it has not seen since the last run into a JSON file that is then used by the agent for analysis.

Setting up Agent OPML

Prerequisites

This setup includes a bunch of things:

• A spare Raspberry Pi (optional)
• Python
• OpenCode
• Markitdown
• Pandoc
• Your OPML file
• A bunch of scripts

Start by setting up a Raspberry Pi with Raspbian, and create a separate non-sudo account for the agent. If you want to run this on your local machine, you can skip this step. Then simply run the following steps.

On your sudo account:

$ sudo apt install pandoc cython3

On the agent account:

$ curl -fsSL https://opencode.ai/install | bash
$ pip -m venv ~/.venv
$ . ~/.venv/bin/activate
$ pip install markitdown feedparser

Code snippets

fetch_publications.py

This mini feed-aggregator fetch_publications.py is a simple Python script: Loop through a list of subscriptions from an OPML given via argument --opml and fetch the feeds, parse feed entries, reformat them into JSON dictionaries and use the URLs as IDs to remove duplicates. The outputs are written to a file given by --output that contains the extracted data. A temporary file called feed_state.json is kept in the same output directory to mark already visited articles.

#!/usr/bin/env python3

import json
import hashlib
import requests
import feedparser
import argparse
import xml.etree.ElementTree as ET

from fake_useragent import UserAgent
from typing import List, Dict, Any, Optional
from pathlib import Path

def load_opml(opml_path: str) -> List[str]:
    tree = ET.parse(opml_path)
    root = tree.getroot()
    urls = []
    for outline in root.findall(".//outline"):
        url = outline.attrib.get("xmlUrl") or outline.attrib.get("url")
        if url:
            urls.append(url)
    return urls

def load_state(path: str) -> Dict[str, List[str]]:
    if not Path(path).exists():
        return {}
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)

def save_state(path: str, state: Dict[str, List[str]]):
    with open(path, "w", encoding="utf-8") as f:
        json.dump(state, f, ensure_ascii=False, indent=2, default=st)

def entry_id(e: Dict[str, Any]) -> str:
    candidate = e.get("id") or e.get("guid") or e.get("link") or ""
    if not candidate:
        composite = (e.get("title","") + e.get("summary","") + e.get("published","")).encode("utf-8")
        return hashlib.sha256(composite).hexdigest()
    return candidate

def normalize_entry(e: Dict[str, Any], feed_url: str) -> Dict[str, Any]:
    return {
        "title": e.get("title"),
        "url": e.get("link"),
        "summary": e.get("summary"),
        "authors": [a.get("name") for a in e.get("authors", [])] if e.get("authors") else [],
    }

def fetch_feed(url: str, timeout=20) -> Dict[str, Any]:
    headers = { "User-Agent": UserAgent().random }
    resp = requests.get(url, headers=headers, timeout=timeout)
    
    try:
        resp.raise_for_status()
    except:
        # retry without user agent
        requests.get(url, headers={}, timeout=timeout)

    resp.raise_for_status()
    
    return feedparser.parse(resp.content)

def load_existing_entries(path: str) -> List[Dict[str, Any]]:
    if not Path(path).exists():
        return []
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)

def save_entries(path: str, entries: List[Dict[str, Any]]):
    with open(path, "w", encoding="utf-8") as f:
        json.dump(entries, f, ensure_ascii=False, indent=2, default=str)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("-i", "--opml")
    parser.add_argument("-o", "--output")

    param = parser.parse_args()

    if (not param.opml or not param.output):
        print("Parameters missing")
        return

    feeds = load_opml(param.opml)
    if not feeds:
        print("No feeds found in OPML.")
        return

    state_file = Path(param.output).with_name("feed_state.json")
    state = load_state(state_file)

    new_entries: List[Dict[str, Any]] = []

    for feed_url in feeds:
        try:
            parsed = fetch_feed(feed_url)
        except Exception as ex:
            print(f"Failed to fetch {feed_url}: {ex}")
            continue

        seen_for_feed = set(state.get(feed_url, []))
        new_ids = []
        for e in parsed.entries:
            eid = entry_id(e)
            if eid in seen_for_feed:
                continue
            ne = normalize_entry(e, feed_url)
            new_entries.append(ne)
            new_ids.append(eid)
            seen_for_feed.add(eid)

        state.setdefault(feed_url, [])
        state[feed_url].extend(new_ids)

    if new_entries:
        save_entries(param.output, new_entries)
    else:
        print("No new entries found.")

    save_state(state_file, state)
    print(f"Collected {len(new_entries)} new entries")

if __name__ == "__main__":
    main()

Re-running the script should only produce a file if any new entries have been found since the last run.

convert-pdf-to-md.sh

OpenCode has a directory ~/.config/opencode/tools which contains small scripts the agent can use to do something without requiring it to go through the process manually, thus saving tokens.

This tool uses markitdown to convert a PDF to Markdown. This is helpful when the agent downloads a PDF and needs to process/read it. The easiest way to do that is to simply convert it to a structured text file.

#!/bin/sh

INPUT=$1
OUTPUT="${INPUT%.*}"

markitdown $INPUT > $OUTPUT.md

The Filter Skills

This is the heart of the analysis and filtering process. These files describe what the agent should do, structured into these three general steps:

1. Read the combined entries that fetch_publications.py spits out
2. Analyse each entry if it's worth your time
3. Create a report of what passed the filter

First is the filter-publications skill, which generically describes how to parse through the JSON file with all the new entries and how to collect them together. Save it as ~/.config/opencode/skills/filter-publications/SKILL.md.

---
name: filter-publications
description: Given a list of new publications, find those relevant to our business use-case and produce a Markdown report on the agents findings.
---

# When to use this skill

Use this skill when the user wants the agent to evaluate and filter publications such as papers, blog posts, announcements, conference proceedings and so on.

Rank each publication by whether its idea, approximation, pipeline, or result could realistically transfer to the users specified use-case.

Be conservative and evidence-grounded. Clearly separate what the publication actually measured from what the agent infers as possible goals that can be reached with high relevance. Do not invent performance estimates, deployment claims, or results that are not supported by the inspected sources.

Use these three steps outlined below:
1. Read Input
2. Analyze Publications
3. Create Output

# Read Input

The agent is given one JSON file by the user (most often called `new_entries.json`). It contains a list of publications, where each entry has: a `title`, a `url`, a `summary`, and a list of `authors`. 

If the file the agent was given by the user is empty or does not exist, immediately halt and do not progress any further! Do not read any other json files or try to get data from somewhere else.

Each entry represents one new publication that was just discovered. The agent will need to analyze each one independently for its merits. Try to read the `summary` first. The summary may or may not contain the full publication. If it seems too short, try to open the URL and read the contents from there instead. If the `summary` is a short message from a social media page and contains a URL, follow that URL, especially if it contains a full program of a scientific conference.

Execute the [Analyze Publications](#analyze-publications) forstep described in the next section for all entries.

# Analyze Publications

## Required behavior

1. Read one publication entry source from the supplied JSON list.
2. Extract or read the best available information from the `summary` field.
3. Summarize the publication's core idea, evidence, and claimed results.
4. Evaluate the publication using the `review-publication` skill and produce an assessment with a clear verdict.

Report missing abstracts, inaccessible PDFs, broken links, paywalls, failed document extraction, low-quality extracted text, missing project pages, missing code, or insufficient evidence.

## Execution strategy

Do not treat a preliminary title/session-based inference as a confirmed analysis. If only title, session, program metadata, or DOI was inspected, label the evidence as **Limited** and make the score conservative.

Before finalizing top recommendations, do a detailed inspection of the strongest candidates. A publication should not appear as a final **Validated Top Recommendations** with a high score unless the agent has inspected substantive evidence such as an abstract, project page, PDF/text extraction, code repository, results, or limitations. If a publication cannot be analyzed in detail because the PDF is too large, paywalled, inaccessible, or time-limited, list it as a **Potential Top Candidates** rather than presenting it as fully validated.

These recommendations should not be homogeneous. Prefer a diverse shortlist that covers different topics that match the *Relevance* and *Goals* sections of the `review-publication` skill.

Prefer lightweight sources first: program page, DOI or publisher pages, abstracts, project pages, code repositories, videos, and supplemental descriptions. Do not download very large PDFs or supplemental files unless necessary. If a large file is skipped, report that limitation.

# Create Output

Create a subdirectory called `output`. All files the agent generates, even intermediate files, need to be written into this directory!

## Markdown report

Create a file called `report.md`. The file must start with a frontmatter. Use this exact style, with no blank line before the closing delimiter:

```yaml
---
title: Daily Report
author: Agent OPML
date: \today
documentclass: report
colorlinks: true
urlcolor: Maroon
linkcolor: Maroon
---
```

Do not modify this frontmatter. The report must only use Markdown headings, bullet points, and tables. Do not use `---` after the frontmatter, and no HTML tags. Structure the report into the following sections and no other headings:

1. **Overview**: A summary - as a list - of the overall statistics of the analysis, how many entries the agent processed, the themes the agent discovered and what the agent discarded. The list can be structured by source or category.
2. **Validated Top Recommendations**: Publications supported by substantive evidence. Use the the output of the `review-publication` skill for each publication.
3. **Potential Top Candidates**: Publications that look promising from title/session/metadata but lack enough evidence for a confident recommendation. Use the the output of the `review-publication` skill for each publication.
4. **Overall Verdict**: A summary of the agents findings and the papers that were recommended.

## JSON list

Create a file called `report.json` which follows the [Markdown report](#markdown-report), and use the JSON outputs from `review-publication` to populate the `validated_top_recommendations` and `potential_top_candidates` lists.

```json
{
  "overview": "{OVERVIEW}",
  "validated_top_recommendations": [],
  "potential_top_candidates": [],
  "overall_verdict": "{OVERALL_VERDICT}"
}
```

In this file you will need adjust and modify the # Analyze Publications section and/or provide a customized skill review-publication that judges and summarizes each individual entry the filtering process goes through.

Next is the review-publication skill that will guide the agent to judge an individual publication. Save it as ~/.config/opencode/skills/review-publication/SKILL.md.

---
name: review-publication
description: Judge a publication based on relevance, constraints and goals, and describes how to score the publication
---

# When to use this skill

The user asks the agent to produce a summary or judgement of a publication.

# Preprocessing

- If the publication is a social media post ...
- If the publication is a video, first ...
- ...

# Relevance

[Summarize here what is considered a relevant publication to you.]

# Constraints

[Describe all constraints you have and whether they can be circumvented.]

# Goals

[Describe the goals you want to achieve by using or implementing something described in a publication.]

# Final Scores

Use a 1 to 10 relevance score. The score should reflect the how well the publications fits the items in [Relevance](#relevance), [Constraints](#constraints) and [Goals](#goals).

...

Generate the following **Scores**:

- **Overall score**: 1 to 10.
- ...

# Output

- **Summary**: Title, authors, source, and core idea.
- **Evidence**: Abstract, PDF, project page, code, video, supplemental, or pasted metadata.
- **Measurements**: 
- ...

Note that some fields are being referenced in the filter-publications skill that drives it, most importantly the Goals, Constraints and Relevance sections. Filling these sections should already give you a working set to get a summary of the inputs.

create_feed.sh

This script reads the Markdown report from the agent, converts it into HTML and creates a feed file ready to deploy to a host where feed readers can subscribe to it. It only adds one entry. This could benefit from some polishing later.

#!/bin/sh

REPORT=$(pandoc -f markdown -t html $1)
DATE=$(date +%Y-%m-%d)
UPDATE=$(date -R)
GUID=$(echo $REPORT | sha256sum | awk '{print $1}')

cat > "$2" <<EOF
<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
  <title>Publication Filter Feed</title>
  <generator>Agent OPML</generator>
  <lastBuildDate>$UPDATE</lastBuildDate>
  <item>
    <author>Agent OPML</author>
    <title>Daily Report - $DATE</title>
    <description><![CDATA[ $REPORT ]]></description>
    <pubDate>$UPDATE</pubDate>
    <guid>$GUID</guid>
</item>
</channel>
</rss>
EOF

The Cronjob

A small script run.sh to implement the pipeline: Fetch new publications, run OpenCode to filter those and upload the resulting feed item.

#!/bin/sh

FEED_FILE="feeds.opml"
OUTPUT_DIR="./output"
JSON_FILE="$OUTPUT_DIR/new_entries.json"
ARCHIVE_DIR="$OUTPUT_DIR/$(date +%s)"

. ~/.venv/bin/activate

cd ~/ai-paper-filter

# create output dir
mkdir -p $OUTPUT_DIR

# fetch publications
python fetch_publications.py -i $FEED_FILE -o $JSON_FILE

# generate filtered report
if [ -f $JSON_FILE ]; then
    ~/.opencode/bin/opencode run "Use filter-publications on $JSON_FILE"

    # upload new feed file
    sh create_feed.sh $OUTPUT_DIR/report.md $OUTPUT_DIR/report.xml    
    curl -T $OUTPUT_DIR/report.xml "ftp://USER:PASS@MYFTP/PATH/report.xml"

    # archive everything
    mkdir -p $ARCHIVE_DIR
    mv $OUTPUT_DIR/report* $ARCHIVE_DIR

    rm $JSON_FILE
fi

Put this into a Cron job by running crontab -e on your target machine:

0 8 * * * ~/run.sh > log.txt

This will execute the pipeline every day at 8:00 in the morning and produce a new report. If fetch_publications.py does not produce any new entries, the pipeline will not run. If it does however, the findings will be uploaded as a feed file to some FTP after the agent finished execution.

Future ideas

More Extraction Tooling

The agent has to read or interpret content, and in some cases it boils down to the extraction mechanism behind it all: Scrape a URL and convert it to Markdown. Through my work on RSS-Librarian I've already found ways to deal with text extraction. For non-raw-text document formats Markitdown and Pandoc already do a great job. I want to find ways however to also address videos (maybe generate dubs automatically) and presentations (either XLSX or PDF), which are harder to parse because of missing structure tags.

Recommendation system for RSS categories

A second thing I plan to test is my own recommendation algorithm for video subscriptions from Youtube/Odysee/Vimeo/PeerTube/etc. A general complaint from those who never grew up on RSS is that subscribing to video channels can quickly drown out good videos in a lot of nonsense (shorts mostly, but also irrelevant videos from channels that veer off topic), whereas the Youtube recommendation algorithm can easily get sidetracked if you click on one wrong video and suddenly everything recommended is just funny slop videos.

I am testing a skill that ingests various video platform feeds (I continue to be amazed that these still exist on Youtube at all), sorts out feed items based on their description and the subtitles downloaded by Markitdown, and then repackages what is left into a new RSS feed that I subscribe to. This feed has one nice property: It is not a discovery feed, meaning the algorithm can never steer from its original goal of giving me only that what I subscribe to. It merely removes entries rather and recommends the highlights in the data stream.

Agentic RSS

Taking the same idea further I can imagine expanding this recommendation algorithm to the entire subscription library: One could subscribe - very liberally - to RSS feeds and end up with thousands of items per day, but the agent runs through them and instead of generating a report, decides whether to keep an item or not, then attaches the ones that were kept to a filtered feed. The agent can have arbitrary many input feeds, but you only ever subscribe to the output feed that has the sorted-out content based on custom rules.

I currently run several digests similar to the paper filter already, but I'd like them to act as pure filters on feed files by abstracting some of the scripting logic more.

Conclusion

This pipeline is not perfect, but has reduced the time I spend on reviewing new publications (especially around SIGGRAPH each year) significantly. Instead of mindlessly eyeballing loads of paper titles and spending hours discarding irrelevant stuff I can concentrate on the things that matter to me the most. Sure the agent might miss something, but so do I when I stare at thousands of unread items in my feed-reader. All I keep doing now is adding publications I find on random sources to a RSS-Librarian feed and adjust the OPML every now and then.

The setup is minimal, and the algorithm behind it all is not some opaque logic from a subscription service, but something I fully control end-to-end. This particular use-case might appeal to you as well: Wading through tons of items for sorting is a common task almost everyone will run into. You may write a paper and need to find new related publications to yours in a sea of articles. You may have too many RSS items in your feed reader. You may need to setup a more in-depth filtering mechanism than a simple keyword search.

Whatever it is, if your task boils down to a for loop with some slightly more complex word-bingo search inside a bunch of feeds, and you've set up this process as well, I'm interested in your story. Write a blog post about it perhaps?

Knowledge is power

Many of you may know me as a huge Warhammer 40,000 nerd. I like the absurd and over-the-top nature of it, the grimdark setting and stagnant vision of the future where, what is left of humanity, slowly rots from within. Many small quotes encapsulate the universe very well, but the following one from Dawn of War is one of my favorites:

"Knowledge is power, hide it well."

Astartes Librarian

Similar to a Black Mirror episode, this quote perfectly reflects our current strange reality of hiding most of the Internet's articles worth reading behind walls of ads, paywalls or questionable design ideas. Knowledge has not become entirely inaccessible, but similar to what happened in the academic community there is a cost to accessing articles: Your privacy, your sanity, or both.

Saving articles for later-reading has turned into a small ecosystem: Safari has a built-in method to store pages. Mozilla, until recently, ran Pocket and then shut it down. Instapaper remains stable and clean commercial solution I really like, and on the open-source front Wallabag is a viable alternative. There are great offline apps such as Later. Saving articles can also take an archival character, which ArchiveBox supports nicely.

I already have my feed-reader where I read everything else. All of these services are extra layers and apps that are too heavy for someone who just wants to read articles, offline, in a clean layout, without ads, without intrusive layouts or pop-up videos or spyware. This particular problem has already been solved many years ago, so rather than re-inventing the wheel I can instead leverage existing, well-established web-standards to let me read arbitrary articles from the web in the comfy environment of my feed-reader, next to my subscriptions.

If you subscribe via RSS/Atom to blogs, podcasts or Youtube channels, then RSS-Librarian is one more way for you to tunnel the Internet through your feed-reader.

Version One

RSS-Librarian is a read-it-later service for RSS purists. You can store articles from the web in your own personal RSS/Atom feed and use your favorite feed-reader software to read your stored articles later. RSS-Librarian uses no database and works without accounts.

I am happy to announce that RSS-Librarian has now arrived at Version One and is ready for production. Since my first article a lot has happened.

The original RSS-Librarian was kept extremely simple at just 211 lines of code in a single file. RSS-Librarian is written in PHP to make it as easy as possible to drop it on a VPS (virtually zero hosters come without PHP support these days). The script receives a URL as input, runs it through a readability service to extract the raw content, and then attaches it to a randomly generated feed file, no database required. The random feed can be revisited to add more articles to it. Once subscribed to it via a feed-reader, the reader software does the rest. I personally am using readers like FeedMe which download the full content of a feed, images included, to my phone, making all stored articles available offline for later-reading during for instance flights.

Since then, I have addressed several things:

1. Bugs: The code structure was mixed up with HTML quite badly. After some iterations it got better and easier to maintain. Still not perfect, but much easier to read. Separating code from structure also got rid of a lot of state-tracking bugs.
2. External Configuration: RSS-Librarian now reads its configuration from a JSON file rather than needing source modifications. The Instance Info section at the footer of the webpage provides an overview of the current configuration.
3. Feed Validation: The original RSS 2.0 output created invalid feeds. This has been fixed.
4. Local Readability: Rather than relying on a centralized external readability service, one can now download an extra library and run the extraction of article content locally.
5. Disabling Content Extraction: Content extraction can now be turned off. Whilst this seems strange, some feed-readers like FeedFlow do not read RSS item content anymore and instead rely on a built-in reader-view similar to the ones in browsers to pull content from the URL directly. To save redundant space in these cases this option will skip adding any content to the feed.
6. Atom Support: The Atom feed format is slightly more convenient when it comes to storing an author without an email address, and some other small things. I decided to leave the format RSS-Librarian uses up to the user. One can easily switch between both and RSS-Librarian will automatically convert between formats for existing feeds once a user adds a new article.
7. User Onboarding: On the original test-instance I noticed many feeds were repeatedly created with just one article in them, suggesting users forgot to bookmark their randomly created feed ID. Now users are onboarded with a warning before adding their first article.
8. Mobile CSS: RSS-Librarian now has a beautiful and easy to tap-on user interface for mobile.
9. iOS Share integration: With iOS Shortcuts I was able to create a share-integration that allows any app to share a URL with your bookmarked RSS-Librarian feed.
10. Stylization: Both custom CSS for RSS-Librarian and XSLT for the feeds can now be configured. When opening the feed's XML file one now gets a full preview of all stored articles in a nicely formatted webpage rather than just raw XML. If no XSLT is present the code will default to feedreader.xyz for a quick preview of the feed. Both the logo and the FavIcon can be configured as well.
11. Administration: All users of an RSS-Librarian instance are pseudonymous, meaning they have an ID but no other identifier. To inform users of the instance about downtimes, changes or any other admin related notices, a special ID is pre-generated on first start up which lets an admin attach a message to all existing user feeds. Furthermore, the admin interface allows cleaning up abandoned feed files. The configuration also contains a field to add contact information for the instance administrator, which defaults to the Github issues of RSS-Librarian. The administrator ID has its own feed that keeps track of any administrator operations and URLs added by any feed managed on the instance.

There are also two contributions I want to mention:

1. Alex Faxå cleaned up the code and added more robust cURL parameters to make RSS-Librarian appear more like a regular browser.
2. Abe Ramseyer added the ability to see the list of stored items and optionally remove them again.

To summarize, the code is now much easier to deploy, much easier to modify, has less bugs and creates valid RSS and Atom feeds. The user interface is visually better and easier to use on a phone.

A full documentation on how to use RSS-Librarian, how to self-host and configure it and an FAQ is now available on the Github Wiki of the project.

Store it well

Most importantly though, I want to announce the official RSS-Librarian instance hosted at

https://www.rsslibrarian.ch/librarian.php

The old test instance hosted on hstn.me had several issues that caused subscription errors, particularly with online readers like FreshRSS, since it's HTTPS certificate did not work correctly and regularly rerouted linked XML files to error pages. Additionally hstn.me would often attach extra URL parameters that needed to be filtered out from that particular instance. W3C feed validation for the same reasons would regularly fail. For Version One therefore I wanted to unveil a stable main instance everyone can use for free. If you have any bugs to report, feature requests or PRs, do not hesitate to open a Github issue.

Join the great crusade against the enshitification of the web. We do not need to burn it all down and re-create it from the ashes, the foundational standards for a better web are already here.

Next-gen challenges for mobile game rendering

Tobias Alexander Franke Machine Learning Meetup Dublin

Abstract

Mobile hardware vendors are facing a growing number of game studios that seek to push AAA content on mobile platforms.

In this talk I present an overview of the requirements for next-gen games on mobile: AAA asset sizes, raytracing, volumetric lighting, transparency for foliage, global illumination, high-detail characters, elaborate postprocessing and supersampling, leveraging on-chip CPUs, GPUs and NPUs.

Each one of these categories needs to overcome certain hardware limitations before being feasible on mobile platforms. The goal is to focus on research of highly adaptive and scalable alternatives to algorithms currently in use in AAA games.

Links

• ADAPT Centere article
• Wayback Machine Archive

Traditional and Neural Order-Independent Transparency

Grigoris Tsopouridis et al. and Tobias Alexander Franke Proceedings of Eurographics 2025

Abstract

Order independent transparency (OIT) is a technique in computer graphics that allows for accurate rendering of transparent objects without the need to sort them in a specific order based on their depth. Traditional transparency methods often suffer from artifacts and inaccuracies due to this sorting process, especially in complex scenes with many overlapping transparent surfaces. OIT is important because it provides a more visually correct representation of transparent materials, ensuring that colors mix accurately and that all elements are rendered consistently, regardless of their draw order. This enhances realism in applications such as video games, simulations, and visual effects in films. The tutorial will provide an overview of traditional (exact, approximate and hybrid) and deep learning approaches to OIT and examine their scope, performance and accuracy.

Links

• Content
• Slides
• CGRG
• EG DigLib
• EG 2025
• Bibtex

Leave no stylesheet alive

I read a lot. I read a lot of papers for instance. They are easy to read, because they have a common format, by which I mean their appearance. This is achieved in the academic community largely by separating markup, the language that describes what - in an article - constitutes a headline, a paragraph and so on, and the formatting of thosed marked up properties, through a common Latex template such as the SIGGRAPH acmtog or using common formatters for simpler markup languages such as Markdown. This has one tremendous benefit: Reading a lot of articles is easy, because they all look the same.

I have rambled about the terrible state of the web before, twice actually. Most webpages have become unreadable, which describes a state where it is hard or impossible to find the actual content of a webpage amongst ads, banners, popups or other annoyances. This is especially disappointing if one opened said webpage with the expectation to just read some piece of text matching a headline.

It is ironic that HTML - being the markup language to differentiate content from style - has been abused by designers so much that a bunch of tools emerged to extract the content and throw away the style. Over a decade ago, when this problem became apparent, it was addressed with a bookmarklet called Readability reformatting webpages to reduce clutter, followed by browser implementations which added so called reader views, for instance in Firefox and Brave.

Now Read-it-later services solve two problems at once and make articles offline-readable: Instapaper, Pocket and the FOSS self-hostable Wallabag download and store bookmarked articles after running them through a readability tool, so that eventually all articles have the same layout and same formatting.

Motivation

As someone who reads a lot for research (papers, blog posts, news articles etc.) I wanted to have the simplest possible version of such a read-it-later service: All my bookmarked articles look the same and are readable offline .

Failure 1

My first experiment was to manually store PDFs of the reader view in Firefox. That's a lot of manual labor though, and syncing across devices requires to store all PDFs on some cloud service. It's also not easy to use on the go.

Failure 2

My second experiment was trying out Archivebox, a neat FOSS project that streamlines archiving. Archivebox will not just download a copy of a webpage, but run it through several polishing tools. One of them is a readability library, which stores a PDF and HTML version alongside the original article. However, I found Archivebox hard to deploy on a webhost with just PHP on it (basically the things you get for free) and it produces quite bit of extra data I don't need, but otherwise ticks almost all the boxes.

Failure 3

My third experiment ended in a personal frustration of mine: My workflow for reading anything I am interested in is by adding a star in my RSS reader to an article, which necessitates that anything I want to read is somehow a subscription or feed I can add to my RSS reader, and that isn't true for individual articles I stumble across. Sometimes this isn't even true for articles on blogs, where the blog is either too huge to subscribe to for just this one article, or (a sad occurunce these days) it doesn't have RSS at all, even though that is trivial to add.

Before switching to the most excellent NetNewsWire reader, I was fond of Reeder, which includes the functionality to add read-it-later services like Instapaper as accounts, showing the same interface for both RSS and read-it-later subscriptions. This, essentially, made Reeder my one-stop reading application, indifferent to whether the article came from a single URL I wanted to read or from any of my RSS feeds. Whenever there was an article I wanted to read, I'd simply push it to an Instapaper or Pocket account and then fetch it via Reeder. However, this had one major downside: I'd use a centralized service that gradually became a huge data dump, as more and more articles ended up in those accounts.

Failure 4

My fourth experiment involved RSS itself. NetNewsWire, like most RSS readers, is a minimalist, pure RSS reader, which means anything you want to star inside the application has to come from an RSS feed you can subscribe to.

Both Pocket and Wallabag allow you to subscribe to your personal collection of articles via RSS, essentially treating your article collection as some kind of blog with new posts appearing whenever you add an article. However, this approach had the same downside as when I was using Reeder: The read-it-later accounts become a huge dump after a while, because I was not really interested in either Pocket or Wallabag, only in redirecting saved articles to my RSS reader. This solution however was extremely close to optimal, with just the exception of me logging into my accounts in regular intervals to delete everything.

Requirements

From the four failures I learned that what I wanted was actually this:

Send a URL to a script, which runs it through readability and attaches the resulting content directly to an RSS feed. Ideally, I can create different feeds for different collections, for example one for work and for personal interests.

Specifically, I want to:

• Store single articles in a RSS reader application
• Avoid third-party read-it-later services such as Pocket, Instapaper or Wallabag
• Minimize the amount of necessary apps for reading articles
• Get rid of accounts and not sign up to anything
• Read articles (offline) in a readable format, but not categorize or store them indefinitely
• Synchronize stored articles to multiple devices
• Optionally be able to self-host the whole architecture

The outcome of this is a project I call RSS-Librarian, a read-it-later service for RSS purists. RSS-Librarian solves all of these bullet points with a single, self-hostable PHP file that extracts content from URLs using a readability service and adds what remains as new entries into a personal RSS feed, without requiring special libraries, a database or user accounts.

How it works

You can drop librarian.php onto any host that supports PHP with no other requirements. RSS-Librarian has two parameters: librarian.php?id=HASH&url=SOMEPAGE.

• id is a random ID for a personal feed. If this parameter is not supplied, RSS-Librarian will generate a new one and add a feed file corresponding to id into the subfolder feeds/.
• url is a URL you submit to RSS-Librarian, whose content will be extracted and added to the feed corresponding to id.

For each url posted to RSS-Librarian, the extracted content will be added to a RSS file derived from id - if it exists in the feeds/ folder. If it does not exist, it will be generated and written. The RSS file will store a maximum of 100 entries before removing the oldest one and adding the new one.

An example use-case

I will be using the this demo instance of RSS-Librarian in combination with NetNewsWire on MacOS (MacOS App).

I can recommend NetNewsWire on iOS (App Store Link), FeedMe on Android (Play Store Link, APK) and FeedBro for various Browsers (Firefox Addon, Brave/Chromium Addon). They all download offline copies of RSS feeds to your device ready for access even when disconnected completely.

Step 1: Create your personal feed

Assume we have the following article we want to store for later reading. First go to the RSS-Librarian instance. You'll be greeted by the following interface.

This instance is currently hosting 8 other feeds. Paste the article URL https://ohshitgit.com/ into the empty field and press Add to feed. Because you are a new user, RSS-Librarian will now generate a random, unique user ID for you.

On the next page you will see two important things:

1. Your personal URL: Store this URL in your bookmarks! This allows you to add more articles to your personal RSS feed.
2. Your personal RSS feed: This is your feed that you can subscribe to with your RSS reader application. It is unique and can only be managed with the personal URL above.

It is important - after adding your first link - to bookmark your personal URL somewhere so you can keep adding links to your feed (instead of creating a new one accidentally)!

Step 2: Open your personal URL

For this demo, the personal URL now links to http://alternator.hstn.me/librarian.php?id=ff4d8b605c2cffacd19639af2a1d3ff8712021d66431de86f425e0279a1e768a. RSS-Librarian, on first use, will create a random hash (in our case ff4d...768a). This hash will be used to store new URLs to your personal RSS feed, which is derived from that hash. You can think of this as your user ID.

Open the personal URL with a browser again.

With this page you can add more articles to your personal RSS feed. Let's add another article to the feed we just created.

You can now point your RSS reader to your personal RSS feed and check out the result. At the bottom of the page you can find additional tools:

• Feed bookmarklet: A bookmarklet you can use instead of the personal URL. This will add the currently open page to your personal RSS feed.
• Feed preview: If you have no RSS viewer at hand you can preview your personal RSS feed with this page.

Step 3: Add it to an RSS reader of your choice

Let's try this out with NetNewsWire on MacOS. Copy the link personal RSS feed, press the + symbol in NetNewsWire and add the URL.

After adding the feed, your stored articles appear in your RSS reader. Most RSS readers will automatically download all articles in a feed, meaning you also have a stored copy offline on the go, for instance when reading on a plane without WiFi.

If you click the title of the article, you will end up on the original URL (in the above sample our first article). If you open the feed itself - RSS-Librarian (ff4d) - you will jump back to your personal URL where you can add more articles to the feed.

If you do not have an RSS reader around, you can also click on Feed preview in the tools section of your personal URL to get a quick web-based view of the feed.

And that is pretty much it. By visiting your personal URL on a RSS-Librarian instance, you can add more articles to your personal RSS feed and they will become easy to read and storable offline by your reader software. You can subscribe to your personal RSS feed with as many readers as you want and even share that feed with others (which however will let them add articles to it too).

Deploying your own instance

Simply get a copy of librarian.php from the Github repository and put it on any VPS, Raspberry Pi or other webserver that has PHP. There are no other requirements. Create a directory called feeds right next to the file and give the webserver write access to that folder.

In librarian.php you can configure some globals:

• $g_max_items is the number of articles stored per feed. Adding a new article beyond that number will remove the last one in the feed. Reduce this number if you're tight on space.
• $g_dir_feeds is the directory where user feeds are stored. Rename this if you want to use a different directory on your instance.

Future work

There are a bunch of things I want to implement.

Protected shareable feeds

Currently the id given to RSS-Librarian is used to create a feed in the feeds/ directory that has the same name. This means that sharing this feed with others will allow them to access the personal URL as well and add articles. I'd like to change this behavior so that the feed's file name is generated by hashing the id again. This way, the user id and the feed id are separated but correlated in one direction: The id can be used to generate the corresponding feed file name, but not vice versa. This comes at the cost of usability: The feed currently links back to it's personal URL, which means you can quickly jump to the place where you can add more articles. This would need to be removed to not give away that page.

Remove FiveFilters dependency

RSS Librarian sends articles to FiveFilters, which extracts them into single RSS entries and then appends that entry to the locally hosted feed. This is not optimal, because the feed generation is essentially centralized and not independent. Instead, RSS-Librarian needs to do the extraction locally via the FiveFilters Readability library for PHP.

Maintenance

Every user can add a new feed at any time and as many as they want. An instance can quickly create a huge dump inside the feeds folder full of garbage feeds that are abandoned because users forgot to bookmark their personal URL, or because they lost the link somehow later.

On every new article added RSS-Librarian could run through the feeds/ folder and simply delete feeds that are too old, for instance those that had no new items added to them for more than 6 months. This is not an issue if the id gets re-used again: RSS-Librarian recreates files for ids when another article URL is added.

Share-feature for mobile

App based read-it-later services tie into the OS and let you share an article with the app, which is the default way to add an article quickly to the read-it-later list. RSS-Librarian however has no app and uses a webpage instead, which makes quickly adding an article to a personal RSS feed tedious.

Currently, one can get to the personal URL by simply opening the personal RSS feed URL. Ideally however I'd like a simple share button that adds some URL being shared from any app.

Main instance

An RSS-Librarian demo instance for testing is currently hosted here. The instance has a self-signed certificate and therefore many RSS readers will run into issues. Using HTTP instead is a no-go though. I want to eventually host a reliable main instance.

The Great Crusade against Web-Enshitification

I've been using RSS-Librarian just for myself for about half a year now and it's been a delight so far, given how compact the code is. I created two feeds for myself: One for articles I'm interested in personally, and one for stuff I need for work. I subscribe to my work feed using my personal devices and with FeedBro on my work machine, which means I can easily add and read articles I find for work everywhere, whereas the feed containing articles I'm just interested in personally is only in my feed readers on my own devices. It's all neatly separated and easy to manage.

Of course I wrote this for myself initially, so if you find it weird, confusing, chaotic, or you'd like to see a feature, please open an issue or let me know otherwise.

The web is terrible: Redux

The screams are getting louder, but the solutions keep getting dumber.

I know that my background as a graphics programmer is adding heavy bias to my point of view, but I really do not send data-oriented design lectures from Mike Acton to random Electron-app developers because I think their code is slow. Nevertheless, corporate web programmers have collectively given up on the notion that there is something called efficiency. (No time for a good rant? Go to the TL;DR!)

Case in point: Most news websites have tiny irrelevant content that hides between full-screen ads, newsletter popups, cookie warnings, autoplaying videos, and Javascript yanking mouse control away from you to add custom scrolling behavior. Look, for instance, at the following capture from Andrej Karpathy.

Of course this browser-violation is just the tip of the iceberg, as there is more code that is running in the background, streaming ever more useless stuff to the machine. Here is a sample captured by Lunasorcery.

Think about just how much all of this, per tab, eats into your CPU time for no good reason, heats up the machine and drains battery of your mobile device, even if you do absolutely nothing on that page. Soon mobile 4G won't provide adequate bandwidth to load a simple news article anymore. Part of the problem here is that modern web design is a lazy pieced together conglomerate of hilarious libraries such as the is-odd node module.

The pyramid-scheme continues with an even greater absurdity, also known as the is-even node module, which relies on - you guessed it - is-odd. To put that into perspective: is-even loads 254 bytes in addition to is-odd loading another 543 bytes for essentially this operation (n % 2) == 0 (12 bytes), or 6641% of the necessary bytes to load. You can imagine how the rest of your average page looks like. If you believe this is just an elaborate prank or a contrived example, I present to you a sample of this being used in a glob matcher/parser before someone had the decency to remove it, make a mistake in the process and summon the comment from hell.

So here we are, with your browser wading through mountains of garbage, to display a simple piece of text to you. One way to deal with this is to fully embrace Stockholm-syndrome and just pay the ransom some of these sites demand from you to make the pain go away. Then there are a variety of tools that have been introduced to combat these issues, ranging from reader-views built into browsers, using Readability services (just read that out loud) such as Wallabag, Pocket or Instapaper, or simply adding dozens of plugins into your browser to make pages look sane such as user scripts like GreaseMonkey and user stylesheets like Stylus to, and I quote:

Redesign your favorite websites

However, all of this trickery will still not get the major problem under control: Browsers need ever more powerful machinery to display the most mundane content to you.

Never fear, the age of cloud is here! So why not put the browser on a different machine that is 10x more powerful and STREAM what you see to your machine, that extra-layer will solve it!

Turns out that this idea is still a tad bit too ludicrous to be implemented these days, so lucky for us this project is dead.

Let's recap where we are right now:

• Your browser is as complex as an operating system
• Browser complexity is the source of many security issues
• (Corporate) websites abuse browser complexity at every opportunity
• Even if you counter the abuse with layers and layers of tools, the issue is getting worse over time
• There is no incentive for cooperations to reverse course
• New programmers are born into habits of being as inefficient as possible

The alternative

My number one suggestion is to leave most of the web entirely and pipe as much content as you can through RSS.

However, that doesn't take care of individual URLs and sites that simply don't have an RSS feed. Every day someone sends me a link to one of the various horrible social media sites that I need to then open in a browser.

This is where a new open-source trend comes to the rescue: The Alternative Frontend movement. Alternative frontends, as the name suggest, are websites that provide a different frontend to an existing webpage.

For instance, Invidious is a popular alternative frontend for Youtube. It provides access to anything on Youtube, however in a much cleaner interface with much less clutter, no ads, zero tracking and directly downloadable MP4s in place of the video streaming.

Right now there are many frontends to popular websites that extract their content and present it in a much saner way. These include Youtube, Twitter, Reddit, Tiktok, Imgur, Reuters, Quora, IMDB, Medium, Google, Stack Overflow, Fandom Wiki, Snopes… the list keeps growing, because most of the original pages are terrible.

A lot of these alternatives will use either public APIs or scraping to fetch the content of the original page and display it to you, but those details are not important.

This way please

One key question for me is how to use these frontends in a manner that is not intrusive, easy to use and allows me to open random links and get the best possible result.

The easiest way I have found is to use a redirector plugin for your browser. These plugins will detect a URL you enter into the browser, match it with a regular expression, and if a match has been found, replace it with another.

For instance, you might detect the pattern https://www.youtube.com/* and replace it with https://invidious.fdn.fr/$1.

A URL to https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw (the 3Blue1Brown channel) will therefore be redirected to https://invidious.fdn.fr/channel/UCYO_jab_esuFRV4b17AJtAw.

These plugins allow you to add multiple patterns for multiple webpages. I have created a configuration you can simply import without the need to add all these patterns by hand, which is available here.

Redirector plugins for Firefox/Librewolf/Torbrowser

• Redirector - General purpose redirector you can customize yourself
• LibRedirect - Redirector to popular alternative frontends, highly configurable
• Privacy Redirect - Redirector to popular alternative frontends, but limited in scope

Redirector plugins for Safari

• Redirect Web for Safari - General purpose redirector you can customize yourself (can load exported Firefox Redirector JSON)
• Privacy Redirect - Redirector to popular alternative frontends, but limited in scope

Load-Balancing

Many of the alternatives are hosted privately by volunteers and are called instances, which means there isn't one URL for one alternative, but many. This decentralization acts as a safety net: Banning one doesn't bring down the service itself and distributes the load.

As an example, there are many Invidious instances available. However, using one of them exclusively has two downsides:

1. It might stop working, get banned, rate-limited or shut down
2. If many users pile on one instance, it will be overloaded and most likely rate-limited more quickly

To solve this issue, ideally recognizing a pattern should redirect to a random instance instead of a fixed one, so that the load distributes equally across all hosted instances and prevents cases of rate-limiting or banning of an instance. This would need two things: For each alternative frontend a list of all available instances that is up to date, and the ability to forward to randomly selected URLs. The forwarding though is the problem.

A solution to this comes in the form of redirecting-gateways such as Farside. It's a simple page that lists frontends and their instances. For each frontend, there is one link at the top of each section which will forward you randomly to one of the instances. You can configure your redirector to forward your URL to use those as targets instead.

As an example, instead of redirecting https://www.youtube.com/* to https://invidious.fdn.fr/$1, you may want to redirect it to https://farside.link/invidious/$1, which will select a random Invidious instance and attach your $1 parameters to it.

I provide a simplified, self-hostable redirector-gateway called Tzeentch which is heavily inspired by Farside, but simpler to deploy. You can check out a demo instance here. It also adds a feature absent in Farside, which is to create a ready-made configuration for your redirector for all the frontends you select.

LibRedirect on the other hand maintains an internal list of instances where you can select one or more favorite ones that it chooses from randomly, but this is connected to a bit of manual labor every now and then.

Conclusion

The battle for a cleaner, saner and less intrusive web is lost.

Alternative frontends remove the need for a barrage of plugins to sanitize the web, are privacy-friendly and just generally provide a better experience. Redirector plugins using gateways which distribute the requests among frontend-instances will ensure decentralization and reduce the likelihood of instances getting rate-limited or banned.

I highly encourage you to redirect everything and check out this way of browsing! Just take a look at LibReddit and compare it with Reddit itself, for instance on a phone. Ask yourself: Is this better than the original?

Engine Design as a Holistic Adventure

Tobias Alexander Franke and Royal O'Brien Tales from the 3rd Dimension Podcast

Abstract

Most listeners know that Huawei isn’t just a mobile phone company. But many might not realize that it runs a global network of research institutes that are working on optimizing engine development. Tobias Alexander Franke, principal game engine architect at Huawei, talks with Royal about why the company prefers working with open source 3D engines over commercial 3D engines. Highlighting the benefits of different perspectives, he believes open source collaboration results in a more well-rounded engine. Tobias and Royal reflect on other hot topics like the early days of the Open 3D Engine (O3DE) in GitHub, the need for standardization for greater interoperability and the development of the Metaverse.

Tobias Alexander Franke, principal game engine architect at Huawei, is an experienced graphics engineer with a background in both graphics research and the gaming industry. He holds a PhD from TU-Darmstadt focused on augmented reality, relighting and global illumination.

Links

• Apple Podcasts
• Spotify Podcasts
• Deezer Podcast

Life in the metaverse will be revolutionary

Tobias Alexander Franke and Sandra O'Connell Irish Times

Abstract

I have been interviewed by the Irish Times on the topic of the Metaverse, what it is, why one would be interested in it, the big problem it is supposed to solve and how it will affect our lives. In this interview, I give a sober analysis of the Metaverse, and my goal is to cut out the hype.

Links

• Irish Times Article
• Wayback Machine Archive

A New Gaming Eco-system for Huawei

Tobias Alexander Franke Huawei GSTS 2022

Abstract

Cloud-based gaming services have seen varying success in practice. Where-as multiplayer games feature a wide variety of aspects which are hosted remotely, most games today run exclusively on one device and require the customer to adapt to new hardware requirements in regular cycles, usually tied to the current console generation. Streaming-based gaming services have in contrast failed so far, underestimating technological challenges as well as selling to the wrong market. Edge computing architectures are designed to distribute computational tasks to other nodes in a network or cloud. Huawei's involvement in the Open 3D Engine presents a unique opportunity to build a more resilient architecture for game engines that scale from one device to an arbitrary number. In doing so, a new gaming eco-system evolves as a business opportunity for Huawei.

Links

• Huawei GSTS 2022

Scalable Global Illumination for O3DE

Tobias Alexander Franke O3DECon 2021

Abstract

Implementing global illumination for games at scale (on mobile devices and up to high end PCs) is a hard problem. O3DE currently uses DDGI as a solution to provide both baked and real-time global illumination. However several key issues need to be addressed before this system is ready for production. In this talk I want to address those issues:

• Scalability and streaming
• Authoring with small or thin objects
• Probe aliasing
• Volume overlaps and blending
• Baking large scenes
• Strange mixtures of baked and real-time probe based volumes
• Baked lighting variants for day-night cycles or similar setups
• Re-using visibility buffers for other effects
• Going beyond diffuse GI

The idea is to highlight each issue, present a small solution (not necessarily the solution) based off experience from developing similar volume based GI systems in other engines, and make more people from the community - both artists and programmers - aware of these issues so we can collectively come up with a proper system that has an easy workflow and provides great results when computing GI for a project. The talk will close with an outlook to some experiments, novel workflows and how the same system could be used to provide GI for large open worlds, maybe even on mobile devices.

Supplemental Video

Links

• Slides
• O3DECon 2021
• Video
• Bibtex

In one of the more bizarre hiring encounters I've experienced, I sit together with my family at lunch when I receive a Twitter message followed by an email that can be paraphrased as Saw your online profile, would you be interested in joining Unity? A couple of weeks later I am in Copenhagen at the office. The atmosphere is relaxed, family-like even, and I am getting interviewed by someone called Joe. I am ignorant of course and do not realize this isn't Joe Shmoe, but the Joachim Ante, one of the founders and CTO of Unity.

My first day at Unity is 1st of January 2016, following my departure at Fraunhofer IGD after 2 years of doing a PhD. I'm getting on-boarded by David Helgason as the only graphics programmer in a room full of mathematicians. David shows a keen interest in everyone and their story, and wants to know one special quirk about everyone. It's a wild mixture under one company roof, and an analogy. His point of course is that one of the hard problems Unity needs to solve is catering to a wide variety of games using one single engine, something quite different from my former research environment where I used to have a very narrow and highly specified problem space in front of me and just wrote single-use code.

Today is my last day at Unity. I am very happy I got to work on this incredible engine. Having pushed code to millions of devices is simultaneously terrifying and exhilarating. In this post I will cover three things I learned during my time at Unity that - in hindsight - may seem obvious, but ultimately freed me from the bubble I came from. I had the privilege of working alongside most excellent people that helped me get there (in fact, many more than on this Twitter-only list): Kuba, Jesper, Kasper, Rasmus, David, Robert, Florent, Ionut, Paul, Arnaud, Veselin, Martin, Göksel, Arthur, Nicholas, Cyril, Adrien, Antoine, Sebastien, Francesco, Ben, Anis, Thomas, Laurent, Thomas, Julien, Christophe, Eric, Kemal, Laurent, Julien, Felipe, Evgenii, Peter, Natasha, Aras, Kleber, Rej and Baldur.

Generate Lighting

Every feature of a modern game engine is a cover on top of a bottomless pit full of problems to solve. For me, that cover was a single button, or so I thought. Initially it felt great to hide behind it: Here is that one feature where you can truly capsule off from everyone else. That button precomputes all static lighting using one of the global illumination backends the user can select. There are a bunch of settings to adjust for like ray counts etc., but all in all it seemed like a very contained world. However, complexity tends to leak.

I had dedicated a good amount of time to a conundrum: As the name suggests, global illumination is a global phenomenon, but for the sake of memory consumption many scenes are often tiled in some form or another, and then computed independently. The disconnect, i.e. the global lighting not being so global anymore, needs to be managed later somehow. A lot of user-complaints and tickets I went through had the same core issue: Two tiles of a scene were precomputed independently, but with - accidental or not - differing settings. Sometimes there was a different skybox involved, sometimes the GI backend was flipped or the textures stored directional lighting in one but not the other tile. For half a year, I wrote a lot of messages explaining that when putting those tiles back together, the contradicting settings will cause all sorts of issues, not just visually.

Even though everyone understood the message, the tickets kept coming. Eventually I sat in front of a project so convoluted that I too made the very same mistakes I tried to educate everyone about, and so I added warning labels to everything. Initially they were quite unpopular, but the flood of tickets came to a grinding halt.

In essence, I learned the hard way that no amount of documentation, videos or talks will iron out the complexities of your software. If there are no visual indicators attached to parameters that can be used wrong very easily, everyone will keep using them wrong until the end of time, not because of ignorance or because it is hard to use, but simply because the software is complex and mistakes are easy to make.

Physically based ork-skin

One of the things that used to baffle me before working in the games industry was the apparent excitement and ignorance artists and graphics programmers alike displayed when it came to physically based rendering. At SIGGRAPH and other conferences, courses dealing with rendering and lighting in games celebrated the fact they've adhered even slightly to the rules of reality and took away all their artists favorite sliders. Zero tolerance for shenanigans like lights with negative energy or materials that violate the conservation of energy! No longer would it be possible to abuse the lighting system, no longer would different game scenes have wildly different behavior and everything would be nice, coherent and predictable.

But did they not get the memo? The literature on physically based rendering was already ancient by graphics standards. Why the victory parade?

Unity's platform support is huge. Unlike tailored engines, games made with Unity often run on many combinations of graphics APIs, operating systems, CPU architectures and end-devices. Additionally, the type of graphics required for a certain look pile on top of the already complex situation: 2D, 3D, realistic, open-world, top-down, minimalistic, stylized, massive, impossible… Then there is the project itself: A game, a prototype, a demo, a movie (real-time or a glorified frame capturer), a UI or some interactive educational tool. All of these factors are just technicalities though. Yes you might end up on a device that is still unable to use linear intensities, yes you may not have RTX support, yes the platform could be unable to stream all your data and yes the game may not even be next-gen.

I remember a brief chat with one of the tech-artists about the perfect renderer, that in my opinion simply required to follow a strict set of physically-guided rules and was augmented by tools that helped artists deal with the assets they need, like the built-in photogrammetry tool.

Sounds good, I just need that sample of ork-skin and we'll get the cameras.

I sometimes tend to forget that most game settings are not situated in this universe. While tinkering with rendering architectures all day long where I focus to get one thing right - the transport of light - and dealing with all the technical issues - memory consumption, serialization, shader architecture, ray-throughput, threading etc. - it did not occur to me that the artist might actually want to break the rules of physics on purpose. In all cases the engine is both tool and stumbling block, either enabling a vision or preventing it. The engine programmer writing the underlying architecture is trapped between two extremes: On one side the desire to write code that behaves predictable and consistent across the wide variety of projects, styles and platforms, on the other to enable the freedom of the artists and their often conflicting and contradictory ideas of what they want see on screen.

At the end of the day, nothing is as important as the vision of the game though. Just like an artist can take a brush, an empty canvas and paint anything, be it an impression of the real world, a fantasy setting or an Escher-like mind-bender, so too should an engine deliver the flexibility to enable the game designer to achieve their goal, even if it is completely weird, unorthodox and against all reason.

And so I learned that in many cases the default implementation should just be easily replaceable. If the implementation does not work for the game, then it should be possible to use a custom one. Games are just different from other software in that regard, because they are pieces of art.

The engine is eternal

Invent yourself and then reinvent yourself is the slogan of the BMW demo showcasing the running version of the high-definition render pipeline using DXR ray tracing, and it perfectly captures what goes on under the hood of Unity. The demo was the end-result of a long development cycle that started with a tiny hackweek project using the brand-new DXR API and its semi-correct documentation to render ambient occlusion. Like all paradigm shifts, the new way of doing things did not really fit nicely into the existing structures, at least in the beginning. Where should the DXR renderer live? Should it still work if the project is setup to use Vulkan, now that we're using two APIs rather than one? Can the shaders run in parallel with the deferred renderer, and if so how do we synchronize them? How are the materials translated? Who is in control of the reflection probes? There were quite a few questions like these that required some back and forth, but in the end the renderer would fit in.

Adapting code to make things fit is sometimes not enough. After many years of service, the built-in render pipeline was simply too monolithic to be used across vastly different devices and could not provide the flexibility that many games needed, be it custom adaptations to the renderer or simply the option to rip out entire features a game is not using. Hence one night over some pizza at the office with Joe and a colleague, he casually pondered how Unity could look like if the render pipeline would be written in C#, as opposed to being built-in. One hackweek later and the Scriptable Render Pipeline was born. Though that was just a prototype. Now we suddenly had to decide who is in control of the flow. Should the renderer push its state down from C# to the rest of the base code in C++, or should it work the other way around? How are custom features - implemented in the a ScriptableRenderPipeline - reflected back, such as a whacky material with some new sliders that the global illumination backends are not aware of? Can the C++ code inject render calls, and if so when?

In 2018 we shipped the GPU Lightmapper, an enhancement of the Progressive Lightmapping backend that so far only ran on the CPU. If the development machine has a capable GPU in store, the GPU backend will render at significantly more rays per second. After a long process of abstracting away most jobs running inside the Lightmapper, we went with OpenCL as the compute solution, as it would run on all supported Editor platforms. And then, at WWDC2018, whilst Book of the Dead was shown on stage to demo the new eGPUs for MacBooks, Twitter began to notice that Apple had just deprecated OpenCL on macOS in favor of Metal.

When I compare projects I do for fun with the code I write for work, I can see a strong divergence between the two. In my hobby projects all code is following a very strict order, and if it does not then it is usually time to fix that. Throughout the code identical concepts are used. Helper functions abstract away repeating patterns and everything looks nice and tidy. The code seems to have a long life ahead, but only because it serves a single purpose.

In comparison, game engine code looks a bit like tectonic plates shifting at the speed of sound. Entire systems exist multiple times over, the control flow is not obvious, and features can get ripped out from the codebase rather abruptly. Code is rewritten much more frequently than in other software. To maintain a bleeding-edge status, developers are forced to constantly re-evaluate their code. Because of that, the balance when planning out features shifts quite dramatically to the near rather than the far future, and that plan is always one round away of loosing the API-, Paper- or Dependency-Russian-Roulette.

Now it might all sound like a lot of duct tape and a constant struggle to keep a patient alive while exchanging all guts at once and simultaneously doing brain surgery. One could get the impression that nothing is here to stay, and that eventually it is all but Unity in name.

But I learned that Unity earned its reputation as one of the most intuitive game engines out there not because it has been around forever unchanged, but because it is constantly adapted and modified to make current and new technology easily accessible to everyone. There may be a lot going on under the hood to keep that reputation, but this is the reason why a decade from now I'll still be able to recognize the motor I helped build, and why the engine is forever.

The Heretic

Veselin Efremov et al. and Tobias Alexander Franke Unity Demos

Abstract

The Heretic is a short film created by Unity's Demo team. After debuting its first part at GDC 2019, the completed short is now available to watch in its entirety.

The Unity Demo team's mission is to push the boundaries of what is visually possible with Unity. With The Heretic, the team used every aspect of Unity's High Definition Rendering Pipeline, created advanced effects with the VFX Graph, and undertook the challenge of creating a realistic digital human.

Supplemental Video

Links

• Project Page
• VFX Workflow
• Making of
• IMDB
• Video

Static Lighting with Light Probes

Tobias Alexander Franke and David Llewelyn Unity Blog

Abstract

In 2019.2, we removed the Lightmap Static Flag, replacing it with the Contribute Global Illumination Flag. We also introduced the ability to choose whether Global Illumination is received from Lightmap or Light Probes. These changes can have a huge impact on your baking performance, quality of Scene lighting, and more! Let’s explore this further.

Links

• Unity Blog Article
• Wayback Machine Archive

Useful things tend to become invisible over time. I have a router running OpenBSD that I had not touched in a while. One day I logged in and got curious.

$ uptime
12:26PM  up 1583 days,  2:33, 1 user, load averages: 0.13, 0.12, 0.08

It had been running continuously for 4 years without interruption, happily doing its singular task. Similarly, tools like SSH, SCP, rsync, the Sublime text editor, VLC or that one script I wrote years ago all vanish in the daily noise because apart from doing their job, they are primarily not annoying.

The same cannot be said for browsing the web anymore. I use a somewhat dated but otherwise fully functional MacBook 13" from 2013 with the latest MacOS. Most things are working just fine, but as soon as I open a browser and go to any bigger website, the poor piece of silicon is choked to death by a barrage of trackers and useless features packed together in a lasagna of Javascript libraries, each running in the background using websockets and Ajax requests to continue the torture.

Even with a beefy machine I notice laggy input in every bigger web application. In some cases browser vendors running big websites are actively abusing APIs to stop you from using their competitors. Apart from the dreaded interface changes that are imposed on visitors every now and then, content is often hidden in completely overloaded designs, buried under advertisements and self playing videos. The problem has become so endemic to everyday online life that several browsers now come with built-in "Reader Views" or services such as Pocket to trim down websites to their bare essence. Without an almost mandatory set of ad- and script blockers, redirect skippers, CDN caches, fingerprint scramblers and privacy containers (the list goes on), almost every modern website is a cocktail of ad-infested privacy nightmares. And then there is the fine threshold where running too many add-ons to curate the web is a bottleneck and security threat itself, so having multiple browsers on a device is also considered "normal" these days. Maciej Ceg&;owski summarized the state of modern web design in his hilarious talk The Website Obesity Crisis with the following image:

A great way to combat many of these problems is to use RSS. I recently noticed that over the last year I had relocated most of what I consume online to an RSS reader. I had been using it for now more than 12 years, mostly to follow blogs, but now on some days I only open the reader instead of my browser. I am not alone in this move: Wired ran an article about the resurgence of RSS because of the way the web had transformed. A reader on Hacker News soon after pointed out the irony that the very same article - suggesting RSS as a solution to this hot mess - came with its own clown-car of trackers. In a very twisted way, this article made clear why most of the web is terrible.

An elegant weapon for a more civilized age

For those unfamiliar, RSS is a standardized container format for content updates, for instance blog posts or news articles. Such updates usually have a title, an author, a publication date, some content (i.e. text and images) and perhaps a list of tags. RSS encapsulates each update in a bit of XML. An RSS reader - often referred to as a feed aggregator - which you point to an RSS stream will, in regular intervals, check the stream if any new updates appeared and download them for you to view later. In many ways, it is like having an inbox for website updates.

But it doesn't have to end at blog posts. Most of what we read and watch every day can be encapsulated with RSS: Facebook posts, Tweets, Instagram pictures, new videos uploaded Youtube/Twitch/Vimeo, Github commits etc. are all updates on websites we check repeatedly every day. That manual labor can be avoided by subscribing to their RSS streams.

Say you are interested in computer graphics. You may have a couple of people posting on Twitter that you follow. There is also the SIGGRAPH Facebook page you open every day to scan for interesting tidbits. You are also on Mastodon, because some people from the community are gathering there. Additionally, you read a ton of blogs, and you follow the Youtube channels of Khronos, GDC and some researchers, for which you get notified through your subscription on Youtube. Except this one person that hosts their videos on Vimeo, that one you need to check on your Vimeo account. At last there are the Github projects with the occasional releases and the starred repositories of people you follow. To sum up: The content category you are interested in - computer graphics - is splattered all over the place and every day you will need to go out and collect everything back together.

With an RSS reader, you can open a folder - let's call it Computer Graphics - and simply add all those sources into it. It is now the RSS readers task to fetch new items in regular intervals and notify you. You will never have to visit any of those pages again to check for updates. After all, if it is just a category you are interested in, who cares were the content comes from?

The image above shows a sample from my Music folder, where I follow Soundcloud accounts next to Youtube, Vimeo and a blog at the same time. All relevant data ends up in the same stream of notifications.

What do I gain?

Apart from reducing the menial task of visiting a bunch of websites every day, RSS has some nice side-effects.

No walled gardens: RSS readers allow you to fetch content from a multitude of sources and group it in any way you please. Decentralization comes naturally and is easily managed. Content is not bound to specific websites or providers.

No accounts: On Youtube for instance, subscribing to channels requires an account. For each other walled garden where you subscribe to something, you will need an additional account. You can get away without any accounts by aggregating all subscriptions via RSS.

Limited tracking: RSS encapsulates raw content of an update. A reader can show you a new item without you visiting any website. All the Javascript nonsense that comes with a page you would usually visit (like the omnipresent Google Analytics) is gone.

No interspersed ads: Twitter is among one of many websites that inject advertisement posts called "promoted tweets" into your stream. Technically, that is possible with RSS too. Without accounts and scripts following your every move, building a profile about you is limited though.

Unfiltered and unsorted by machine learning: Many services will employ some type of machine learning to show you something they deem interesting for you. Chronological order goes out the window, as these services push the "top" or "most liked post" to the beginning of your stream. RSS is not a service controlled by anyone, therefore the order is yours to control.

Faster browsing: Content is served as raw text that may be formatted with simple HTML. Instead of opening 10 websites that each harass your browser, one feed with limited HTML or text-only messages and pictures remains. All content is served in a common theme.

No social features: Without walled gardens and no centralized sources, there are also no common comment sections or share buttons.

Privacy: Given all of the points above, privacy of whom you follow and what you read is maintained.

Security: Generally, the less intelligent and powerful a system is, the less damage it can inflict. Technological blights like Cross Site Scripting attacks are simply not a thing when you read raw text.

What can go wrong?

Of course not everything is suited for RSS, such as legitimate web applications like shops or banking portals, and there are some down-sides to consider when relying heavily on RSS.

RSS may not be supported: A website you like and want get updates from may provide no or only inadequate RSS streams. For instance, a news page could give you an RSS stream for each category, or the entire site, or none at all. In these cases you might need to rely on third-party scrapers that will generate RSS content from the website. However, a nice side-effect of those services is that they act as proxies: Since you are not directly subscribing to the pages RSS, it won't even know you where ever there.

Non-public data inaccessible: Scrapers can turn any public website into an RSS stream, but anything locked behind an account can become a problem. Some websites implement private RSS streams using secret tokens (for instance the personal GitHub news feed), but in most cases you will be out of luck.

Only previews available: Many RSS streams do not provide the full content of the update, only a preview, forcing you to open the original website again. It is not the end of the world, but annoying. There are again third party services that will automatically expand to full content RSS streams from preview-only streams.

No social features: Without walled gardens and no centralized sources, there are also no common comment sections or share buttons.

No recommendations: Platforms such as Vimeo or Instagram will, given user tracking through accounts or other means, compile recommendations of what to watch or whom to follow. Decentralized and without accounts, there is no recommendation page anymore. Online RSS readers such as Feedly might recommend more feeds, but in general terms this feature does not exist anymore.

Synchronization between devices: To keep track which article on which feed was marked read or unread, RSS readers keep an index database. These index databases are not standardized and often cannot be synchronized easily between different readers. Online RSS readers such as Feedly solve this problem by synchronizing an online index with a proprietary protocol. An effort is underway to solves this issue.

Multimedia content: Some items can be subscribed to via RSS, but not shown as purely raw text-content, such as video or music subscriptions. In these cases your RSS reader will use a built-in or external browser to open a link. However, getting notified this way is still preferable.

On Readers

Armed with the knowledge of the pros and cons of using RSS, you can start by installing an RSS reader to subscribe to something.

Generally, you want to look out for a reader that supports import and export of your subscriptions. The commonly used format for this is called OPML. Simply put, OPML is a standardized list of all your subscriptions. A current backup of those subscriptions can be used to quickly change from one reader to another, and they are very easy to edit in case one wants to mass-change RSS feed subscriptions to an alternative. For instance, you can export all your Youtube subscriptions as OPML and easily migrate them to Invidious, an open-source Youtube proxy, using a simple search-and-replace with your favorite text editor:

https://www.youtube.com/feeds/videos.xml?channel_id=ID
-> 
https://invidio.us/feed/channel/ID

Keeping a backup of all your subscriptions is essential to being independent of any third-party services or readers!

There are two types of readers: Online readers running in a browser, which keep all your subscriptions behind an online account, or local readers, which may synchronize with an online reader account. The go-to online reader and winner of the Google Reader exodus is Feedly. Feedly supports OPML import-export and can synchronize with a wide range of mobile and desktop readers. If you have no reader application at hand, you can always use the web version. Feedly and other online RSS readers often limit the amount of RSS feeds you can subscribe to for free. An alternative is to use an offline RSS reader or to host your own online RSS reader. FreshRSS is an open-source project that can be set up with minimal friction on a webserver with PHP.

There is a multitude of very nice RSS readers out there, but I'll mention only a few very basic ones here:

• Thunderbird, the well known Mozilla email client, supports local RSS subscriptions.
• Feedbro, a browser extension, comes with OPML support and additionally can render MathJax in blog posts. Since it is browser-based, it can be used in conjunction with uBlock to weed out any additional requests coming from embedded media.
• RSS Mobile, the most basic iOS RSS reader I could find. Free, no ads, runs without any web service.
• Readify, an open-source reader and equivalent of RSS Mobile for Android.

RSS streams of common services

Apart from blogs, one of the most interesting use cases of RSS is to escape social media bubbles and walled gardens. Because RSS prevents social media platforms from tracking you, most have dropped their RSS support entirely. However, this is where scrapers such as RSS-Bridge come to the rescue: These services parse website content and turn it into an RSS stream, shielding you away entirely from the site. In the list below, replace RSS-BRIDGE-INSTANCE with for example https://bridge.suumitsu.eu/. There are many more publicly hosted instances of RSS-Bridge, and of course you can run your own.

Here is a sample of RSS streams you can use to replace typical social media:

• Twitter: https://RSS-BRIDGE-INSTANCE/?action=display&bridge=Twitter&format=Atom&u=USERNAME
• Mastodon: https://MASTODON-INSTANCE/@USERNAME
• Instagram: http://instatom.freelancis.net/USERNAME
• Facebook: https://RSS-BRIDGE-INSTANCE/?action=display&bridge=Facebook&media_type=all&limit=-1&format=Atom&u=USERNAME
• Hacker News: http://feeds.feedburner.com/fullhackernews
• Reddit: http://inline-reddit.com/feed/?subreddit=SUBREDDIT
• Youtube: https://invidio.us/feed/channel/CHANNELID
• Vimeo: https://vimeo.com/USERNAME/videos/rss
• Twitch: https://twitchrss.appspot.com/vod/USERNAME
• Tumblr: https://USERNAME.tumblr.com/rss
• GitHub: https://github.com/USERNAME/PROJECT/releases.atom
• Soundcloud: https://rssbox.herokuapp.com/

I keep a more comprehensive list updated on this Github Gist. Note that for most of these pages there are often multiple alternative scraper projects. If you cannot find an RSS stream of a website or service you want to subscribe to, consider contributing to projects such as RSS-Bridge, RSSHub or RSSBox. Multiple instances of these projects are hosted for free by volunteers, and you may want to consider running your own as well.

Conclusion

RSS is a great way to keep track of content updates on any number of sources you visit daily. It naturally shields against unnecessary ad-tracking and separates actual content from virtual garbage, making the experience of reading and viewing more pleasant. RSS frees from the bonds of walled gardens and allows you to organize all of your sources by categories again instead of where they originate from: It is no longer your "Youtube subscriptions" or "people you follow on Instagram", but subscription bundles of "cat videos" or "food pictures", wherever they might come from. A well maintained subscription list will reduce the time you spend mindlessly surfing the web in search for new stuff.

If you are as upset as I am about the current state of the WWW, I encourage you to consider this alternative way of consuming the web, or to revive your RSS reader if you have abandoned it a couple of years back.

Here is my recommendation:

1. Find a good reader you feel comfortable with or sign up to Feedly. I'm on MacOS and prefer Reeder.
2. Add a couple of blogs.
3. Subscribe to some of your favorite Youtube channels (here is one I like called 3Blue1Brown).
4. Use it for a couple of days or weeks to get a feel for it.
5. Use this list to populate your reader with more content.
6. If your reader supports a usage overview (like Organize Sources on Feedly), identify and eliminate spammy sources.

And then, if you prefer it, slowly, but steadily, subscribe to to everything you watch, read or listen to online. You can start with my log right now.

References

1. Wikipedia, RSS
2. Tobias A. Franke, A list of RSS endpoints, readers and resources
3. Wikipedia, Comparison of readers
4. RSS-Bridge, RSS-Bridge Project

An Introduction to Lightmapping in Unity

Tobias Alexander Franke and Jakub Cupisz Unite Berlin 2018

Abstract

Get an intro to the basics of lightmapping and the Progressive Lightmapper, and learn how to address and solve the most common issues our users experience. We also dig into the future roadmap for lightmapping in Unity.

Supplemental Video

Links

• Slides
• Unite Berlin 2018
• Video
• Bibtex

Book of the Dead

Veselin Efremov et al. and Tobias Alexander Franke Unity Demos

Abstract

Unity's Demo Team, creators of Adam (2016) and The Blacksmith (2015), are excited to announce Book of the Dead, a first-person interactive story showcasing the capabilities of Unity 2018 for powering high-end visuals for game productions.

The trailer is representative of the actual gameplay experience.

Unity's Demo productions drive advanced use of the Unity engine through autonomous creative projects. The team consists of experienced game professionals led by Creative Director Veselin Efremov, who writes and directs the demos and is responsible for their final look.

Supplemental Video

Links

• Project Page
• Assets
• Environment
• WWDC2018
• Video

Most introductions and implementations of Precomputed Radiance Transfer will deal fairly well with the easiest use case: The double product integral for diffuse reflections. Beyond this scope however, few show how proper rotation of frequency-space encoded lighting works, and even fewer dive into the problem of view-dependency in PRT. Both of these are related, as they rely on the ability to transform one set of coefficients into another: Rotating a vector creates another vector, and view dependent reflection transforms incident light represented as coefficients into coefficients of reflected light. From linear algebra, we know how to deal with such a scenario already: To transform one vector into another we need a matrix, so in terms of a lighting scenario we need to talk about Matrix Radiance Transfer and the Triple Product Integral.

Function transforms

In the last post on function transforms I took a quick look at the convolution theorem, which one can roughly describe as the ability to shortcut an integration over the product of two functions, i.e., a convolution, with a dot product in the frequency domain. Typically, the convolution theorem is introduced with the Fourier basis, but other function bases can be used as well.

This holds true for any function basis that is orthonormal. To test whether a set of functions forms an orthonormal basis, one needs to do two things: Make sure that any two basis functions ${\mathrm{\Phi <!--\; \Phi \; -->}}_i(s)$ and ${\mathrm{\Phi <!--\; \Phi \; -->}}_j(s)$ integrate to 0 if $i\ne <!--\; \ne \; -->j$, and to 1 if $i=j$.

This works completely analogous to a vector basis.

The Triple Product Integral

So lets assume we have two functions $f(s)$ and $g(s)$ and we want to compute the function product of both as a new function $e(s)$.

Assume further that we have projected $f(s)$ and $g(s)$ into the frequency domain of the function basis ${\mathrm{\Phi <!--\; \Phi \; -->}}_i(s)$. This means that we have the coefficients for both at hand.

It may be possible to do the same operation in Equation $\text{3}$ - multiplying two functions together - in the frequency space of the basis $\mathrm{\Phi <!--\; \Phi \; -->}$ that we chose. How would we be able to determine the coefficients $e_i$ of $e(s)$? We can start the usual way by integrating it with the basis function.

But since we already have the coefficients for both $f(s)$ and $g(s)$, we can first replace $e(s)$ by Equation $\text{3}$ and then replace further with both Equation $\text{4}$ and $\text{5}$.

We can rearrange some things: All the terms that are not dependent on $s$ can be moved out of the integral.

Now that is an interesting looking integral! Three basis functions in one, and at first sight the whole process looks quite familiar too. It is almost identical to what we did to derive the convolution theorem, now we only need to get rid of this integral and we're done.

The term ${\int <!--\; \int \; -->}_S{\mathrm{\Phi <!--\; \Phi \; -->}}_i(s){\mathrm{\Phi <!--\; \Phi \; -->}}_j(s){\mathrm{\Phi <!--\; \Phi \; -->}}_k(s)ds$ is called the triple product integral for more or less obvious reasons, and $C_{ijk}$ is a tripling coefficient. Its smaller sibling, the double product integral, is reducable to a Kroenecker Delta if the basis functions ${\mathrm{\Phi <!--\; \Phi \; -->}}_i(s)$ form an orthonormal basis. But for $C_{ijk}$, things are not so easy. In fact, there is no general analytical solution for any arbitrary function basis.

Computing tripling coefficients

Luckily, for the two most common bases used for PRT, Haar-Wavelets and Spherical Harmonics, an analytical formula to compute their tripling coefficients exists.

Tripling coefficients of Spherical Harmonics can be expressed through a relationship with Clebsch-Gordan coefficients, which in turn can be expressed by Wigner 3j symbols. Below is an older implementation of mine, but as far as I'm aware the GNU Scientific Library contains one as well.

template<typename T>
inline T wigner_3j(int j1, int j2, int j3, 
                   int m1, int m2, int m3)
{
    assert(std::abs(m1) <= j1 && 
    	"wigner_3j: m1 is out of bounds");
    assert(std::abs(m2) <= j2 && 
    	"wigner_3j: m2 is out of bounds");
    assert(std::abs(m3) <= j3 && 
    	"wigner_3j: m3 is out of bounds");
    
    if (!triangle_inequality(j1, j2, j3)) return 0;
    if (m1+m2+m3 != 0) return 0;
    if (std::abs(m1) > j1) return 0;
    if (std::abs(m2) > j2) return 0;
    if (std::abs(m3) > j3) return 0;
    
    int t1 = j2 - m1 - j3;
    int t2 = j1 + m2 - j3;
    int t3 = j1 + j2 - j3;
    int t4 = j1 - m1;
    int t5 = j2 + m2;
    
    int tmin = std::max(0, std::max(t1, t2));
    int tmax = std::min(t3, std::min(t4, t5));
    
    if (tmin > tmax)
        return 0;
    
    T wigner = 0;
    
    auto f = factorial<T>;
    
    for (int t = tmin; t <= tmax; ++t)
        wigner = wigner + static_cast<T>(std::pow(-1.0, t))/
            (f<T>(t)    * f<T>(t-t1) * f<T>(t-t2) *
             f<T>(t3-t) * f<T>(t4-t) * f<T>(t5-t));
    
    wigner = wigner * static_cast<T>(std::pow(-1.0, j1-j2-m3)) * 
    	static_cast<T>(std::sqrt(
    		f<T>(j1+j2-j3) * f<T>(j1-j2+j3) * 
    		f<T>(-j1+j2+j3) / f<T>(j1+j2+j3+1) * 
    		f<T>(j1+m1) * f<T>(j1-m1) * 
    		f<T>(j2+m2) * f<T>(j2-m2) * 
    		f<T>(j3+m3) * f<T>(j3-m3)
    	));
    
    return wigner;
}

With the tripling coefficients at hand, we can create a function product of arbitrary functions encoded in the same frequency domain. A major downside of regular PRT is that the BRDF, visibility and Lambert factor are all represented simply as one transfer function, which therefore gets encoded as one transfer vector. The Lambert factor however is a very low-frequency signal, whilst visibility and BRDF may contain high-frequency peaks. Packing them all together means choosing either to produce a lot of waste coefficients or compromising on the quality of the representation. Ng et al. get around this by decoupling visibility and the BRDF in the transfer function again with a triple product composition.

Another use case presented in Precomputed Shadow Fields for Dynamic Scenes wraps an object in a shell which, at several sample points, has coefficients encoding a visibility function from the point on that shell. If the shell collides with another PRT-object, coefficients of both objects visibility can be combined to compute dynamic shadowing between the two, getting rid of some of the rigidness requirements that PRT imposes on the scene. This paper is naturally enhanced in Precomputed Radiance Transfer Field for Rendering Interreflections in Dynamic Scenes, where the authors not just encode visibility, but indirect diffuse transfer between objects as well.

Transforming coefficients

Returning to Equation $\text{14}$ we can easily construct $e_i$ in another way.

Instead of using the tripling coefficient tensor $C_{ijk}$ and $g_k$, we can also use a matrix $T_{ij}$ (one way to construct it would be to multiply $C_{ijk}$ and $g_k$). This matrix will transform coefficients $f_j$ into coefficients $e_i$, which represent the function $e(s)$ in the basis $\mathrm{\Phi <!--\; \Phi \; -->}$ that we chose.

The concept of a matrix to transform a coefficient vector is sometimes mentioned in PRT tutorials for signal rotation. Rather than having an environment map $m(s)$, rotate it with $\mathbf{R}(m)(s)$ and then resample and recompute the Spherical Harmonic coefficients for it, we can likewise rotate the coefficients $m_i$ with a special frequency-space rotation matrix ${\mathbf{R}}^{\star <!--\; \star \; -->}$. In essence, we would produce a new set of coefficients that simply match those of the rotated environment map in image space.

Here is an old implementation of mine of one such matrix to do the job. The input is a regular 3x3 rotation matrix, but the result is a matrix that needs to be as big as the coefficient vector you want to rotate. In case of 4-band Spherical Harmonic vector for instance, the result will be a 16x16 Spherical Harmonic rotation matrix.

Matrix Radiance Transfer

In the diffuse case of PRT, we only need to compute a single number/color, because diffuse surfaces reflect the same intensity in all directions. The double product integral fits the job, because the convolution acts like a blur-filter that adds up all the light hitting the surface and reflects a single value. But for non-diffuse surfaces like metals this outgoing radiance varies in different directions, so we need more than just one number; we need a function that represents all light reflected into all different directions. We need a coefficient vector of the reflected light that we generate from the incident light coefficient vector.

For diffuse PRT, we turn a transfer function of just the incidenct light direction into the transfer coefficient vector. Now however we must turn a transfer function of an incident and outgoing direction into a transfer matrix. But how to construct the matrix for a function that has two variables? We can do so by imagining that we have a special set of transfer coefficients for each outgoing direction ${\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{o}}$, rather than just a single set for all of them. Where before we would turn a transfer function into a set of coefficients, we now basically convert it into a set of functions $T_m({\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{o}})$ which return the $m$-th coefficient of the reflected light for an outgoing direction ${\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{o}}$. To do this, we first integrate $T$ along all incident directions ${\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{i}}$.

We then integrate each resulting function $T_m({\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{o}})$, this time over all outgoing directions ${\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{o}}$.

In the previous article, the double integral product was used to compute the outgoing reflected light from the incident light coefficients $l_n$ and the surface transfer coefficients $t_n$. We transferred incident light into single value (or, if you want to think about it this way, into function which equals a constant value), which is perfect for diffuse reflection. Now with Matrix Radiance Transfer a matrix $T_{mn}$ instead represents the transfer of incident to exit light.

Where we used to have a dot product of two vectors, we now produce a reflection coefficient vector from incident light, and then reconstruct the light reflected into a specific direction ${\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{o}}$ by multiplying it with the basis $\mathrm{\Phi <!--\; \Phi \; -->}$.

Conclusion

Like regular vectors in 3D, coefficient vectors can likewise be transformed with matrices. This is useful when a function - encoded as such a vector - should be transformed into a different function, such as unoccluded to occluded light, normal to rotated environment, incident to exit radiance or when simply decoupling one transfer function into several parts and then multiplying them back together again.

References

1. Wikipedia, Clebsch-Gordon Coefficients as Wigner-3j expression
2. Free Software Foundation, GNU Scientific Library
3. Ng et al., Triple Product Wavelet Integrals for All-Frequency Relighting
4. Zhou et al., Precomputed Shadow Fields for Dynamic Scenes
5. Pan et al., Precomputed Radiance Transfer Field for Rendering Interreflections in Dynamic Scenes
6. Tobias Alexander Franke, SH rotation matrix implementation
7. Lehtinen and Kautz, Matrix Radiance Transfer

The first group likes to revise or invent new terms. For instance, using a different word to hide inefficiencies in a certain method. So instead of having augmenting objects, Microsoft prefers to call them Holograms. Why? Because in a see-through display like the HoloLens, you still can’t render proper black (that is, block light from passing through the lens entirely). Better find a new word for those translucent things on the screen. Additionally there are those who come up with entirely meaningless marketing terms like Cross-Reality (XR) or, for maximum chaos, Merged Reality.

The second is a consistently growing group of people who like to claim one of the following things:

• Mixed Reality is Augmented Reality and vice versa
• Mixed Reality = Virtual Reality + Augmented Reality
• Mixed Reality is a superset of Virtual Reality
• Mixed Reality = Some new I/O device
• Mixed Reality and Augmented Reality are unrelated
• Augmented Reality is the opposite of Virtual Reality

This group is not just simply wrong, but also responsible for the spread of mass-confusion around these terms which have been clearly defined for a long time and which have been used in many research papers.

Mixed Reality

The term Mixed Reality was introduced by Paul Milgram et al. in a 1994 publication called Augmented Reality: A class of displays on the reality-virtuality continuum. It presented the idea that between virtual and real there is a spectrum of different mixtures of both.

For instance, there could be a real news-anchor inside a virtual news-room (a thing called Augmented Virtuality). Or a virtual object in a real setting, like Gollum in Lord of the Rings (a.k.a. Augmented Reality). Things don’t necessarily have to look real. The spectrum could be 2D, with one axis defining if something is added or even removed (the latter being called Diminished Reality). The gist of it though is that you can arbitrarily lerp between real and virtual with any degree to define realism.

The paper introduces the term Mixed Reality as follows:

Within this framework it is straightforward to define a generic Mixed Reality (MR) environment as one in which real world and virtual world objects are presented together within a single display, that is, anywhere between the extrema of the RV continuum.

This excludes two realities: For lack of a better term “Real Reality”, and the polar opposite, Virtual Reality. You don’t need a definition for the obvious though, because Mixed Reality implies that you are mixing realities in the first place.

There are those who think they might come up with a clever argument that the observer/controller/player of a Virtual Reality simulation is somehow the mixed-in real element and therefore any interaction with Virtual Reality constitutes a Mixed Reality. This is of course silly, since that would include people using a computer and render the term meaningless. It seems that the reasoning behind this line of thinking is that there is a fancy new input device such as the Kinect or the Leap Motion which reads “real gestures”, whereas your old-school mouse or keyboard is just not magical enough.

Conclusion

• Virtual Reality is NOT Mixed Reality or included by it
• Virtual Reality is NOT Augmented Reality or included by it
• Augmented Reality IS a sub-type of Mixed Reality
• Augmented Reality is NOT the opposite of Virtual Reality
• Mixed Reality is MORE than Augmented Reality
• Mixed Reality is NOT a new I/O device
• AR + VR IS meaningless

Or to make it really simple: If you are not mixing two or more realities, you’re not doing Mixed Reality.

References

1. Michael Abrash, Why You Won’t See Hard AR Anytime Soon
2. Gamasutra, Magic Leap’s ‘chief game wizard’ has big ideas for the mixed reality future
3. The Foundry, VR? AR? MR? Sorry, I’m confused.
4. Twitter, #MixedReality
5. Intel, Merged Reality
6. Milgram et al., Augmented Reality: A class of displays on the reality-virtuality continuum

In Linear Algebra, we're used to build a vector $\mathbf{v}$ out of other vectors ${\mathbf{v}}_{\mathbf{1}},{\mathbf{v}}_{\mathbf{2}}$.

Each vector is, at the very least, implicitly constructed out of its basis vectors. If there is no specific basis mentioned anywhere, we assume it to be a basis of unit vectors. For instance, for a two-dimensional space these are ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{1}}=(1,0)$ and ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{2}}=(0,1)$, and that a vector $\mathbf{v}=(3,2)$ is just the sum of scaled basis vectors.

The numbers $3$ and $2$ are the coefficients of the basis vectors ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{i}}$. We can call them $c_i$.

The same is true for functions. We can build a function $v(x)$ out of other functions $v_1(x)$ and $v_2(x)$.

We can likewise represent one function as the sum of a set of scaled basis functions, just like we can represent one vector as a sum of scaled basis vectors.

The only question is, how do we know the scaling coefficients (i.e., $c_1$ and $c_2$) to build the desired function with a set of basis functions?

Function Transforms

When transforming a vector $\mathbf{v}$ from one basis to another, what we do is to multiply the vector $\mathbf{v}$ by each basis vector ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{i}}$ to get the new coefficients. The multiplication operation that we do is the dot product, or more generally the inner product $⟨<!--\; ⟨\; -->,⟩<!--\; ⟩\; -->$, a kind of matrix multiplication to project $\mathbf{v}$ onto each basis vector ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{i}}$.

This can also be written as a product of the individual vector components $v_k$ and ${\mathrm{\Phi <!--\; \Phi \; -->}}_{ik}$.

To reconstruct vector $\mathbf{v}$ in the new basis, we simply multiply the basis by the coefficients.

Notice how Equation $\text{8}$ is just the general form of Equation $\text{3}$. Transforming a function $v(s)$ into a set of coefficients $c_i$ for a function basis ${\mathrm{\Phi <!--\; \Phi \; -->}}_i(s)$ is almost the same process as in Equation $\text{7}$. Here we have to integrate it over the function domain $S$ with each basis function ${\mathrm{\Phi <!--\; \Phi \; -->}}_i$ individually.

Note that the only difference between Equation $\text{7}$ and $\text{9}$ is that in one case we sum up a discrete representation (i.e., the vector components) and in the other we have to integrate it instead. This is why we write down an inner product rather than a dot product, because it is independent of the fact that the basis is one of functions, or vectors or anything else.

Reconstructing the original function works just as in Equation $\text{8}$.

Function Basis Properties

With vectors, we can use the inner product (that is, a dot product) to determine some properties about their relationship. For instance, we can tell from the projection whether or not vectors ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{i}}$ and ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{j}}$ are orthonormal to one another.

If the coefficient $c$ is $0$, then both vectors ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{i}}$ and ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{j}}$ are orthogonal to each other. Additionally if ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{i}}$ and ${\mathbf{\Phi <!--\; \Phi \; -->}}_{\mathbf{j}}$ happen to be identical and $c$ turns out to be $1$ and not an arbitrary number $n$, then we know that both vectors also form an orthonormal basis. ${\mathrm{\delta <!--\; \delta \; -->}}_{ij}$ is called Kronecker Delta and is usually just a shorthand of such a basis behavior. With functions, the same concept is true, and the formula is identical. Instead of a dot product, the inner product for two functions ${\mathrm{\Phi <!--\; \Phi \; -->}}_i(s)$ and ${\mathrm{\Phi <!--\; \Phi \; -->}}_j(s)$ represents an integration. But again, if $c$ turns out to be $0$, we know both functions are orthogonal to each other, and if every other result is $1$, then the function basis is orthonormal.

We now have all the tools necessary to define properties of a function space: Equation $\text{13}$ can be used to define orthogonality and orthonormality of two functions, and $\text{11}$ can be used to define linear dependency (one function ${\mathrm{\Phi <!--\; \Phi \; -->}}_i$ is a sum of scalar multiples of other functions ${\mathrm{\Phi <!--\; \Phi \; -->}}_j$ of the same basis) and therefore also the rank and determinant of a function basis.

The Convolution Theorem

Now that we know how to transform a function into coefficients of a function basis, and how to test whether or not a function basis has certain properties like orthonormality, let's perform a magic trick. Imagine we have an orthonormal basis ${\mathrm{\Phi <!--\; \Phi \; -->}}_i$, i.e. two functions of that basis will always integrate to $1$ if they are identical, or $0$ if they are not.

We can represent any function $f(s)$ by a bunch of coefficients in that basis. Imagine now that we have two functions: $f(s)$ and $g(s)$.

Let's multiply them together and integrate the result. We can replace the functions with the sum in Equation $\text{14}$ and $\text{15}$.

In Equation $\text{16}$ we can reorder some things because everything is linear: We can move the two sums $\underset{i}{\sum <!--\; \sum \; -->}$ and $\underset{j}{\sum <!--\; \sum \; -->}$ out of the integral, together with their coefficients, because they do not depend on $s$.

However, we just noticed in the previous section that a function basis can be orthonormal. An orthonormal function basis is very nice to have, because the integrated product of the function basis vectors (i.e., the inner product of two functions) will be either $1$ or $0$, so the integral in Equation $\text{18}$ can be replaced with the Kronecker Delta from Equation $\text{13}$!

But if we have a Kronecker Delta, that just means that every time $i$ and $j$ are not equal, the product is just $0$. So we can simplify even further and remove one variable.

What is this? Equation $\text{25}$ looks like a dot product. So in essence that means we can shortcut an integration of a product of two functions $f(s)$ and $g(s)$ by a dot product of their coefficients $f_i$ and $g_i$ if the function basis is orthonormal!

Why is this important?

Filtering

Filter operations always take on a formula just like Equation $\text{16}$. For instance, a Gauss Filter is a Gauss function that is, in a small window, multiplied with another function. One can do that in a pixel-based domain, where each pixel is multiplied with a value from the Gauss function at the same position, but it is really just two functions multiplied together.

However, if one part of the function is already available in a harmonic basis function, for instance as a JPEG image, then we can apply the filter in the function basis directly and do an operation which would normally be very expensive with a cheap dot product! If the number of coefficients is smaller than the operations necessary on the original domain $S$ of the function (that is, the number of pixels we need to sum up), we can save a tremendous amount of computation time.

Precomputed Radiance Transfer

A main observation of Precomputed Radiance Transfer is the following: One can simplify the rendering equation to something we have seen above by throwing out self-emittance, and unify everything in the integral that isn't incoming light into one homogeneous diffuse transfer function $T({\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{i}})$.

The transfer function is simply everything - materials, visibility, Lambert factor etc. - packed into a single function. A simple visualization is in the above figure, where the transfer function of a point shows a visibility function. Imagine now that next to a transfer function, we also have a function representing our environment light. The crucial observation is that Equation $\text{27}$ looks just like Equation $\text{16}$, and this means that if we can get both as coefficients of some basis function, we can compute the integral in Equation $\text{27}$ with just one simple dot product of their coefficients!

Why is this practical? Getting the coefficients is really expensive, so before we can do this we would need to compute all transfer functions (probably with a raytracer) which is already very costly, and then do the Monte Carlo estimation to get their coefficients. This is way too expensive to do in real-time. But, if the car in the figure never moves, then the transfer function never changes! So if we compute all transfer in an offline step and save it (one might almost be tempted to call this pre-computing the transfer), then we only need to get the coefficients for the light at runtime, and that should be fairly easy to do. In fact, if we only have a bunch of different lighting configurations, we can precompute the coefficient vectors for all of them, and at runtime when switching between skyboxes fetch the respective coefficients.

References

1. Sloan et al, Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low- Frequency Lighting Environments. Peter-Pike
2. Robin Green, Spherical Harmonic Lighting: The Gritty Details

For Augmented Reality rendering, this go-to publication is Paul Debevec's Rendering Synthetic Objects into Real Scenes.

Fusing two realities

Imagine the following situation: You have a green-ish surface, and you want to project a virtual object on top of it, like this Stanford Buddha.

Just as other real objects on this surface would do, this Stanford Buddha should also drop a shadow onto it. To do this, you figure out the position of a real light source (let's pretend the room in which this shot was taken has only one light), and put a virtual copy of it into the virtual scene at the same relative location.

What now? To render a proper shadow, we also need a surface to drop the shadow onto. We can use a green-ish virtual surface as a placeholder for the green-ish looking real one to receive the shadow. Let's give it a try.

But how do we now fuse this image with the real background? Do we just paste it over the image?

That doesn't look quite right. So far we only had a vague interpretation of what the underlying real surface looks like: Green-ish. The description is not very accurate, and the result shows.

Maybe we could reconstruct the real surface more accurately and then get a more correct looking image. For instance, we could texture the so far green-ish placeholder with an image of the real surface. Of course from that texture we need to carefully remove all the interfering lights first (a process sometimes called delighting) so we're really left with just the albedo. Assume we did this correctly and now have this texture ready. Just put it on top of the surface and try again.

It looks better, but now we run into other problems:

1. The scaling of the virtual plane and its texture is slightly off, which makes it look a bit blurry.
2. The tracker didn't reconstruct the real world position of this patch down to the nanometer, so we end up with some very annoying seams around the border.
3. The renderer uses a point light to model the real world light source. You can see the falloff toward the back of the plane, where it is darker than it should be. The real light source in that room is a large neon tube and incident light spreads much more equally across the surface.

Also, it looks like there is some type of coloring issue. Maybe it's a gamma problem? Is the reconstructed albedo texture really correct? Maybe we need a path-tracer? Is there a red bounce missing from the plane in the back? Maybe this, maybe that, maybe maybe maybe…

Early Augmented Reality applications in the 90s dealt with this fusion-problem in various ways:

• Just leave it as it is.
• If you need shadows, just make the pixels darker. Even if there are multiple non-white lights in the room no one will notice.
• Reconstruct the entire object rather than just a patch of it. At least then there won't be any visible seams or other texture-related differences.
• Blend the partly reconstructed surface with the real background image.
• Why invest so much hard work rendering correctly when you can make the problem space much easier? Just use simpler real world objects! For instance a diffuse, homogeneous green surface with no texture at all.

Differential Rendering

While all of these tactics can work, they are unsatisfying at best and often require manual tweaking to make just that one scene look good.

The main weakness in the above scenarios is this: They rely on the placeholder to represent reality as close as possible. Could we perhaps get rid of the placeholder entirely and just be left with the actual change that the Buddha introduces to the real scene (i.e., the shadow)?

This is the question that Differential Rendering deals with. The idea is simple: Extract the difference introduced by the object and leave out the rest. How? Let's look at it again. We have a patch of the green floor that is supposed to be the a reconstruction of the real world. If we tag all the pixels of a reconstructed surface in a mask, we can isolate the green placeholder.

We can render the same patch again, but without the object, and repeat the same process of masking out the result. Subtracting both images from one another will leave us with only the difference due to the objects presence in one of the scenes.

The shadowed area will produce negative energy, as there is less energy present on the augmented patch than before, while the rest of the image will just be a zero-sum. If we add this difference back to our very first image, the shadow ends up in the real world and looks like it belongs there.

Conclusion

The main strength of Differential Rendering is its simplicity, as it boils down to a post-processing effect (you can easily repeat the above steps with Gimp, Krita or Photoshop) to extract the influence a virtual object has onto its surrounding, such as blocking real light, bouncing off reflections or transmitting light. This difference is then simply added onto a real background image together with the object. The viewer is left with a sense that the object is actually in the scene rather than being glued on top of it.

As a side note, the extracted difference contains positive contributions (such as bounce light) as well as negative ones (the shadows subtract light from the background). The latter is a quantity called antiradiance and has been discussed in a few publications, most notable Implicit Visibility and Antiradiance for Interactive Global Illumination. I will come back to this some other time.

More importantly, if the renderer is able to compute global illumination, the entire extracted difference should integrate to zero: The object is merely "re-routing" light to a different location, and therefore all energy is conserved.

References

1. Paul Debevec, Rendering Synthetic Objects into Real Scenes
2. Dachsbacher et al, Implicit Visibility and Antiradiance for Interactive Global Illumination

The Delta Radiance Field

Tobias Alexander Franke Technische Universität Darmstadt

Abstract

The wide availability of mobile devices capable of computing high fidelity graphics in real-time has sparked a renewed interest in the development and research of Augmented Reality applications. Within the large spectrum of mixed real and virtual elements one specific area is dedicated to produce realistic augmentations with the aim of presenting virtual copies of real existing objects or soon to be produced products. Surprisingly though, the current state of this area leaves much to be desired: Augmenting objects in current systems are often presented without any reconstructed lighting whatsoever and therefore transfer an impression of being glued over a camera image rather than augmenting reality. In light of the advances in the movie industry, which has handled cases of mixed realities from one extreme end to another, it is a legitimate question to ask why such advances did not fully reflect onto Augmented Reality simulations as well.

Generally understood to be real-time applications which reconstruct the spatial relation of real world elements and virtual objects, Augmented Reality has to deal with several uncertainties. Among them, unknown illumination and real scene conditions are the most important. Any kind of reconstruction of real world properties in an ad-hoc manner must likewise be incorporated into an algorithm responsible for shading virtual objects and transferring virtual light to real surfaces in an ad-hoc fashion. The immersiveness of an Augmented Reality simulation is, next to its realism and accuracy, primarily dependent on its responsiveness. Any computation affecting the final image must be computed in real-time. This condition rules out many of the methods used for movie production.

The remaining real-time options face three problems: The shading of virtual surfaces under real natural illumination, the relighting of real surfaces according to the change in illumination due to the introduction of a new object into a scene, and the believable global interaction of real and virtual light. This dissertation presents contributions to answer the problems at hand.

Current state-of-the-art methods build on Differential Rendering techniques to fuse global illumination algorithms into AR environments. This simple approach has a computationally costly downside, which limits the options for believable light transfer even further. This dissertation explores new shading and relighting algorithms built on a mathematical foundation replacing Differential Rendering. The result not only presents a more efficient competitor to the current state-of-the-art in global illumination relighting, but also advances the field with the ability to simulate effects which have not been demonstrated by contemporary publications until now.

Links

• PHD thesis
• German National Library
• EG DGLIB
• IGD
• Bibtex

The State of the Art in Real-time Relighting for Augmented Reality

Tobias Alexander Franke ETH Zurich Visual Computing Talk

Abstract

With the high availability of powerful mobile devices a renewed interest in augmented reality and the plausible fusion of virtual objects into real environments emerged. This in turn has recently lead to new research seeking to adapt real-time global illumination algorithms for AR relighting.

Because augmenting objects are directly exposed to the physical surrounding, discrepancies to reality, errors in the shading model and inconsistencies in the mutual interaction of real and virtual light are easily spotted by human observers. The problem of shading virtual surfaces according to real world illumination has been attacked before with image-based lighting, but the proper fusion of both virtual and real spaces requires physically plausible illumination exchange on both ends. True immersiveness additionally requires that any solution operates on a tight per-frame budget.

This talk will introduce the relighting problem, cover the current state of the art in global illumination relighting in AR and will conclude with an analysis of the different methods currently available - from Instant Radiosity to Irradiance Caching to my own research, which seeks to solve the problem of simulating glossy reflections off real surfaces with varying degrees of roughness - and their practical application in modern AR renderers.

Links

• Slides
• ETHZ
• Bibtex

Delta Voxel Cone Tracing

Tobias Alexander Franke 2014 IEEE International Symposium on Mixed and Augmented Reality

Abstract

Mixed reality applications which must provide visual coherence between synthetic and real objects need relighting solutions for both: synthetic objects have to match lighting conditions of their real counterparts, while real surfaces need to account for the change in illumination introduced by the presence of an additional synthetic object. In this paper we present a novel relighting solution called Delta Voxel Cone Tracing to compute both direct shadows and first bounce mutual indirect illumination. We introduce a voxelized, pre-filtered representation of the combined real and synthetic surfaces together with the extracted illumination difference due to the augmentation. In a final gathering step this representation is cone-traced and superimposed onto both types of surfaces, adding additional light from indirect bounces and synthetic shadows from antiradiance present in the volume. The algorithm computes results at interactive rates, is temporally coherent and to our knowledge provides the first real-time rasterizer solution for mutual diffuse, glossy and perfect specular indirect reflections between synthetic and real surfaces in mixed reality.

Preview

Supplemental Video

Links

• Paper
• Slides
• Source
• IEEE
• ISMAR 2014
• Video
• Bibtex

Screen space cone tracing for glossy reflections

Lukas Hermanns and Tobias Alexander Franke ACM SIGGRAPH 2014 Posters

Abstract

A typical modern engine has a postprocessing pipeline which can be used to augment the final image from a previous render process with several effects. These usually include depth-of-field, crepuscular rays, tonemapping or morphological antialiasing. Such effects can be easily added to any existing renderer, since they usually rely only on information readily available in screen space. Recently, global illumination algorithms have been mapped to postprocessing effects, such as the wide selection of Screen Space Ambient Occlusion methods. An insight in [Ritschel et al. 2009] is that screen space algorithms can sample more information than just occlusion: in addition to visibility Screen Space Direct Occlusion samples neighboring pixels to gather indirect bounces. Soler et al. [Soler et al. 2010] use mipmapped buffers to sample diffuse far- field indirect illumination and importance sample specular cones for glossy reflections, but do not consider to use mipmaps to access prefiltered bounces for specular cones. We present Screen Space Cone Tracing (SSCT), a method to simulate glossy and specular reflections. Instead of regular screen-space ray tracing, we adopt the concept of cone-tracing on hierarchical, prefiltered buffers to reduce integration costs.

Preview

Links

• Abstract
• Poster
• ACM
• S2014
• Bibtex

Interactive Relighting of Arbitrary Rough Surfaces

Tobias Alexander Franke ACM SIGGRAPH 2014 Posters

Abstract

When presenting synthetic objects in a real environment - for instance for pre-visualization in advertisements - special attention needs to be directed at the mutual interaction of light reflecting off synthetic and real surfaces to form a coherent appearance. If the user is to be convinced that the synthetic object is part of the real scene, a relighting method has to handle shadowing and reflecting illumination between both synthetic and real surfaces. Even though a range of relighting methods are available for static scenes such as photographs, this aspect has been traditionally ignored in real-time augmented reality (AR) systems. A method often employed to merge synthetic light and shadows cast from synthetic objects with a real background is Differential Rendering, leaving out indirect illumination. Attempts have been made to resolve this issue with Differential Instant Radiosity [Lensing and Broll 2012], however requiring many VPLs to suppress flickering. Delta Light Propagation Volumes [Franke 2013] cluster many VPLs in a small volume, but suffer from bleeding artifacts. A GPU raytracer in [Kan and Kaufmann 2013] supports diffuse bounces with Differential Irradiance Caching, albeit at much higher cost than rasterizer based counterparts. We present a novel relighting solution called Delta Voxel Cone Tracing (DVCT) to enable mutual diffuse, glossy and specular indirect bounces between real and synthetic geometries. The method is temporally coherent and to our knowledge the first real-time solution to support arbitrary glossy reflections in AR.

Preview

Supplemental Video

Links

• Abstract
• Poster
• ACM
• S2014
• Video
• Bibtex

It is clear however that the cause for this confusion is that video encoding is firmly situated at the borderline to black magic. There are certainly tools like iMovie which tremendously help to overcome most troubles in standard editing and encoding, but somewhere in between incompatible codecs and file containers, video bitrates, quantization and scaling issues, frame re-timing, color compression, and total boredom because you just wanted a video everyone will give up sooner or later.

In this post I'm going to cover the topic of how to create a web-browser compatible, pixel-matching HD or Full-HD, H.264 compressed MP4 of a reasonable size from a recording of a graphics application. But first, we need some ground rules.

Getting it right

The first commandment, from which all others are derived, is that you shall absolutely and unequivocally avoid re-encoding your video multiple times. With each re-encoding, the quality of your video will degenerate into the image above. This is especially true if you know there will be yet another re-encoding step ahead where the settings are beyond your reach, for instance when uploading to Youtube: in this case a mostly unprocessed video is your best option.

These days the world has come down to some fairly standard aspect ratios: 16:9 for movies, 16:10 for displays. If you are recording 4:3 videos you're doing it wrong! Because we're creating a movie for the web, you may end up uploading it to various streaming platforms such as Youtube and Vimeo, which only deal with 16:9 ratios, so let's just stick with that.

On the resolution front of these ratios the choices have been reduced to 720p (1280x720 pixels) and 1080p (1920x1080 pixels). We'll most likely see new standards emerge for the upcoming 4k madness. In order to get a nice video file at the end, make sure that whatever you intend to record comes in either one of those two resolutions.

Not going for a standard aspect ratio and a standard resolution will force editing tools to rescale your video, causing sampling issues which will lead to blurry image quality. Sometimes you can't avoid having a resolution that isn't 720p or 1080p (capturing webcam videos for instance), but if you control the output just stick with the default. Videos which do not conform to either aspect ratio or resolution should be cropped and scaled manually with an external tool so that you control the quality and output instead of getting lucky with whatever your editing tool does to those videos.

Finally, all your videos should run with the same frame rate! If they don't you either have to re-time everything to the lowest common denominator (which doesn't require anything special but may not be great), or magically re-construct the missing frames with some optical-flow tool like Apple Cinema Tools or Twixtor (a lot of manual work ahead).

Dumping data

A popular way to obtain videos of a D3D or GL application are hooks like Fraps, NVIDIA ShadowPlay or DXTory. On Mac OS X, you can use Quicktime (File -> New Screen Recording) to record portions of the screen. If you haven't done so already, check out a demo of one of them and try to record from a running game. An important decision has to be made upfront about resource division: record uncompressed videos for maximum quality and speed at very high data rates which can easily fill even the largest of hard disks, or sacrifice potentially vital CPU time to compress live during recording. There is no easy answer to this and it depends on your preferences and hardware settings. Personally, I prefer to record uncompressed and later downsize the videos into smaller, compressed segments for archiving. This certainly is no option to record World of Warcraft Boss fights of 10 minutes and more though.

I haven't played around with Fraps too much and recently found a way to completely avoid it by piping raw RGBA frames from my D3D app directly into FFmpeg. You may use the following class to do the same.

#include <iomanip>
#include <ctime>
#include <sstream>

#include <D3D11.h>

#define tstringstream std::wstringstream
#define tcerr std::wcerr
#define tclog std::wclog

template<typename T>
void safe_release(T& obj)
{
    if (obj)
    {
        obj->Release();
        obj = nullptr;
    }
}

class video_recorder
{
protected:
    UINT                    width_,
                            height_,
                            fps_;
    ID3D11Texture2D*        ffmpeg_texture_;
    FILE*                   ffmpeg_;
    tstring                 path_;

public:
    void create(ID3D11Device* device, UINT width, UINT height, UINT fps, const tstring& path = L"../../data")
    {
        width_ = width;
        height_ = height;
        fps_ = fps;
        path_ = path;

        D3D11_TEXTURE2D_DESC desc;
        desc.Width = static_cast<UINT>(width);
        desc.Height = static_cast<UINT>(height);
        desc.MipLevels = 1;
        desc.ArraySize = 1;
        desc.SampleDesc.Count = 1;
        desc.SampleDesc.Quality = 0;
        desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
        desc.Usage = D3D11_USAGE_STAGING;
        desc.BindFlags = 0;
        desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
        desc.MiscFlags = 0;

        if (device->CreateTexture2D(&desc, nullptr, &ffmpeg_texture_) != S_OK)
            tcerr << L"Failed to create staging texture for recording" << std::endl;
    }

    void start_recording(bool compressed = false)
    {
        if (!ffmpeg_texture_)
            return;

        std::time_t t = std::time(nullptr);

        std::tm tm;
        localtime_s(&tm, &t);

        tstringstream file;
        file << path_ << "/record-" << std::put_time(&tm, L"%Y%m%d-%H%M%S") << L".mp4";

        // adapted from https://blog.mmacklin.com/2013/06/11/real-time-video-capture-with-ffmpeg/
        tstringstream cmd;
        cmd << L"ffmpeg -r " << fps_ << " -f rawvideo -pix_fmt rgba "
            << L"-s " << width_ << "x" << height_ << " "
            << L"-i - "
            << L"-threads 2 -y "
            << L"-c:v libx264 "
            << (compressed) ? L"-preset ultrafast -qp 0 " : L"-preset fast "
            << make_absolute_path(file.str())
            ;

        tclog << L"Recording video with: " << cmd.str() << std::endl;

#ifdef UNICODE
        ffmpeg_ = _wpopen(cmd.str().c_str(), L"wb");
#else
        ffmpeg_ = _popen(cmd.str().c_str(), "wb");
#endif

        if (!ffmpeg_)
            tcerr << L"Failed to initialize ffmpeg" << std::endl;
    }

    void stop_recording()
    {
        if (ffmpeg_)
            _pclose(ffmpeg_);

        ffmpeg_ = nullptr;
    }

    void add_frame(ID3D11DeviceContext* context, ID3D11RenderTargetView* rtv)
    {
        ID3D11Resource* resource;
        rtv->GetResource(&resource);
        add_frame(context, resource);
        safe_release(resource);
    }

    void add_frame(ID3D11DeviceContext* context, ID3D11Resource* resource)
    {
        context->CopyResource(ffmpeg_texture_, resource);

        D3D11_MAPPED_SUBRESOURCE msr;
        UINT subresource = D3D11CalcSubresource(0, 0, 0);
        context->Map(ffmpeg_texture_, subresource, D3D11_MAP_READ, 0, &msr);

        if (msr.pData && ffmpeg_)
        {
            fwrite(msr.pData, (width_ * height_ * 4), 1, ffmpeg_);
            context->Unmap(ffmpeg_texture_, subresource);
        }
    }

    void destroy()
    {
        stop_recording();
        safe_release(ffmpeg_texture_);
    }
};

Essentially the video_encoder class creates an open pipe to ffmpeg using the libx264 codec with zero compression. add_frame() copies a resource behind a RenderTargetView to a staging texture, which is then CPU read and fed to the pipe.

The crucial line is where the cmd stringstream is compiled: here you may want toggle compressed to have direct compression enabled and choose custom settings for -qp and -crf to control the compression rate. Either way, be warned that the resulting video is NOT compatible with MP4 support in browsers, because most of them only play content in YUV 4:2:0 color space and these settings output RGBA. I don't have access to D3D11.1 where the staging texture can have DXGI_FORMAT_NV12, but if you do you may want to try this.

If you need maximum performance though, save the video uncompressed and compress it afterwards.

$ ffmpeg -i record-xxxx-xxxx.mp4 -c:v libx264\
         -crf 23 out.mp4

This will give you a nice, compressed MP4 which modern browser can directly display without plugins. You can upload it to your webpage and embed the video with an HTML5 video tag or put it on a Dropbox folder and people can directly watch it. If you plan to cut and edit the video though, skip this step.

Embedding videos into a webpage is easily done with the following code.

<video style="width:100%" controls autoplay>
	<source src="https://foo/bar.mp4" type="video/mp4">
	Your browser does not support the video tag.
</video>

The video should run on any modern browser without any plugins.

Editing

When it comes to video editing my weapon of choice is Final Cut 7, which I'll use to illustrate an important point: most if not all bigger editing tools use a special codec for cutting a video. Apple tools such as FCP and iMovie use the ProRes codec, which comes in several forms (Proxy, LT, HQ). If you value your time, don't want your editing tool to arbitrarily recompress your snippets and maintain full control consider re-encoding your raw FFmpeg dump manually with such a codec. In case of FCP, this will also allow you to use blending operations and other effects without FCPs further interference.

At this point I cannot overstate the magnificent gloriosity of MPEGStreamClip: it's an easy to use free frontend for the Quicktime compressor on Windows and Mac OS X and it has a GUI for batch processing. I usually dump all my videos into the batch list and encode them with Apple ProRes 422 (HQ) at 100% quality. You can also use FFmpeg and simply script this behavior if you're good with the command line. If you have a funny resolution, MPEGStreamClip also has some options to crop and scale the video. A very similar aweseome tool is Handbrake which comes with some neat settings for common enddevices such as smartphone and tablets. Yet another tool for simple cutting, scaling and cropping operations is Virtual Dub, which however doesn't handle the encoding side as nicely.

Converting to the final format

When you are done with the clip, the video is exported into its final format. Here you should take care to use a standard container (no, not WMV/ASF) with a standard video and audio codec (no, not Fraps or Cinepak or DivX or …) so that the video will work in Safari, Chrome, Firefox and Opera directly and play with the default video player of the operating system, even on mobile devices. You may use the command above for FFMpeg (and add audio settings if necessary), or use MPEGStreamclip like this:

• Open your uncompressed, edited video
• Go to File -> Export to MPEG-4
• Choose H.264 compression
• Select 100% quality
• Limit the data rate to 3000-8000 kbps
• Disable sound if there is none in the video
• Export the video

And that's it, you're done. Prepare some template Photoshop intro and outro images (which contain paper title, author names, logos of the conference) and creating new videos from recorded material should be an effort of minutes!

The checklist

• All videos have the same resolution and framerate
• All videos are either 720p or 1080p
• You have re-encoded them with the intermediate codec of your editing tool
• You have created a project in said editing tool with the same resolution and framerate of your videos
• The final output is H.264

Another helpful resource is the Youtube advanced encoding page, which has two tables for recommended bitrates for standard and high quality movies.

References

1. FFmpeg
2. MPEGStreamClip
3. Fraps
4. DXTory
5. NVIDIA, Shadowplay
6. Miles Macklin, Real-Time Video Capture with FFmpeg
7. Google, Youtube Advanced Encoding

Phong

The PDF for a normalized Phong BRDF with the specular power notation is:

We generate two functions to sample theta and phi independently. To do this, we first integrate $p(\mathrm{\theta <!--\; \theta \; -->},\mathrm{\varphi <!--\; \varphi \; -->})$ along the domain of $\mathrm{\varphi <!--\; \varphi \; -->}$:

integrate((n+1)/(2 * %pi) * cos(t)^n * sin(t), p, 0, 2 * %pi)

This is the marginal density function of $\mathrm{\theta <!--\; \theta \; -->}$. Note that the extra $\sin <!--\; \; -->\mathrm{\theta <!--\; \theta \; -->}$ at the end is there because we are dealing with differential solid angles in spherical coordinates. We can retrieve the PDF for $\mathrm{\varphi <!--\; \varphi \; -->}$ with the conditional probability $p(\mathrm{\varphi <!--\; \varphi \; -->}|\mathrm{\theta <!--\; \theta \; -->})$, the conditional density function:

For isotropic NDFs this is always the case, no matter which PDF you will integrate over the domain of $\mathrm{\varphi <!--\; \varphi \; -->}$, so all you need to calculate for the next PDF is the function for $\mathrm{\theta <!--\; \theta \; -->}$.

Now that we have two functions for each variable, we can integrate both to generate a CDF each. The case of $\mathrm{\varphi <!--\; \varphi \; -->}$ is trivial:

integrate(1/(2 * %pi), p, 0, s)

If we set $P(s_{\mathrm{\varphi <!--\; \varphi \; -->}})$ to a random variable ${\mathrm{\xi <!--\; \xi \; -->}}_{\mathrm{\varphi <!--\; \varphi \; -->}}$ and solve for $s_{\mathrm{\varphi <!--\; \varphi \; -->}}$, we get:

Again, sampling around the azimuthal angle of an isotropic NDF is always the same because the material is rotationally invariant along this angle, so you don't need to derive this formula again.

We now repeat the process and create a CDF for $p(\mathrm{\theta <!--\; \theta \; -->})$:

integrate((n+1) * cos(t)^n * sin(t), t, 0, s)

Just like in the case of $\mathrm{\varphi <!--\; \varphi \; -->}$, we set $P(s_{\mathrm{\theta <!--\; \theta \; -->}})$ to a random variable again and solve for $s$:

solve(1 - cos(s)^(n+1) = x, s)

Therefore, a GLSL shader which generates important directions for a Phong NDF from random, uniform values looks like this:

vec2 importance_sample_phong(vec2 xi)
{
  float phi = 2.0f * PI * xi.x;
  float theta = acos(pow(1.0f - xi.y, 1.0f/(n+1.0f)));
  return vec2(phi, theta);
}

Note that since ${\mathrm{\xi <!--\; \xi \; -->}}_{\mathrm{\theta <!--\; \theta \; -->}}\in <!--\; \in \; -->[0,1)$ the expression $1-<!--\; -\; -->{\mathrm{\xi <!--\; \xi \; -->}}_{\mathrm{\theta <!--\; \theta \; -->}}$ is also a random variable in the same range.

You can for instance use a low-discrepancy sequence as a source for uniformly distributed random values xi and generate important samples which aren't clumped.

Trowbridge-Reitz aka GGX

The GGX normal distribution function is part of a microfacet model, which gives you the probability of micro-surface normals oriented along a certain direction. The term has to be normalized though before integrating over the hemisphere to account for the projected micro-surface area:

The NDF itself is defined as:

Just like above, we start out with the PDF for GGX:

As in the case of Phong, we create two functions for $\mathrm{\theta <!--\; \theta \; -->}$ and $\mathrm{\varphi <!--\; \varphi \; -->}$. First let's create $p(\mathrm{\theta <!--\; \theta \; -->})$:

integrate((a^2 * cos(t) * sin(t))/(%pi * ((a^2−1) * cos(t)^2+1)^2), p, 0, 2 * %pi)

The integration for $\mathrm{\varphi <!--\; \varphi \; -->}$ is the same as above, so we skip it and instead now create the CDF for $p(\mathrm{\theta <!--\; \theta \; -->})$:

integrate((2 * a^2 * cos(t) * sin(t))/((a^2−1) * cos(t)^2+1)^2, t, 0, s)

Setting the CDF to a random variable and solving for $s$ yields:

solve(2 * a^2 * (1/((2 * a^4−4 * a^2+2) * cos(s)^2+2 * a^2−2)−1/(2 * a^4−2 * a^2)) = x, s)

A simple GLSL function to generate important directions looks like this:

vec2 importance_sample_ggx(vec2 xi)
{
  float phi = 2.0f * PI * xi.x;
  float theta = acos(sqrt((1.0f - xi.y)/
                          ((a*a - 1.0f) * xi.y + 1.0f)
                         ));
  return vec2(phi, theta);
}

Conclusion

The ultimate goal of mathematics is to eliminate any need for intelligent thought.

Other NDFs can be used to create important samples with the exact same procedure: given a hemispherical PDF (i.e. the specular part of the BRDF), create two independent PDFs for $\mathrm{\theta <!--\; \theta \; -->}$ and $\mathrm{\varphi <!--\; \varphi \; -->}$, integrate both from $0$ to $s$ (i.e. create a CDF for each), set the result to $\mathrm{\xi <!--\; \xi \; -->}$ and solve for $s$.

References

1. Walter et al., Microfacet Models for Refraction through Rough Surfaces
2. Wikipedia, Constructions of low-discrepancy sequences

A particularly brutal aspect of real-time rendering in AR environments is that your object is directly exposed to the physical reality around it for comparison. Unlike your typical game-engine, masking or hiding physical inaccuracies in the model used to simulate real reflection of light on your virtual object is not going to work very well when all the other objects on screen somehow behave differently. If the augmented object doesn't match up with reality, a knee-jerk reaction is to edit the responsible shader and simply tweak the output with a bunch of random multipliers until all looks right again. However, before all the magic-number adjustments get out of hand, it might be time to review the basics and avoid a non-physically-based-shading-post-traumatic-stress-disorder™.

Many real-time engine writers have concentrated on getting their equations right. The most recent result is the Siggraph Physically Based Shading Course (2012 Edition, 2013 Edition). I will roughly lay out the basic principles of PBS and refer to other very good online blogs which have covered most of the details that do not need to be repeated ad absurdum.

Consider the standard form of the rendering equation:

When shading your object, it is important that light reflection off the surface behaves physically plausible, that is the BRDF $f_r$ has to respect certain conditions:

• Positivity: the value of the BRDF is always positive.

• Energy conservation: the total amount of energy reflected over all directions of the surface must be less or equal to the total amount of energy incident to it. In practical terms this means that the visible energy (i.e. reflected light) can at best decrease after bouncing off a surface, while the rest turns to heat or some other form which isn't part of the simulation. A non-emissive surface however cannot emit more light than it received.

• Helmholtz reciprocity: the standard assumption in geometric optics is that exchanging in- and outgoing light direction ${\mathrm{\omega <!--\; \omega \; -->}}_{\mathbf{o}}$ in the BRDF doesn't change the outcome.

• Superposition: the BRDF is a linear function. Contribution of different light sources may be added up independently. There is a debate about whether this is a property of the BRDF or light.

It is usually the energy conservation in the specular term where things go wrong first. Without normalization Blinn-Phong for instance, a popular and easy way to model specular reflection, can easily be too bright with small specular powers and quickly loose too much energy when increasing the term.

But there is more! Many renderers assume that there are materials which are perfectly diffuse, i.e. they scatter incident light in all directions equally. There is no such thing. John Hable demonstrated this by showing materials which would be considered to be diffuse reflectors. You can read more in his article Everything has Fresnel.

So here we are, with a BRDF that can output too much energy, darkens too quickly and can't simulate the shininess of real world objects because the model is built with unrealistic parameters. How do we proceed?

One solution is to find a normalization factor for Blinn-Phong to fix the energy issues with the model and add a Fresnel term. There are also several other reflection models to choose from: Oren-Nayar, Cook-Torrance, Ashikhmin-Shirley, Ward…

Microfacet Models

Physically based BRDF models are built on the theory of microfacets, which states that the surface of an object is composed of many tiny flat mirrors, each with its own orientation. The idea of a microfacet model is to capture the appearance of a macro-surface not by integration over its micro-factets, but by statistical means.

A microfacet model looks like this (with the notation for direct visibility):

where

This specular part of the BRDF has three important components:

• the Fresnel factor $F$
• a geometric term $G$
• a Normal Distribution Function (NDF) $D$

The Fresnel term $F$ simulates the Fresnel behavior and can be implemented with the Schlick approximation.

The geometric term $G$ models the self-occlusion behavior of the microfacets on the surface and can be thought of as a visibility factor for a micro-landscape which simply depends on one parameter for the surface roughness.

The NDF $D$ is the term that gives you the distribution of microfacet normals across the surface from a certain point of view. If more microfacets are oriented in the half-vector direction $\mathbf{h}$, the specular highlight will be brighter. $D$ is the density of normals oriented in direction $\mathbf{h}$.

Conclusion

To sum up, the idea here is to start out with the correct shading model to avoid inconsistencies that might turn up later (I'm speaking out of negative experience tweaking arbitrary parameters of my old AR renderer). Turning to such a model might not produce visible differences immediately, but will later be noticeable once GI is added to the system where the light bounces multiply any error one made right at the start.

A neat way to check how you are on the reality-scale is Disney's BRDF Explorer, which comes with GLSL implementations of several microfacet terms and other BRDFs (have a look at the brdf/ subdirectory and open one of the .brdf files).

You can download measured materials from the MERL database, load them in the BRDF Explorer and compare them to see how well analytical BRDFs match up to reality.

More References

1. Hill et al., Siggraph PBS 2012
2. Hill et al., Siggraph PBS 2013
3. Brian Karis, Specular BRDF Reference
4. John Hable, Everything has Fresnel
5. Rory Driscoll, Energy Conservation In Games
6. Rory Driscoll, Physically-Based Shading
7. Simon Yeung, Microfacet BRDF
8. Sébastien Lagarde, PI or not to PI in game lighting equation
9. Wikipedia, Schlick's approximation
10. Christian Schüler, The Blinn-​Phong Normalization Zoo
11. D3D Book, Cook-Torrance
12. Disney, BRDF Explorer
13. MERL, MERL BRDF Database