Fragments of RealityShedding real light on some not so real things
https://www.tobias-franke.eu/log
In praise of syndication
<h1 id="the-web-is-terrible">The web is terrible</h1>
<p>Useful things tend to become invisible over time. I have a router running <a href="https://www.openbsd.org/">OpenBSD</a> that I had not touched in a while. One day I logged in and got curious.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">uptime
</span>12:26PM up 1583 days, 2:33, 1 user, load averages: 0.13, 0.12, 0.08</code></pre></figure>
<p>It had been running continuously for 4 years without interruption, happily doing its singular task. Similarly, tools like SSH, SCP, rsync, the Sublime text editor, VLC or that one script I wrote years ago all vanish in the daily noise because apart from doing their job, they are primarily <em>not annoying</em>.</p>
<p>The same cannot be said for browsing the web anymore. I use a somewhat dated but otherwise fully functional MacBook 13” from 2013 with the latest MacOS. Most things are working just fine, but as soon as I open a browser and go to any bigger website, the poor piece of silicon is choked to death by a barrage of trackers and useless features packed together in a lasagna of Javascript libraries, each running in the background using websockets and Ajax requests to continue the torture.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2019_08_firefox-cpu-usage.png"><img src="https://www.tobias-franke.eu/layout/logcache/2019_08_firefox-cpu-usage.png" alt="" /></a></p>
<p>Even with a beefy machine I notice laggy input in every bigger web application. In some cases <a href="https://twitter.com/cpeterso/status/1021626510296285185">browser vendors running big websites are actively abusing APIs</a> to stop you from using their competitors. Apart from the dreaded interface changes that are imposed on visitors every now and then, content is often hidden in completely overloaded designs, buried under advertisements and self playing videos. The problem has become so endemic to everyday online life that several browsers now come with built-in “Reader Views” or services such as <a href="https://www.getpocket.com">Pocket</a> to trim down websites to their bare essence. Without an almost mandatory set of ad- and script blockers, redirect skippers, CDN caches, fingerprint scramblers and privacy containers (the list goes on), almost every modern website is a cocktail of ad-infested privacy nightmares. And then there is the fine threshold where running too many add-ons to curate the web is a bottleneck and security threat itself, so having multiple browsers on a device is also considered “normal” these days. Maciej Cegłowski summarized the state of modern web design in his hilarious talk <a href="https://idlewords.com/talks/website_obesity.htm">The Website Obesity Crisis</a> with the following image:</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2019_08_the-web-pyramid.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2019_08_the-web-pyramid.jpg" alt="" /></a></p>
<p>A great way to combat many of these problems is to use <a href="https://en.m.wikipedia.org/wiki/RSS">RSS</a>. I recently noticed that over the last year I had relocated most of what I consume online to an RSS reader. I had been using it for now more than 12 years, mostly to follow blogs, but now on some days I only open the reader instead of my browser. I am not alone in this move: <a href="https://www.wired.com/story/rss-readers-feedly-inoreader-old-reader/">Wired ran an article</a> about the resurgence of RSS because of the way the web had transformed. A reader on Hacker News soon after <a href="https://news.ycombinator.com/item?id=16722351">pointed out the irony</a> that the very same article - suggesting RSS as a solution to this hot mess - came with its own clown-car of trackers. In a very twisted way, this article made clear why most of the web is terrible.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2019_08_wired-ublock.png"><img src="https://www.tobias-franke.eu/layout/logcache/2019_08_wired-ublock.png" alt="" /></a></p>
<h1 id="an-elegant-weapon-for-a-more-civilized-age">An elegant weapon for a more civilized age</h1>
<p>For those unfamiliar, RSS is a standardized container format for content updates, for instance blog posts or news articles. Such updates usually have a title, an author, a publication date, some content (i.e. text and images) and perhaps a list of tags. RSS encapsulates each update in a bit of XML. An RSS reader - often referred to as a feed aggregator - which you point to an RSS stream will, in regular intervals, check the stream if any new updates appeared and download them for you to view later. In many ways, it is like having an inbox for website updates.</p>
<p>But it doesn’t have to end at blog posts. Most of what we read and watch every day can be encapsulated with RSS: Facebook posts, Tweets, Instagram pictures, new videos uploaded Youtube/Twitch/Vimeo, Github commits etc. are all updates on websites we check repeatedly every day. That manual labor can be avoided by subscribing to their RSS streams.</p>
<p>Say you are interested in <em>computer graphics</em>. You may have a couple of people posting on Twitter that you follow. There is also the SIGGRAPH Facebook page you open every day to scan for interesting tidbits. You are also on Mastodon, because some people from the community are gathering there. Additionally, you read a ton of blogs, and you follow the Youtube channels of Khronos, GDC and some researchers, for which you get notified through your subscription on Youtube. Except this one person that hosts their videos on Vimeo, that one you need to check on your Vimeo account. At last there are the Github projects with the occasional releases and the starred repositories of people you follow. To sum up: The content category you are interested in - <em>computer graphics</em> - is splattered all over the place and every day you will need to go out and collect everything back together.</p>
<p>With an RSS reader, you can open a folder - let’s call it <em>Computer Graphics</em> - and simply add all those sources into it. It is now the RSS readers task to fetch new items in regular intervals and notify you. You will never have to visit any of those pages again to check for updates. After all, if it is just a category you are interested in, who cares were the content comes from?</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2019_08_music-subs.png"><img src="https://www.tobias-franke.eu/layout/logcache/2019_08_music-subs.png" alt="" /></a></p>
<p>The image above shows a sample from my <em>Music</em> folder, where I follow Soundcloud accounts next to Youtube, Vimeo and a blog at the same time. All relevant data ends up in the same stream of notifications.</p>
<h1 id="what-do-i-gain">What do I gain?</h1>
<p>Apart from reducing the menial task of visiting a bunch of websites every day, RSS has some nice side-effects.</p>
<p><em>No walled gardens</em>: RSS readers allow you to fetch content from a multitude of sources and group it in any way you please. Decentralization comes naturally and is easily managed. Content is not bound to specific websites or providers.</p>
<p><em>No accounts</em>: On Youtube for instance, subscribing to channels requires an account. For each other walled garden where you subscribe to something, you will need an additional account. You can get away without any accounts by aggregating all subscriptions via RSS.</p>
<p><em>Limited tracking</em>: RSS encapsulates raw content of an update. A reader can show you a new item without you visiting any website. All the Javascript nonsense that comes with a page you would usually visit (like the omnipresent Google Analytics) is gone.</p>
<p><em>No interspersed ads</em>: Twitter is among one of many websites that inject advertisement posts called “promoted tweets” into your stream. Technically, that is possible with RSS too. Without accounts and scripts following your every move, building a profile about you is limited though.</p>
<p><em>Unfiltered and unsorted by machine learning</em>: Many services will employ some type of machine learning to show you something <em>they</em> deem interesting for <em>you</em>. Chronological order goes out the window, as these services push the “top” or “most liked post” to the beginning of your stream. RSS is not a service controlled by anyone, therefore the order is yours to control.</p>
<p><em>Faster browsing</em>: Content is served as raw text that may be formatted with simple HTML. Instead of opening 10 websites that each harass your browser, one feed with limited HTML or text-only messages and pictures remains. All content is served in a common theme.</p>
<p><em>No social features</em>: Without walled gardens and no centralized sources, there are also no common comment sections or share buttons.</p>
<p><em>Privacy</em>: Given all of the points above, privacy of whom you follow and what you read is maintained.</p>
<p><em>Security</em>: Generally, the less intelligent and powerful a system is, the less damage it can inflict. Technological blights like <a href="https://en.wikipedia.org/wiki/Cross-site_scripting">Cross Site Scripting attacks</a> are simply not a thing when you read raw text.</p>
<h1 id="what-can-go-wrong">What can go wrong?</h1>
<p>Of course not everything is suited for RSS, such as legitimate web applications like shops or banking portals, and there are some down-sides to consider when relying heavily on RSS.</p>
<p><em>RSS may not be supported</em>: A website you like and want get updates from may provide no or only inadequate RSS streams. For instance, a news page could give you an RSS stream for each category, or the entire site, or none at all. In these cases you might need to rely on <a href="https://github.com/RSS-Bridge/rss-bridge">third-party scrapers</a> that will generate RSS content from the website. However, a nice side-effect of those services is that they act as proxies: Since you are not directly subscribing to the pages RSS, it won’t even know you where ever there.</p>
<p><em>Non-public data inaccessible</em>: Scrapers can turn any public website into an RSS stream, but anything locked behind an account can become a problem. Some websites implement private RSS streams using secret tokens (for instance the personal GitHub news feed), but in most cases you will be out of luck.</p>
<p><em>Only previews available</em>: Many RSS streams do not provide the full content of the update, only a preview, forcing you to open the original website again. It is not the end of the world, but annoying. There are again <a href="https://rssbridge.net/fulltext/">third party services that will automatically expand to full content RSS streams</a> from preview-only streams.</p>
<p><em>No social features</em>: Without walled gardens and no centralized sources, there are also no common comment sections or share buttons.</p>
<p><em>No recommendations</em>: Platforms such as Vimeo or Instagram will, given user tracking through accounts or other means, compile recommendations of what to watch or whom to follow. Decentralized and without accounts, there is no recommendation page anymore. Online RSS readers such as Feedly might recommend more feeds, but in general terms this feature does not exist anymore.</p>
<p><em>Synchronization between devices</em>: To keep track which article on which feed was marked read or unread, RSS readers keep an index database. These index databases are not standardized and often cannot be synchronized easily between different readers. Online RSS readers such as <a href="https://feedly.com/">Feedly</a> solve this problem by synchronizing an online index with a proprietary protocol. <a href="https://rss-sync.github.io/Open-Reader-API/">An effort is underway</a> to solves this issue.</p>
<p><em>Multimedia content</em>: Some items can be subscribed to via RSS, but not shown as purely raw text-content, such as video or music subscriptions. In these cases your RSS reader will use a built-in or external browser to open a link. However, getting notified this way is still preferable.</p>
<h1 id="on-readers">On Readers</h1>
<p>Armed with the knowledge of the pros and cons of using RSS, you can start by installing an RSS reader to subscribe to something.</p>
<p>Generally, you want to look out for a reader that supports import and export of your subscriptions. The commonly used format for this is called <a href="https://en.wikipedia.org/wiki/OPML">OPML</a>. Simply put, OPML is a standardized list of all your subscriptions. A current backup of those subscriptions can be used to quickly change from one reader to another, and they are very easy to edit in case one wants to mass-change RSS feed subscriptions to an alternative. For instance, <a href="https://support.google.com/youtube/answer/6224202?hl=en">you can export all your Youtube subscriptions as OPML</a> and easily migrate them to <a href="https://www.invidio.us">Invidious</a>, an open-source Youtube proxy, using a simple search-and-replace with your favorite text editor:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html">https://www.youtube.com/feeds/videos.xml?channel_id=ID
->
https://invidio.us/feed/channel/ID</code></pre></figure>
<p><em>Keeping a backup of all your subscriptions is essential to being independent of any third-party services or readers!</em></p>
<p>There are two types of readers: Online readers running in a browser, which keep all your subscriptions behind an online account, or local readers, which may synchronize with an online reader account. The go-to online reader and winner of the Google Reader exodus is <a href="https://www.feedly.com">Feedly</a>. Feedly supports OPML import-export and can synchronize with a wide range of mobile and desktop readers. If you have no reader application at hand, you can always use the web version. Feedly and other online RSS readers often limit the amount of RSS feeds you can subscribe to for free. An alternative is to use an offline RSS reader or to host your own online RSS reader. <a href="https://freshrss.org/">FreshRSS</a> is an open-source project that can be set up with minimal friction on a webserver with PHP.</p>
<p>There is <a href="https://en.wikipedia.org/wiki/Comparison_of_feed_aggregators">a multitude of very nice RSS readers</a> out there, but I’ll mention only a few very basic ones here:</p>
<ul>
<li><a href="https://www.thunderbird.net/">Thunderbird</a>, the well known Mozilla email client, supports local RSS subscriptions.</li>
<li><a href="https://nodetics.com/feedbro/">Feedbro</a>, a browser extension, comes with OPML support and additionally can render <a href="https://www.mathjax.org">MathJax</a> in blog posts. Since it is browser-based, it can be used in conjunction with uBlock to weed out any additional requests coming from embedded media.</li>
<li><a href="https://itunes.apple.com/us/app/rss-mobile/id533007246">RSS Mobile</a>, the most basic iOS RSS reader I could find. Free, no ads, runs without any web service.</li>
<li><a href="https://github.com/ahmaabdo/ReadifyRSS">Readify</a>, an open-source reader and equivalent of RSS Mobile for Android.</li>
</ul>
<h1 id="rss-streams-of-common-services">RSS streams of common services</h1>
<p>Apart from blogs, one of the most interesting use cases of RSS is to escape social media bubbles and walled gardens. Because RSS prevents social media platforms from tracking you, most have <a href="https://mashable.com/2012/09/05/twitter-api-rss/?europe=true">dropped their RSS support</a> entirely. However, this is where scrapers such as <a href="https://github.com/RSS-Bridge/rss-bridge">RSS-Bridge</a> come to the rescue: These services parse website content and turn it into an RSS stream, shielding you away entirely from the site. In the list below, replace <code class="highlighter-rouge">RSS-BRIDGE-INSTANCE</code> with for example <a href="https://bridge.suumitsu.eu/">https://bridge.suumitsu.eu/</a>. There are many more publicly hosted instances of RSS-Bridge, and of course you can run your own.</p>
<p>Here is a sample of RSS streams you can use to replace typical social media:</p>
<ul>
<li><em>Twitter</em>: <code class="highlighter-rouge">https://RSS-BRIDGE-INSTANCE/?action=display&bridge=Twitter&format=Atom&u=USERNAME</code></li>
<li><em>Mastodon</em>: <code class="highlighter-rouge">https://MASTODON-INSTANCE/@USERNAME</code></li>
<li><em>Instagram</em>: <code class="highlighter-rouge">http://instatom.freelancis.net/USERNAME</code></li>
<li><em>Facebook</em>: <code class="highlighter-rouge">https://RSS-BRIDGE-INSTANCE/?action=display&bridge=Facebook&media_type=all&limit=-1&format=Atom&u=USERNAME</code></li>
<li><em>Hacker News</em>: <code class="highlighter-rouge">http://feeds.feedburner.com/fullhackernews</code></li>
<li><em>Reddit</em>: <code class="highlighter-rouge">http://inline-reddit.com/feed/?subreddit=SUBREDDIT</code></li>
<li><em>Youtube</em>: <code class="highlighter-rouge">https://invidio.us/feed/channel/CHANNELID</code></li>
<li><em>Vimeo</em>: <code class="highlighter-rouge">https://vimeo.com/USERNAME/videos/rss</code></li>
<li><em>Twitch</em>: <code class="highlighter-rouge">https://twitchrss.appspot.com/vod/USERNAME</code></li>
<li><em>Tumblr</em>: <code class="highlighter-rouge">https://USERNAME.tumblr.com/rss</code></li>
<li><em>GitHub</em>: <code class="highlighter-rouge">https://github.com/USERNAME/PROJECT/releases.atom</code></li>
<li><em>Soundcloud</em>: <code class="highlighter-rouge">https://rssbox.herokuapp.com/</code></li>
</ul>
<p>I keep a more comprehensive list updated on this <a href="https://gist.github.com/thefranke/63853a6f8c499dc97bc17838f6cedcc2">Github Gist</a>. Note that for most of these pages there are often multiple alternative scraper projects. If you cannot find an RSS stream of a website or service you want to subscribe to, consider contributing to projects such as <a href="https://github.com/RSS-Bridge/rss-bridge">RSS-Bridge</a>, <a href="https://github.com/DIYgod/RSSHub">RSSHub</a> or <a href="https://github.com/stefansundin/rssbox">RSSBox</a>. Multiple instances of these projects are hosted for free by volunteers, and you may want to consider running your own as well.</p>
<h1 id="conclusion">Conclusion</h1>
<p>RSS is a great way to keep track of content updates on any number of sources you visit daily. It naturally shields against unnecessary ad-tracking and separates actual content from virtual garbage, making the experience of reading and viewing more pleasant. RSS frees from the bonds of walled gardens and allows you to organize all of your sources by categories again instead of where they originate from: It is no longer your “Youtube subscriptions” or “people you follow on Instagram”, but subscription bundles of “cat videos” or “food pictures”, wherever they might come from. A well maintained subscription list will reduce the time you spend mindlessly surfing the web in search for new stuff.</p>
<p>If you are as upset as I am about the current state of the WWW, I encourage you to consider this alternative way of consuming the web, or to revive your RSS reader if you have abandoned it a couple of years back.</p>
<p>Here is my recommendation:</p>
<ol>
<li>Find a good reader you feel comfortable with or sign up to <a href="https://feedly.com/">Feedly</a>. I’m on MacOS and prefer <a href="https://reederapp.com/">Reeder</a>.</li>
<li>Add a couple of blogs.</li>
<li>Subscribe to some of your favorite Youtube channels (here is one I like called <a href="https://rsshub.app/youtube/channel/UCYO_jab_esuFRV4b17AJtAw">3Blue1Brown</a>).</li>
<li><em>Use it for a couple of days or weeks to get a feel for it.</em></li>
<li>Use <a href="https://gist.github.com/thefranke/63853a6f8c499dc97bc17838f6cedcc2">this list</a> to populate your reader with more content.</li>
<li>If your reader supports a usage overview (like <a href="https://feedly.com/i/organize/my">Organize Sources</a> on Feedly), identify and eliminate spammy sources.</li>
</ol>
<p>And then, if you prefer it, slowly, but steadily, subscribe to to everything you watch, read or listen to online. You can <a href="https://www.tobias-franke.eu/log/rss/index.xml">start with my log right now</a>.</p>
<h1 id="references">References</h1>
<ol>
<li>Wikipedia, <a href="https://en.m.wikipedia.org/wiki/RSS">RSS</a></li>
<li>Tobias A. Franke, <a href="https://gist.github.com/thefranke/63853a6f8c499dc97bc17838f6cedcc2">A list of RSS endpoints, readers and resources</a></li>
<li>Wikipedia, <a href="https://en.wikipedia.org/wiki/Comparison_of_feed_aggregators">Comparison of readers</a></li>
<li>RSS-Bridge, <a href="https://github.com/RSS-Bridge/rss-bridge">RSS-Bridge Project</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Wed, 07 Aug 2019 00:00:00 +0200
https://www.tobias-franke.eu/log/2019/08/07/in-praise-of-syndication.html
https://www.tobias-franke.eu/log/2019/08/07/in-praise-of-syndication.htmlNotesRSSTriple Product Integrals
<h1 id="the-grand-picture">The Grand Picture</h1>
<p>Most introductions and implementations of <em>Precomputed Radiance Transfer</em> will deal fairly well with the easiest use case: The double product integral for diffuse reflections. Beyond this scope however, few show how proper rotation of frequency-space encoded lighting works, and even fewer dive into the problem of view-dependency in PRT. Both of these are related, as they rely on the ability to transform one set of coefficents into another: Rotating a vector creates another vector, and view dependent reflection transforms incident light represented as coefficients into coefficients of reflected light. From linear algebra, we know how to deal with such a scenario already: To transform one vector into another we need a matrix, so in terms of a lighting scenario we need to talk about <em>Matrix Radiance Transfer</em> and the <em>Triple Product Integral</em>.</p>
<h1 id="function-transforms">Function transforms</h1>
<p>In <a href="https://www.tobias-franke.eu/log/2016/10/18/the_convolution_theorem.html">the last post on function transforms</a> I took a quick look at the convolution theorem, which one can roughly describe as the ability to shortcut an integration over the product of two functions, i.e., a convolution, with a dot product in the frequency domain. Typically, the convolution theorem is introduced with the Fourier basis, but other function bases can be used as well.</p>
<script type="math/tex; mode=display">\begin{equation}
\int_S f(s) g(s) ds = \sum_i f_i g_i
\end{equation}</script>
<p>This holds true for any function basis that is <em>orthonormal</em>. To test whether a set of functions forms an orthonormal basis, one needs to do two things: Make sure that any two basis functions $\Phi_i(s)$ and $\Phi_j(s)$ integrate to 0 if $i \neq j$, and to 1 if $i = j$.</p>
<script type="math/tex; mode=display">\begin{equation}\label{dirac_delta}
\int_S \Phi_i(s) \cdot \Phi_j(s) ds = \delta_{ij} =
\left\{
\begin{array}{c}
1, i = j \\
0, i \neq j
\end{array}
\right.
\end{equation}</script>
<p>This works completely analogous to a vector basis.</p>
<h1 id="the-triple-product-integral">The Triple Product Integral</h1>
<p>So lets assume we have two functions $f(s)$ and $g(s)$ and we want to compute the function product of both as a new function $e(s)$.</p>
<script type="math/tex; mode=display">\begin{equation}\label{e-fun}
e(s) = f(s) g(s)
\end{equation}</script>
<p>Assume further that we have projected $f(s)$ and $g(s)$ into the frequency domain of the function basis $\Phi_i(s)$. This means that we have the coefficients for both at hand.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
f(s) & = & \sum_j f_j \Phi_j(s) \label{f-fun} \\
g(s) & = & \sum_k g_k \Phi_k(s) \label{g-fun}
\end{eqnarray} %]]></script>
<p>It may be possible to do the same operation in Equation $\ref{e-fun}$ - multiplying two functions together - in the frequency space of the basis $\Phi$ that we chose. How would we be able to determine the coefficients $e_i$ of $e(s)$? We can start the usual way by integrating it with the basis function.</p>
<script type="math/tex; mode=display">\begin{equation}
e_i = \int_S e(s) \Phi_i(s) ds
\end{equation}</script>
<p>But since we already have the coefficients for both $f(s)$ and $g(s)$, we can first replace $e(s)$ by Equation \ref{e-fun} and then replace further with both Equation \ref{f-fun} and \ref{g-fun}.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
e_i & = & \int_S \Phi_i(s) e(s) ds \\
& = & \int_S \Phi_i(s) \left[ f(s) g(s) \right] ds \\
& = & \int_S \Phi_i(s) \left[ \left( \sum_j f_j \Phi_j(s) \right) \left( \sum_k g_k \Phi_k(s) \right) \right] ds
\end{eqnarray} %]]></script>
<p>We can rearrange some things: All the terms that are not dependent on $s$ can be moved out of the integral.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
e_i & = & \int_S \Phi_i(s) e(s) ds \\
& = & \int_S \Phi_i(s) \left[ f(s) g(s) \right] ds \\
& = & \int_S \Phi_i(s) \left[ \left( \sum_j f_j \Phi_j(s) \right) \left( \sum_k g_k \Phi_k(s) \right) \right] ds \\
& = & \sum_j \sum_k f_j g_k \int_S \Phi_i(s) \Phi_j(s) \Phi_k(s) ds \\
& = & \sum_j \sum_k f_j g_k C_{ijk} \label{f-tripling-coeff}
\end{eqnarray} %]]></script>
<p>Now that is an interesting looking integral! Three basis functions in one, and at first sight the whole process looks quite familiar too. It is almost identical to what we did to derive the convolution theorem, now we <em>only</em> need to get rid of this integral and we’re done.</p>
<p>The term $\int_S \Phi_i(s) \Phi_j(s) \Phi_k(s) ds$ is called <em>the triple product integral</em> for more or less obvious reasons, and $C_{ijk}$ is a <em>tripling coefficient</em>. Its smaller sibling, the double product integral, is reducable to a Kroenecker Delta if the basis functions $\Phi_i(s)$ form an orthonormal basis. But for $C_{ijk}$, things are not so easy. In fact, there is no general analytical solution for any arbitrary function basis.</p>
<h1 id="computing-tripling-coefficients">Computing tripling coefficients</h1>
<p>Luckily, for the two most common bases used for PRT, Haar-Wavelets and Spherical Harmonics, an analytical formula to compute their tripling coefficents exists.</p>
<p>Tripling coefficients of Spherical Harmonics can be expressed through <a href="https://en.wikipedia.org/wiki/Clebsch–Gordan_coefficients#Relation_to_Wigner_3-j_symbols">a relationship with Clebsch-Gordan coefficients</a>, which in turn can be expressed by Wigner 3j symbols. Below is an older implementation of mine, but as far as I’m aware the <a href="https://www.gnu.org/software/gsl/">GNU Scientific Library</a> contains one as well.</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">T</span><span class="o">></span>
<span class="kr">inline</span> <span class="n">T</span> <span class="n">wigner_3j</span><span class="p">(</span><span class="kt">int</span> <span class="n">j1</span><span class="p">,</span> <span class="kt">int</span> <span class="n">j2</span><span class="p">,</span> <span class="kt">int</span> <span class="n">j3</span><span class="p">,</span>
<span class="kt">int</span> <span class="n">m1</span><span class="p">,</span> <span class="kt">int</span> <span class="n">m2</span><span class="p">,</span> <span class="kt">int</span> <span class="n">m3</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">assert</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">abs</span><span class="p">(</span><span class="n">m1</span><span class="p">)</span> <span class="o"><=</span> <span class="n">j1</span> <span class="o">&&</span>
<span class="s">"wigner_3j: m1 is out of bounds"</span><span class="p">);</span>
<span class="n">assert</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">abs</span><span class="p">(</span><span class="n">m2</span><span class="p">)</span> <span class="o"><=</span> <span class="n">j2</span> <span class="o">&&</span>
<span class="s">"wigner_3j: m2 is out of bounds"</span><span class="p">);</span>
<span class="n">assert</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">abs</span><span class="p">(</span><span class="n">m3</span><span class="p">)</span> <span class="o"><=</span> <span class="n">j3</span> <span class="o">&&</span>
<span class="s">"wigner_3j: m3 is out of bounds"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">triangle_inequality</span><span class="p">(</span><span class="n">j1</span><span class="p">,</span> <span class="n">j2</span><span class="p">,</span> <span class="n">j3</span><span class="p">))</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">m1</span><span class="o">+</span><span class="n">m2</span><span class="o">+</span><span class="n">m3</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">abs</span><span class="p">(</span><span class="n">m1</span><span class="p">)</span> <span class="o">></span> <span class="n">j1</span><span class="p">)</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">abs</span><span class="p">(</span><span class="n">m2</span><span class="p">)</span> <span class="o">></span> <span class="n">j2</span><span class="p">)</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">abs</span><span class="p">(</span><span class="n">m3</span><span class="p">)</span> <span class="o">></span> <span class="n">j3</span><span class="p">)</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">j2</span> <span class="o">-</span> <span class="n">m1</span> <span class="o">-</span> <span class="n">j3</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">j1</span> <span class="o">+</span> <span class="n">m2</span> <span class="o">-</span> <span class="n">j3</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">t3</span> <span class="o">=</span> <span class="n">j1</span> <span class="o">+</span> <span class="n">j2</span> <span class="o">-</span> <span class="n">j3</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">t4</span> <span class="o">=</span> <span class="n">j1</span> <span class="o">-</span> <span class="n">m1</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">t5</span> <span class="o">=</span> <span class="n">j2</span> <span class="o">+</span> <span class="n">m2</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">tmin</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">max</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">max</span><span class="p">(</span><span class="n">t1</span><span class="p">,</span> <span class="n">t2</span><span class="p">));</span>
<span class="kt">int</span> <span class="n">tmax</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">min</span><span class="p">(</span><span class="n">t3</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">min</span><span class="p">(</span><span class="n">t4</span><span class="p">,</span> <span class="n">t5</span><span class="p">));</span>
<span class="k">if</span> <span class="p">(</span><span class="n">tmin</span> <span class="o">></span> <span class="n">tmax</span><span class="p">)</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">T</span> <span class="n">wigner</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">auto</span> <span class="n">f</span> <span class="o">=</span> <span class="n">factorial</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">t</span> <span class="o">=</span> <span class="n">tmin</span><span class="p">;</span> <span class="n">t</span> <span class="o"><=</span> <span class="n">tmax</span><span class="p">;</span> <span class="o">++</span><span class="n">t</span><span class="p">)</span>
<span class="n">wigner</span> <span class="o">=</span> <span class="n">wigner</span> <span class="o">+</span> <span class="k">static_cast</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">pow</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">t</span><span class="p">))</span><span class="o">/</span>
<span class="p">(</span><span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">t</span><span class="o">-</span><span class="n">t1</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">t</span><span class="o">-</span><span class="n">t2</span><span class="p">)</span> <span class="o">*</span>
<span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">t3</span><span class="o">-</span><span class="n">t</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">t4</span><span class="o">-</span><span class="n">t</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">t5</span><span class="o">-</span><span class="n">t</span><span class="p">));</span>
<span class="n">wigner</span> <span class="o">=</span> <span class="n">wigner</span> <span class="o">*</span> <span class="k">static_cast</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">pow</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">j1</span><span class="o">-</span><span class="n">j2</span><span class="o">-</span><span class="n">m3</span><span class="p">))</span> <span class="o">*</span>
<span class="k">static_cast</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">sqrt</span><span class="p">(</span>
<span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j1</span><span class="o">+</span><span class="n">j2</span><span class="o">-</span><span class="n">j3</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j1</span><span class="o">-</span><span class="n">j2</span><span class="o">+</span><span class="n">j3</span><span class="p">)</span> <span class="o">*</span>
<span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="o">-</span><span class="n">j1</span><span class="o">+</span><span class="n">j2</span><span class="o">+</span><span class="n">j3</span><span class="p">)</span> <span class="o">/</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j1</span><span class="o">+</span><span class="n">j2</span><span class="o">+</span><span class="n">j3</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span>
<span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j1</span><span class="o">+</span><span class="n">m1</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j1</span><span class="o">-</span><span class="n">m1</span><span class="p">)</span> <span class="o">*</span>
<span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j2</span><span class="o">+</span><span class="n">m2</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j2</span><span class="o">-</span><span class="n">m2</span><span class="p">)</span> <span class="o">*</span>
<span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j3</span><span class="o">+</span><span class="n">m3</span><span class="p">)</span> <span class="o">*</span> <span class="n">f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">j3</span><span class="o">-</span><span class="n">m3</span><span class="p">)</span>
<span class="p">));</span>
<span class="k">return</span> <span class="n">wigner</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>With the tripling coefficients at hand, we can create a function product of aribtrary functions encoded in the same frequency domain. A major downside of regular PRT is that the BRDF, visibility and Lambert factor are all represented simply as one transfer function, which therefore gets encoded as one transfer vector. The Lambert factor however is a very low-frequency signal, whilst visibility and BRDF may contain high-frequency peaks. Packing them all together means choosing either to produce a lot of waste coefficients or compromising on the quality of the representation. Ng et al. get around this by <a href="https://graphics.stanford.edu/papers/allfreqmat/">decoupling visibility and the BRDF in the transfer function again</a> with a triple product composition.</p>
<p>Another use case presented in <a href="https://www.microsoft.com/en-us/research/publication/precomputed-shadow-fields-for-dynamic-scenes/">Precomputed Shadow Fields for Dynamic Scenes</a> wraps an object in a shell which, at several sample points, has coefficients encoding a visibility function from the point on that shell. If the shell collides with another PRT-object, coefficients of both objects visibility can be combined to compute dynamic shadowing between the two, getting rid of some of the rigidness requirements that PRT imposes on the scene. This paper is naturally enhanced in <a href="https://diglib.eg.org/handle/10.2312/CGF.v26i3pp485-493">Precomputed Radiance Transfer Field for Rendering Interreflections in Dynamic Scenes</a>, where the authors not just encode visibility, but indirect diffuse transfer between objects as well.</p>
<h1 id="transforming-coefficients">Transforming coefficients</h1>
<p>Returning to Equation \ref{f-tripling-coeff} we can easily construct $e_i$ in another way.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
e_i & = & \sum_j \sum_k f_j g_k C_{ijk} \\
e_i & = & \sum_j f_j T_{ij}
\end{eqnarray} %]]></script>
<p>Instead of using the tripling coefficient tensor $C_{ijk}$ and $g_k$, we can also use a matrix $T_{ij}$ (one way to construct it would be to multiply $C_{ijk}$ and $g_k$). This matrix will <em>transform</em> coefficients $f_j$ into coefficients $e_i$, which represent the function $e(s)$ in the basis $\Phi$ that we chose.</p>
<p>The concept of a matrix to transform a coefficient vector is sometimes mentioned in PRT tutorials for signal rotation. Rather than having an environment map $m(s)$, rotate it with $\mathbf{R}(m)(s)$ and then resample and recompute the Spherical Harmonic coefficients for it, we can likewise <em>rotate</em> the coefficents $m_i$ with a special frequency-space rotation matrix $\mathbf{R^{\star}}$. In essence, we would produce a new set of coefficients that simply match those of the rotated environment map in image space.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
m(s) & = & \sum_i m_i \Phi_i(s) ds \\
\mathbf{R}(m)(s) & = & \sum_i \sum_j (m_j \cdot R^{\star}_{ij}) \Phi_i(s) ds \\
\end{eqnarray} %]]></script>
<p><a href="https://github.com/thefranke/deimos/blob/master/math/spherical_harmonics.h#L344">Here is an old implementation of mine</a> of one such matrix to do the job. The input is a regular 3x3 rotation matrix, but the result is a matrix that needs to be as big as the coefficient vector you want to rotate. In case of 4-band Spherical Harmonic vector for instance, the result will be a 16x16 <em>Spherical Harmonic rotation matrix</em>.</p>
<h1 id="matrix-radiance-transfer">Matrix Radiance Transfer</h1>
<p>In the diffuse case of PRT, we only need to compute a single number/color, because diffuse surfaces reflect the same intensity in all directions. The double product integral fits the job, because the convolution acts like a blur-filter that adds up all the light hitting the surface and reflects a single value. But for non-diffuse surfaces like metals this outgoing radiance varies in different directions, so we need more than just one number; we need a function that represents all light reflected into all different directions. We need a coefficient vector of the reflected light that we generate from the incident light coefficient vector.</p>
<p>For diffuse PRT, we turn a transfer function of <em>just</em> the incidenct light direction into the transfer coefficient vector. Now however we must turn a transfer function of an incident and outgoing direction into a transfer matrix. But how to construct the matrix for a function that has two variables? We can do so by imagining that we have a special set of transfer coefficients <em>for each outgoing direction</em> $\mathbf{\omega_o}$, rather than just a single set for all of them. Where before we would turn a transfer function into a set of coefficients, we now basically convert it into a set of functions $T_m(\mathbf{\omega_o})$ which return the $m$-th coefficient of the reflected light for an outgoing direction $\mathbf{\omega_o}$. To do this, we first integrate $T$ along all incident directions $\mathbf{\omega_i}$.</p>
<script type="math/tex; mode=display">\begin{equation}
T_m(\mathbf{\omega_o}) = \int_{\Omega} T(\mathbf{\omega_i}, \mathbf{\omega_o}) \Phi_m(\mathbf{\omega_i}) d{\mathbf{\omega_i}}
\end{equation}</script>
<p>We then integrate each resulting function $T_m(\mathbf{\omega_o})$, this time over all outgoing directions $\mathbf{\omega_o}$.</p>
<script type="math/tex; mode=display">\begin{equation}
T_{mn} = \int_{\Omega} T_m(\mathbf{\omega_o}) \Phi_n(\mathbf{\omega_o}) d{\mathbf{\omega_o}}
\end{equation}</script>
<p>In <a href="https://www.tobias-franke.eu/log/2016/10/18/the_convolution_theorem.html">the previous article</a>, the double integral product was used to compute the outgoing reflected light from the incident light coefficients $l_n$ and the surface transfer coefficients $t_n$. We <em>transferred</em> incident light into single value (or, if you want to think about it this way, into function which equals a constant value), which is perfect for diffuse reflection. Now with <a href="https://mediatech.aalto.fi/~jaakko/publications/lehtinen2003i3d_paper.pdf">Matrix Radiance Transfer</a> a matrix $T_{mn}$ instead represents the transfer of incident to exit light.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
L(\mathbf{\omega_o}) & = & L_e(\mathbf{\omega_o}) + \int_\Omega L(\mathbf{\omega_i}) \cdot f( \mathbf{\omega_i}, \mathbf{\omega_o}) \cdot \langle \mathbf{n}, \mathbf{\omega_i} \rangle d\mathbf{\omega_i} \\
& = & L_e(\mathbf{\omega_o}) + \int_\Omega L(\mathbf{\omega_i}) \cdot T( \mathbf{\omega_i}, \mathbf{\omega_o}) d\mathbf{\omega_i} \\
& = & L_e(\mathbf{\omega_o}) + \sum_n \sum_m \left( l_m T_{mn} \right) \Phi_n(\mathbf{\omega_o}) \\
& = & L_e(\mathbf{\omega_o}) + \langle \langle \mathbf{l}, \mathbf{T} \rangle, \mathbf{\Phi}(\mathbf{\omega_o}) \rangle \\
\end{eqnarray} %]]></script>
<p>Where we used to have a dot product of two vectors, we now produce a <em>reflection coefficent vector</em> from incident light, and then reconstruct the light reflected into a <em>specific</em> direction $\mathbf{\omega_o}$ by multiplying it with the basis $\Phi$.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Like <em>regular</em> vectors in 3D, coefficient vectors can likewise be transformed with matrices. This is useful when a function - encoded as such a vector - should be transformed into a different function, such as unoccluded to occluded light, normal to rotated environment, incident to exit radiance or when simply decoupling one transfer function into several parts and then multiplying them back together again.</p>
<h1 id="references">References</h1>
<ol>
<li>Wikipedia, <a href="https://en.wikipedia.org/wiki/Clebsch%E2%80%93Gordan_coefficients#Relation_to_Wigner_3-j_symbols">Clebsch-Gordon Coefficients as Wigner-3j expression</a></li>
<li>Free Software Foundation, <a href="https://www.gnu.org/software/gsl/">GNU Scientific Library</a></li>
<li>Ng et al., <a href="https://graphics.stanford.edu/papers/allfreqmat/">Triple Product Wavelet Integrals for All-Frequency Relighting</a></li>
<li>Zhou et al., <a href="https://www.microsoft.com/en-us/research/publication/precomputed-shadow-fields-for-dynamic-scenes/">Precomputed Shadow Fields for Dynamic Scenes</a></li>
<li>Pan et al., <a href="https://diglib.eg.org/handle/10.2312/CGF.v26i3pp485-493">Precomputed Radiance Transfer Field for Rendering Interreflections in Dynamic Scenes</a></li>
<li>Tobias Alexander Franke, <a href="https://github.com/thefranke/deimos/blob/master/math/spherical_harmonics.h#L344">SH rotation matrix implementation</a></li>
<li>Lehtinen and Kautz, <a href="https://mediatech.aalto.fi/~jaakko/publications/lehtinen2003i3d_paper.pdf">Matrix Radiance Transfer</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Wed, 19 Apr 2017 00:00:00 +0200
https://www.tobias-franke.eu/log/2017/04/19/triple-products.html
https://www.tobias-franke.eu/log/2017/04/19/triple-products.htmlNotesFunctionTransformMRTPRTRadianceFieldOn nomenclature
<p>Ever since the graphics field got expanded by a variety of new gadgets such as the Oculus Rift, Microsoft’s Kinect and HoloLens or the Leap Motion, and new parties moved in to expand their usage (in particular Magic Leap), I have seen <a href="https://twitter.com/search?q=%23MixedReality">a whole brigade of marketeers and fanboys</a> come with them who have simply made stuff up or <a href="https://www.thefoundry.co.uk/solutions/virtual-reality/vr-ar-mr-sorry-im-confused/">juggled around with terms</a> they are <a href="http://www.gamasutra.com/view/news/281838/Magic_Leaps_chief_game_wizard_has_big_ideas_for_the_mixed_reality_future.php">evidently confused about at best</a>.</p>
<p>The first group likes to revise or invent new terms. For instance, using a different word to hide inefficiencies in a certain method. So instead of having <em>augmenting objects</em>, Microsoft prefers to call them <em>Holograms</em>. Why? Because in a see-through display like the HoloLens, you still can’t render proper black (that is, block light from passing through the lens entirely). Better find a new word for those translucent things on the screen. Additionally there are those who come up with entirely meaningless marketing terms like Cross-Reality (XR) or, for maximum chaos, <a href="https://www.youtube.com/watch?v=DIIk89cmcsU">Merged Reality</a>.</p>
<p>The second is a consistently growing group of people who like to claim one of the following things:</p>
<ul>
<li>Mixed Reality is Augmented Reality and vice versa</li>
<li>Mixed Reality = Virtual Reality + Augmented Reality</li>
<li>Mixed Reality is a superset of Virtual Reality</li>
<li>Mixed Reality = Some new I/O device</li>
<li>Mixed Reality and Augmented Reality are unrelated</li>
<li>Augmented Reality is the opposite of Virtual Reality</li>
</ul>
<p>This group is not just simply wrong, but also responsible for the spread of mass-confusion around these terms which have been clearly defined for a long time and which have been used in many research papers.</p>
<h1 id="mixed-reality">Mixed Reality</h1>
<p>The term Mixed Reality was introduced by Paul Milgram et al. in a 1994 publication called <a href="http://etclab.mie.utoronto.ca/publication/1994/Milgram_Takemura_SPIE1994.pdf"><em>Augmented Reality: A class of displays on the reality-virtuality continuum</em></a>. It presented the idea that between virtual and real there is a spectrum of different mixtures of both.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_12_mr-continuum.png"><img src="https://www.tobias-franke.eu/layout/logcache/2016_12_mr-continuum.png" alt="" /></a><br />
<em>From Milgram et al., The Reality-Virtuality Continuum</em></p>
<p>For instance, there could be a real news-anchor inside a virtual news-room (a thing called <em>Augmented Virtuality</em>). Or a virtual object in a real setting, like Gollum in Lord of the Rings (a.k.a. <em>Augmented Reality</em>). Things don’t necessarily have to look real. The spectrum could be 2D, with one axis defining if something is added or even removed (the latter being called <em>Diminished Reality</em>). The gist of it though is that you can arbitrarily <em>lerp</em> between real and virtual with any degree to define realism.</p>
<p>The paper introduces the term <em>Mixed Reality</em> as follows:</p>
<blockquote>
<p>Within this framework it is straightforward to define a generic Mixed Reality (MR) environment as one in which real world and virtual world objects are presented together within a single display, that is, anywhere <strong>between</strong> the extrema of the RV continuum.</p>
</blockquote>
<p>This <strong>excludes</strong> two realities: For lack of a better term “Real Reality”, and the polar opposite, Virtual Reality. You don’t need a definition for the obvious though, because Mixed Reality <em>implies</em> that you are <strong>mixing</strong> realities in the first place.</p>
<p><a href="https://media.giphy.com/media/3oriNRuBrSsJp7y0iA/source.gif"><img src="https://media.giphy.com/media/3oriNRuBrSsJp7y0iA/source.gif" alt="" /></a>
<em>Four stages on the RV continuum. Courtesy <a href="https://twitter.com/vmccurley">Vincent McCurley</a></em></p>
<p>There are those who think they might come up with a clever argument that the observer/controller/player of a Virtual Reality simulation is somehow the mixed-in real element and therefore any interaction with Virtual Reality constitutes a Mixed Reality. This is of course silly, since that would include <em>people using a computer</em> and render the term meaningless. It seems that the reasoning behind this line of thinking is that there is a fancy new input device such as the Kinect or the Leap Motion which reads “real gestures”, whereas your old-school mouse or keyboard is just not magical enough.</p>
<h1 id="conclusion">Conclusion</h1>
<ul>
<li>Virtual Reality is <strong>NOT</strong> Mixed Reality or included by it</li>
<li>Virtual Reality is <strong>NOT</strong> Augmented Reality or included by it</li>
<li>Augmented Reality <strong>IS</strong> a sub-type of Mixed Reality</li>
<li>Augmented Reality is <strong>NOT</strong> the opposite of Virtual Reality</li>
<li>Mixed Reality is <strong>MORE</strong> than Augmented Reality</li>
<li>Mixed Reality is <strong>NOT</strong> a new I/O device</li>
<li>AR + VR <strong>IS</strong> meaningless</li>
</ul>
<p>Or to make it really simple: If you are not mixing two or more realities, you’re not doing Mixed Reality.</p>
<h1 id="references">References</h1>
<ol>
<li>Michael Abrash, <a href="http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-ar-anytime-soon/">Why You Won’t See Hard AR Anytime Soon</a></li>
<li>Gamasutra, <a href="http://www.gamasutra.com/view/news/281838/Magic_Leaps_chief_game_wizard_has_big_ideas_for_the_mixed_reality_future.php">Magic Leap’s ‘chief game wizard’ has big ideas for the mixed reality future</a></li>
<li>The Foundry, <a href="https://www.thefoundry.co.uk/solutions/virtual-reality/vr-ar-mr-sorry-im-confused/">VR? AR? MR? Sorry, I’m confused.</a></li>
<li>Twitter, <a href="https://twitter.com/search?q=%23MixedReality">#MixedReality</a></li>
<li>Intel, <a href="https://www.youtube.com/watch?v=DIIk89cmcsU">Merged Reality</a></li>
<li>Milgram et al., <a href="http://etclab.mie.utoronto.ca/publication/1994/Milgram_Takemura_SPIE1994.pdf">Augmented Reality: A class of displays on the reality-virtuality continuum</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Wed, 28 Dec 2016 00:00:00 +0100
https://www.tobias-franke.eu/log/2016/12/28/on_nomenclature.html
https://www.tobias-franke.eu/log/2016/12/28/on_nomenclature.htmlGeneralIntroductionPublicationThe Convolution Theorem
<h1 id="the-basis">The Basis</h1>
<p>In Linear Algebra, we’re used to build a vector $\mathbf{v}$ out of other vectors $\mathbf{v_1}, \mathbf{v_2}$.</p>
<script type="math/tex; mode=display">\begin{equation}
\mathbf{v} = \mathbf{v_1} + \mathbf{v_2}
\end{equation}</script>
<p>Each vector is, at the very least, implicitly constructed out of its basis vectors. If there is no specific basis mentioned anywhere, we assume it to be a basis of unit vectors. For instance, for a two-dimensional space these are $\mathbf{\Phi_1} = (1, 0)$ and $\mathbf{\Phi_2} = (0, 1)$, and that a vector $\mathbf{v} = (3, 2)$ is just the sum of scaled basis vectors.</p>
<script type="math/tex; mode=display">\begin{equation}
\left(
\begin{array}{c}
3\\
2
\end{array}
\right) = 3 \cdot \mathbf{\Phi_1} + 2 \cdot \mathbf{\Phi_2}
\end{equation}</script>
<p>The numbers $3$ and $2$ are the <em>coefficients</em> of the basis vectors $\mathbf{\Phi_i}$. We can call them $c_i$.</p>
<script type="math/tex; mode=display">\begin{equation}\label{vector_reconstruction_sample}
\mathbf{v} = c_1 \cdot \mathbf{\Phi_1} + c_2 \cdot \mathbf{\Phi_2}
\end{equation}</script>
<p>The same is true for functions. We can build a function $v(x)$ out of other functions $v_1(x)$ and $v_2(x)$.</p>
<script type="math/tex; mode=display">\begin{equation}
v(x) = v_1(x) + v_2(x)
\end{equation}</script>
<p>We can likewise represent one function as the sum of a set of scaled basis functions, just like we can represent one vector as a sum of scaled basis vectors.</p>
<script type="math/tex; mode=display">\begin{equation}
v(x) = c_1 \cdot \Phi_1(x) + c_2 \cdot \Phi_2(x)
\end{equation}</script>
<p>The only question is, how do we know the scaling coefficients (i.e., $c_1$ and $c_2$) to build the desired function with a set of basis functions?</p>
<h1 id="function-transforms">Function Transforms</h1>
<p>When transforming a vector $\mathbf{v}$ from one basis to another, what we do is to multiply the vector $\mathbf{v}$ by each basis vector $\mathbf{\Phi_i}$ to get the new coefficients. The multiplication operation that we do is the dot product, or more generally the <em>inner product</em> $\langle, \rangle$, a kind of matrix multiplication to project $\mathbf{v}$ onto each basis vector $\mathbf{\Phi_i}$.</p>
<script type="math/tex; mode=display">\begin{equation}
\langle \mathbf{v}, \mathbf{\Phi_i} \rangle = c_i
\end{equation}</script>
<p>This can also be written as a product of the individual vector components $v_k$ and $\Phi_{ik}$.</p>
<script type="math/tex; mode=display">\begin{equation}\label{vector_basis_transform}
\sum_k v_k \cdot \Phi_{ik} = c_i
\end{equation}</script>
<p>To reconstruct vector $\mathbf{v}$ in the new basis, we simply multiply the basis by the coefficients.</p>
<script type="math/tex; mode=display">\begin{equation}\label{vector_basis_reconstruction}
\sum_i c_i \cdot \mathbf{\Phi_i} = \mathbf{v}
\end{equation}</script>
<p>Notice how Equation $\ref{vector_basis_reconstruction}$ is just the general form of Equation $\ref{vector_reconstruction_sample}$. Transforming a <em>function</em> $v(s)$ into a set of coefficients $c_i$ for a <em>function basis</em> $\Phi_i(s)$ is almost the same process as in Equation $\ref{vector_basis_transform}$. Here we have to integrate it over the function domain $S$ with each basis function $\Phi_i$ individually.</p>
<script type="math/tex; mode=display">\begin{equation}\label{function_basis_transform}
\int_S v(s) \Phi_i(s) ds = c_i
\end{equation}</script>
<p>Note that the only difference between Equation $\ref{vector_basis_transform}$ and $\ref{function_basis_transform}$ is that in one case we sum up a discrete representation (i.e., the vector components) and in the other we have to integrate it instead. This is why we write down an <em>inner product</em> rather than a dot product, because it is <em>independent</em> of the fact that the basis is one of functions, or vectors or anything else.</p>
<script type="math/tex; mode=display">\begin{equation}
\langle v, \Phi_i \rangle = c_i
\end{equation}</script>
<p>Reconstructing the original function works just as in Equation $\ref{vector_basis_reconstruction}$.</p>
<script type="math/tex; mode=display">\begin{equation}\label{function_basis_reconstruction}
\sum_i c_i \cdot \Phi_i(s) = v(s)
\end{equation}</script>
<h1 id="function-basis-properties">Function Basis Properties</h1>
<p>With vectors, we can use the inner product (that is, a dot product) to determine some properties about their relationship. For instance, we can tell from the projection whether or not vectors $\mathbf{\Phi_i}$ and $\mathbf{\Phi_j}$ are orthonormal to one another.</p>
<script type="math/tex; mode=display">\begin{equation}\label{dirac_delta}
c = \langle \mathbf{\Phi_i}, \mathbf{\Phi_j} \rangle = \mathbf{\Phi_i} \cdot \mathbf{\Phi_j} = \delta_{ij} =
\left\{
\begin{array}{c}
1, i = j \\
0, i \neq j
\end{array}
\right.
\end{equation}</script>
<p>If the coefficient $c$ is $0$, then both vectors $\mathbf{\Phi_i}$ and $\mathbf{\Phi_j}$ are orthogonal to each other. Additionally if $\mathbf{\Phi_i}$ and $\mathbf{\Phi_j}$ happen to be identical and $c$ turns out to be $1$ and not an arbitrary number $n$, then we know that both vectors also form an orthonormal basis. $\delta_{ij}$ is called <em>Kronecker Delta</em> and is usually just a shorthand of such a basis behavior. With functions, the same concept is true, and the formula is identical. Instead of a dot product, the inner product for two functions $\Phi_i(s)$ and $\Phi_j(s)$ represents an integration. But again, if $c$ turns out to be $0$, we know both functions are orthogonal to each other, and if every other result is $1$, then the function basis is orthonormal.</p>
<script type="math/tex; mode=display">\begin{equation}\label{dirac_delta_functions}
c = \langle \Phi_i, \Phi_j \rangle = \int_S \Phi_i(s) \cdot \Phi_j(s) ds = \delta_{ij} =
\left\{
\begin{array}{c}
1, i = j \\
0, i \neq j
\end{array}
\right.
\end{equation}</script>
<p>We now have all the tools necessary to define properties of a function space: Equation $\ref{dirac_delta_functions}$ can be used to define orthogonality and orthonormality of two functions, and $\ref{function_basis_reconstruction}$ can be used to define linear dependency (one function $\Phi_i$ is a sum of scalar multiples of other functions $\Phi_j$ of the same basis) and therefore also the rank and determinant of a function basis.</p>
<h1 id="the-convolution-theorem">The Convolution Theorem</h1>
<p>Now that we know how to transform a function into coefficients of a function basis, and how to test whether or not a function basis has certain properties like orthonormality, let’s perform a magic trick. Imagine we have an <em>orthonormal</em> basis $\Phi_i$, i.e. two functions of that basis will always integrate to $1$ if they are identical, or $0$ if they are not.</p>
<p>We can represent any function $f(s)$ by a bunch of coefficients in that basis. Imagine now that we have two functions: $f(s)$ and $g(s)$.</p>
<script type="math/tex; mode=display">\begin{eqnarray}
f(s) = \sum_i f_i \cdot \Phi_i(s) \label{f_reconstruction}\\
g(s) = \sum_j g_j \cdot \Phi_j(s) \label{g_reconstruction}
\end{eqnarray}</script>
<p>Let’s multiply them together and integrate the result. We can replace the functions with the sum in Equation $\ref{f_reconstruction}$ and $\ref{g_reconstruction}$.</p>
<script type="math/tex; mode=display">\begin{equation}\label{convolution_step1}
\int_S f(s) \cdot g(s) ds = \int_S \left( \sum_i f_i \cdot \Phi_i(s) \right) \cdot \left( \sum_j g_j \cdot \Phi_j(s) \right)ds
\end{equation}</script>
<p>In Equation $\ref{convolution_step1}$ we can reorder some things because everything is linear: We can move the two sums $\sum_i$ and $\sum_j$ out of the integral, together with their coefficients, because they do not depend on $s$.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
\int_S f(s) \cdot g(s) ds & = & \int_S \left( \sum_i f_i \cdot \Phi_i(s) \right) \cdot \left( \sum_j g_j \cdot \Phi_j(s) \right)ds \\
& = & \sum_i \sum_j f_i \cdot g_j \cdot \int_S \Phi_i(s) \cdot \Phi_j(s) ds \label{convolution_step2}
\end{eqnarray} %]]></script>
<p>However, we just noticed in the previous section that a function basis can be orthonormal. An orthonormal function basis is very nice to have, because the integrated product of the function basis vectors (i.e., the inner product of two functions) will be either $1$ or $0$, so the integral in Equation $\ref{convolution_step2}$ <em>can be replaced with the Kronecker Delta from Equation $\ref{dirac_delta_functions}$!</em></p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
\int_S f(s) \cdot g(s) ds & = & \int_S \left( \sum_i f_i \cdot \Phi_i(s) \right) \cdot \left( \sum_j g_j \cdot \Phi_j(s) \right)ds \\
& = & \sum_i \sum_j f_i \cdot g_j \cdot \int_S \Phi_i(s) \cdot \Phi_j(s) ds \\
& = & \sum_i \sum_j f_i \cdot g_j \cdot \delta_{ij} \label{convolution_step3}
\end{eqnarray} %]]></script>
<p>But if we have a Kronecker Delta, that just means that every time $i$ and $j$ are not equal, the product is just $0$. So we can simplify even further and remove one variable.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
\int_S f(s) \cdot g(s) ds & = & \int_S \left( \sum_i f_i \cdot \Phi_i(s) \right) \cdot \left( \sum_j g_j \cdot \Phi_j(s) \right)ds \\
& = & \sum_i \sum_j f_i \cdot g_j \cdot \int_S \Phi_i(s) \cdot \Phi_j(s) ds \\
& = & \sum_i \sum_j f_i \cdot g_j \cdot \delta_{ij} \\
& = & \sum_i f_i \cdot g_i \label{convolution_step4}
\end{eqnarray} %]]></script>
<p>What is this? Equation $\ref{convolution_step4}$ looks like a dot product. So in essence that means <em>we can shortcut an integration of a product of two functions $f(s)$ and $g(s)$ by a dot product of their coefficients $f_i$ and $g_i$ if the function basis is orthonormal!</em></p>
<p>Why is this important?</p>
<h1 id="filtering">Filtering</h1>
<p>Filter operations always take on a formula just like Equation $\ref{convolution_step1}$. For instance, a Gauss Filter is a Gauss function that is, in a small window, multiplied with another function. One can do that in a pixel-based domain, where each pixel is multiplied with a value from the Gauss function at the same position, but it is really just two functions multiplied together.</p>
<p>However, if one part of the function is already available in a harmonic basis function, for instance as a JPEG image, then we can apply the filter in the function basis directly and do an operation which would normally be very expensive with a cheap dot product! If the number of coefficients is smaller than the operations necessary on the original domain $S$ of the function (that is, the number of pixels we need to sum up), we can save a tremendous amount of computation time.</p>
<h1 id="precomputed-radiance-transfer">Precomputed Radiance Transfer</h1>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_prt.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_prt.jpg" alt="" /></a><br />
<em>Transfer function visualized. Every white pixel in this $360^{\circ}$ image is one which, evaluated by a raytracer, was blocked by some geometry. All other pixels represent values where a ray shot from this point could travel freely away from all geometry.</em></p>
<p>A main observation of Precomputed Radiance Transfer is the following: One can simplify the rendering equation to something we have seen above by throwing out self-emittance, and unify everything in the integral that isn’t incoming light into one homogeneous diffuse <em>transfer function</em> $T(\mathbf{\omega_i})$.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
L(\mathbf{\omega_o}) & = & L_e(\mathbf{\omega_o}) + \int_\Omega L(\mathbf{\omega_i}) \cdot f(x, \mathbf{\omega_i}, \mathbf{\omega_o}) \cdot \langle \mathbf{n}, \mathbf{\omega_i} \rangle d\mathbf{\omega_i}
\end{eqnarray} %]]></script>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}\label{prt_integral}
& \rightarrow & \int_\Omega L(\mathbf{\omega_i}) \cdot T(\mathbf{\omega_i}) d\mathbf{\omega_i}
\end{eqnarray} %]]></script>
<p>The transfer function is simply everything - materials, visibility, Lambert factor etc. - packed into a single function. A simple visualization is in the above figure, where the transfer function of a point shows a visibility function. Imagine now that next to a transfer function, we also have a function representing our environment light. The crucial observation is that Equation $\ref{prt_integral}$ looks just like Equation $\ref{convolution_step1}$, and this means that if we can get both as coefficients of some basis function, we can compute the integral in Equation $\ref{prt_integral}$ with just one simple dot product of their coefficients!</p>
<p>Why is this practical? Getting the coefficients is really expensive, so before we can do this we would need to compute all transfer functions (probably with a raytracer) which is already very costly, and then do the Monte Carlo estimation to get their coefficients. This is way too expensive to do in real-time. <em>But</em>, if the car in the figure never moves, then the transfer function never changes! So if we compute all transfer in an offline step and save it (one might almost be tempted to call this <em><a href="http://www.cs.jhu.edu/~misha/ReadingSeminar/Papers/Sloan02.pdf">pre-computing the transfer</a></em>), then we only need to get the coefficients for the light at runtime, and that should be fairly easy to do. In fact, if we only have a bunch of different lighting configurations, we can precompute the coefficient vectors for all of them, and at runtime when switching between skyboxes fetch the respective coefficients.</p>
<h1 id="references">References</h1>
<ol>
<li>Sloan et al, <a href="http://www.cs.jhu.edu/~misha/ReadingSeminar/Papers/Sloan02.pdf">Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low- Frequency Lighting Environments. Peter-Pike</a></li>
<li>Robin Green, <a href="http://silviojemma.com/public/papers/lighting/spherical-harmonic-lighting.pdf">Spherical Harmonic Lighting: The Gritty Details</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Tue, 18 Oct 2016 00:00:00 +0200
https://www.tobias-franke.eu/log/2016/10/18/the_convolution_theorem.html
https://www.tobias-franke.eu/log/2016/10/18/the_convolution_theorem.htmlNotesPRTFunctionTransformDifferential Rendering
<p>Every graphics sub-discipline has its own holy grail, some go-to paper that everyone must have read to get into the necessary basics of the field: Physically Based Rendering has <a href="http://www.pbrt.org/">a book by the same name</a>, Path Tracing has <a href="http://graphics.stanford.edu/papers/veach_thesis/">Eric Veach’s dissertation</a>, real-time global illumination has <a href="http://dl.acm.org/citation.cfm?id=1053460">Reflective Shadow Maps</a>.</p>
<p>For Augmented Reality rendering, this go-to publication is <a href="http://www.pauldebevec.com/Research/IBL/">Paul Debevec’s <em>Rendering Synthetic Objects into Real Scenes</em></a>.</p>
<h1 id="fusing-two-realities">Fusing two realities</h1>
<p>Imagine the following situation: You have a green-ish surface, and you want to project a virtual object on top of it, like this Stanford Buddha.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-1.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-1.jpg" alt="" /></a></p>
<p>Just as other real objects on this surface would do, this Stanford Buddha should also drop a shadow onto it. To do this, you figure out the position of a real light source (let’s pretend the room in which this shot was taken has only one light), and put a virtual copy of it into the virtual scene at the same relative location.</p>
<p>What now? To render a proper shadow, we also need a surface to drop the shadow onto. We can use a green-ish virtual surface as a placeholder for the green-ish looking real one to receive the shadow. Let’s give it a try.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-2.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-2.jpg" alt="" /></a></p>
<p>But how do we now fuse this image with the real background? Do we just paste it over the image?</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-3.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-3.jpg" alt="" /></a></p>
<p>That doesn’t look quite right. So far we only had a vague interpretation of what the underlying real surface looks like: Green-ish. The description is not very accurate, and the result shows.</p>
<p>Maybe we could reconstruct the real surface more accurately and then get a more correct looking image. For instance, we could texture the so far green-ish placeholder with an image of the real surface. Of course from that texture we need to carefully remove all the interfering lights first (a process sometimes called <em>delighting</em>) so we’re really left with just the albedo. Assume we did this correctly and now have this texture ready. Just put it on top of the surface and try again.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-4.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-4.jpg" alt="" /></a>
<a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-5.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-5.jpg" alt="" /></a></p>
<p>It looks better, but now we run into other problems:</p>
<ol>
<li>The scaling of the virtual plane and its texture is slightly off, which makes it look a bit blurry.</li>
<li>The tracker didn’t reconstruct the real world position of this patch down to the nanometer, so we end up with some very annoying seams around the border.</li>
<li>The renderer uses a point light to model the real world light source. You can see the falloff toward the back of the plane, where it is darker than it should be. The real light source in that room is a large neon tube and incident light spreads much more equally across the surface.</li>
</ol>
<p>Also, it looks like there is some type of coloring issue. Maybe it’s a gamma problem? Is the reconstructed albedo texture really correct? Maybe we need a path-tracer? Is there a red bounce missing from the plane in the back? Maybe this, maybe that, maybe maybe maybe…</p>
<p>Early Augmented Reality applications in the 90s dealt with this <strong>fusion-problem</strong> in various ways:</p>
<ul>
<li>Just leave it as it is.</li>
<li>If you need shadows, just make the pixels darker. Even if there are multiple non-white lights in the room no one will notice.</li>
<li>Reconstruct the entire object rather than just a patch of it. At least then there won’t be any visible seams or other texture-related differences.</li>
<li>Blend the partly reconstructed surface with the real background image.</li>
<li>Why invest so much hard work rendering correctly when you can make the problem space much easier? Just use simpler real world objects! For instance a diffuse, homogeneous green surface with no texture at all.</li>
</ul>
<h1 id="differential-rendering">Differential Rendering</h1>
<p>While all of these tactics <em>can</em> work, they are unsatisfying at best and often require manual tweaking to make just that one scene look good.</p>
<p>The main weakness in the above scenarios is this: They rely on the placeholder to represent reality as close as possible. Could we perhaps get rid of the placeholder entirely and just be left with the actual <strong>change</strong> that the Buddha introduces to the real scene (i.e., the shadow)?</p>
<p>This is the question that <em>Differential Rendering</em> deals with. The idea is simple: Extract the difference introduced by the object and leave out the rest. How? Let’s look at it again. We have a patch of the green floor that is supposed to be the a reconstruction of the real world. If we tag all the pixels of a reconstructed surface in a mask, we can isolate the green placeholder.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-6.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-6.jpg" alt="" /></a></p>
<p>We can render the same patch again, but without the object, and repeat the same process of masking out the result. Subtracting both images from one another will leave us with only <strong>the difference</strong> due to the objects presence in one of the scenes.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-7.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-7.jpg" alt="" /></a></p>
<p>The shadowed area will produce <em>negative</em> energy, as there is less energy present on the augmented patch than before, while the rest of the image will just be a zero-sum. If we add this difference back to our very first image, the shadow ends up in the real world and looks like it belongs there.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-8.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2016_10_diff-8.jpg" alt="" /></a></p>
<h1 id="conclusion">Conclusion</h1>
<p>The main strength of Differential Rendering is its simplicity, as it boils down to a post-processing effect (you can easily repeat the above steps with Gimp, Krita or Photoshop) to extract the influence a virtual object has onto its surrounding, such as blocking real light, bouncing off reflections or transmitting light. This difference is then simply added onto a real background image together with the object. The viewer is left with a sense that the object is actually <em>in</em> the scene rather than being glued on top of it.</p>
<p>As a side note, the extracted difference contains positive contributions (such as bounce light) as well as negative ones (the shadows subtract light from the background). The latter is a quantity called <em>antiradiance</em> and has been discussed in a few publications, most notable <a href="https://www-sop.inria.fr/reves/Basilic/2007/DSDD07/">Implicit Visibility and Antiradiance for Interactive Global Illumination</a>. I will come back to this some other time.</p>
<p>More importantly, if the renderer is able to compute global illumination, the entire extracted difference should integrate to zero: The object is merely “re-routing” light to a different location, and therefore all energy is conserved.</p>
<h1 id="references">References</h1>
<ol>
<li>Paul Debevec, <a href="http://www.pauldebevec.com/Research/IBL/">Rendering Synthetic Objects into Real Scenes</a></li>
<li>Dachsbacher et al, <a href="https://www-sop.inria.fr/reves/Basilic/2007/DSDD07/">Implicit Visibility and Antiradiance for Interactive Global Illumination</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Tue, 04 Oct 2016 00:00:00 +0200
https://www.tobias-franke.eu/log/2016/10/04/differential-rendering.html
https://www.tobias-franke.eu/log/2016/10/04/differential-rendering.htmlGeneralRelightingPublicationSIGGRAPH 2014
<p>I was at <a href="http://s2014.siggraph.org">SIGGRAPH 2014</a> to present, among <a href="http://www.tobias-franke.eu/publications/hermanns14ssct/">other things</a>, <a href="http://www.tobias-franke.eu/publications/franke14irars/">a poster of a new relighting method using Voxel Cone Tracing</a>. My schedule was as per usual completely overloaded. Nevertheless, here are some impressions.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2014_09_calendar.png"><img src="https://www.tobias-franke.eu/layout/logcache/2014_09_calendar.png" alt="" /></a></p>
<p>My favorite part about the conference are the courses, specificaly these three:</p>
<ul>
<li><a href="http://advances.realtimerendering.com/s2014/index.html">Advances in Real-Time Rendering in Games</a></li>
<li><a href="http://blog.selfshadow.com/publications/s2014-shading-course/">Physically Based Shading in Theory and Practice</a></li>
<li><a href="http://cgg.mff.cuni.cz/~jaroslav/papers/2014-ltscourse/">Recent Advances in Light Transport Simulation</a></li>
</ul>
<p>These are great synchronization points for me. In <em>Advances in Real-Time Rendering</em> I especially enjoyed three talks: Bart Wronskis (<a href="https://twitter.com/BartWronsk">@BartWronsk</a>) Volumetric Fog method for Assassin’s Creed 4 and two talks about SSR from Eidos and Guerilla Games, which is currently a thing I’m looking into to simulate low-cost reflections in AR. Also, Wade Brainerd (<a href="https://twitter.com/wadetb">@wadetb</a>) gave a great talk which was running completely inside the Call of Duty engine, which is I think a perfect way to show case what you have live on the “slides”.</p>
<p><em>Physically Based Shading</em> had the usual very informative introduction by Naty Hoffman (<a href="https://twitter.com/renderwonk">@renderwonk</a>) (here are the expanding <a href="http://blog.selfshadow.com/publications/s2013-shading-course/hoffman/s2013_pbs_physics_math_notes.pdf">course notes from last year</a>), followed by an important in-depth talk from Eric Heitz about the masking function in microfacet BRDFs. One of the fun talks was delivered at the end by Pixar, which I guess was a clear demonstration on how artists abuse Renderman to circumvent everything phyiscally-based that was introduced by the programmers in the first place.</p>
<p>Although I’m not (yet) into path-tracing, I regularily visit the <em>Light Transport</em> session to delve into the general sampling and filtering madness that dominates the entire subfield. The course usually starts with a nice introduction into path integral forumlations and then expands on papers which show different methods to cope with the noise. One presentation that I found interessting by Ondre Karlik presented the <a href="https://corona-renderer.com/">Corona Renderer</a>, a project with a focus on being more easy to use, with less but more intuitive parameters to manipulate.</p>
<p>The rest of the conference, such as the material of the paper sessions <em>Fast Rendering</em> and <em>Light Transport</em>, unfortunately still rests on my backlog.</p>
<p>I was missing the Geek Bar this year. This place comes in handy when you have multiple sessions overlapping the same time slot, which happens <em>every</em> time for GI/rendering talks. The Geek Bar streams multiple sessions into the same room, and you can switch around channels with your headset.</p>
<p>Vancouver as a location was pretty nice, had some good party locations and was a welcome change to LA. You can see some photos I took <a href="https://plus.google.com/photos/111436173885363813780/albums/6055335398577194193">right here</a>.</p>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Mon, 08 Sep 2014 00:00:00 +0200
https://www.tobias-franke.eu/log/2014/09/08/siggraph-2014.html
https://www.tobias-franke.eu/log/2014/09/08/siggraph-2014.htmlConferenceNotesOn publication videos
<p>I have reviewed my fair share of papers over the years and seen some heinous crimes in video editing for the customary media attachement: blurry, too large, strange codecs only recognized by VLC, mouse cursor and other GUI elements visible on screen, double letterboxing or outright <em>interesting</em> choices of aspect ratio. Several factors play into this, such as the usual last minute pressure when compiling a paper, the <em>oh-gawd-we-surpassed-the-maximum-attachement-size</em> limitations of most submission systems or that the constant recompression just takes too long and the authors are happy with what they got.</p>
<p><img src="https://www.tobias-franke.eu/layout/logcache/2014_07_moarjpeg.jpg" alt="" /></p>
<p>It is clear however that the cause for this confusion is that video encoding is firmly situated at the borderline to black magic. There are certainly tools like iMovie which tremendously help to overcome most troubles in standard editing and encoding, but somewhere in between incompatible codecs and file containers, video bitrates, quantization and scaling issues, frame re-timing, color compression, and total boredom because you <em>just wanted a video</em> everyone will give up sooner or later.</p>
<p>In this post I’m going to cover the topic of how to create a web-browser compatible, pixel-matching HD or Full-HD, H.264 compressed MP4 of a reasonable size from a recording of a graphics application. But first, we need some ground rules.</p>
<h1 id="getting-it-right">Getting it right</h1>
<p>The first commandment, from which all others are derived, is that you shall absolutely and unequivocally avoid re-encoding your video multiple times. With each re-encoding, the quality of your video will degenrate into the image above. This is <em>especially</em> true if you know there will be yet another re-encoding step ahead where the settings are beyond your reach, for instance when uploading to Youtube: in this case a mostly unprocessed video is your best option.</p>
<p>These days the world has come down to some fairly standard aspect ratios: 16:9 for movies, 16:10 for displays. If you are recording 4:3 videos you’re doing it wrong! Because we’re creating a movie for the web, you may end up uploading it to various streaming platforms such as Youtube and Vimeo, which only deal with 16:9 ratios, so let’s just stick with that.</p>
<p>On the resolution front of these ratios the choices have been reduced to 720p (1280x720 pixels) and 1080p (1920x1080 pixels). We’ll most likely see new standards emerge for the upcoming 4k madness. In order to get a nice video file at the end, make sure that whatever you intend to record comes in either one of those two resolutions.</p>
<p>Not going for a standard aspect ratio and a standard resolution will force editing tools to rescale your video, causing sampling issues which will lead to blurry image quality. Sometimes you can’t avoid having a resolution that isn’t 720p or 1080p (capturing webcam videos for instance), but if you control the output just stick with the default. Videos which do not conform to either aspect ratio or resolution should be cropped and scaled manually with an external tool so that you control the quality and output instead of getting lucky with whatever your editing tool does to those videos.</p>
<p>Finally, all your videos should run with the same frame rate! If they don’t you either have to re-time everything to the lowest common denominator (which doesn’t require anything special but may not be great), or magically re-construct the missing frames with some optical-flow tool like Apple Cinema Tools or Twixtor (a lot of manual work ahead).</p>
<h1 id="dumping-data">Dumping data</h1>
<p>A popular way to obtain videos of a D3D or GL application are hooks like Fraps, NVIDIA ShadowPlay or DXTory. On Mac OS X, you can use Quicktime (File -> New Screen Recording) to record portions of the screen. If you haven’t done so already, check out a demo of one of them and try to record from a running game. An important decision has to be made upfront about resource division: record uncompressed videos for maximum quality and speed at very high data rates which can easily fill even the largest of hard disks, or sacrifice potentially vital CPU time to compress live during recording. There is no easy answer to this and it depends on your preferences and hardware settings. Personally, I prefer to record uncompressed and later downsize the videos into smaller, compressed segments for archiving. This certainly is no option to record World of Warcraft Boss fights of 10 minutes and more though.</p>
<p>I haven’t played around with Fraps too much and <a href="http://blog.mmacklin.com/2013/06/11/real-time-video-capture-with-ffmpeg/">recently found a way to completely avoid it</a> by piping raw RGBA frames from my D3D app <a href="http://www.ffmpeg.org/">directly into FFmpeg</a>. You may use the following class to do the same.</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="cp">#include <iomanip>
#include <ctime>
#include <sstream>
</span>
<span class="cp">#include <D3D11.h>
</span>
<span class="cp">#define tstringstream std::wstringstream
#define tcerr std::wcerr
#define tclog std::wclog
</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">T</span><span class="o">></span>
<span class="kt">void</span> <span class="n">safe_release</span><span class="p">(</span><span class="n">T</span><span class="o">&</span> <span class="n">obj</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">obj</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">obj</span><span class="o">-></span><span class="n">Release</span><span class="p">();</span>
<span class="n">obj</span> <span class="o">=</span> <span class="nb">nullptr</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">class</span> <span class="nc">video_recorder</span>
<span class="p">{</span>
<span class="k">protected</span><span class="o">:</span>
<span class="n">UINT</span> <span class="n">width_</span><span class="p">,</span>
<span class="n">height_</span><span class="p">,</span>
<span class="n">fps_</span><span class="p">;</span>
<span class="n">ID3D11Texture2D</span><span class="o">*</span> <span class="n">ffmpeg_texture_</span><span class="p">;</span>
<span class="kt">FILE</span><span class="o">*</span> <span class="n">ffmpeg_</span><span class="p">;</span>
<span class="n">tstring</span> <span class="n">path_</span><span class="p">;</span>
<span class="k">public</span><span class="o">:</span>
<span class="kt">void</span> <span class="n">create</span><span class="p">(</span><span class="n">ID3D11Device</span><span class="o">*</span> <span class="n">device</span><span class="p">,</span> <span class="n">UINT</span> <span class="n">width</span><span class="p">,</span> <span class="n">UINT</span> <span class="n">height</span><span class="p">,</span> <span class="n">UINT</span> <span class="n">fps</span><span class="p">,</span> <span class="k">const</span> <span class="n">tstring</span><span class="o">&</span> <span class="n">path</span> <span class="o">=</span> <span class="s">L"../../data"</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">width_</span> <span class="o">=</span> <span class="n">width</span><span class="p">;</span>
<span class="n">height_</span> <span class="o">=</span> <span class="n">height</span><span class="p">;</span>
<span class="n">fps_</span> <span class="o">=</span> <span class="n">fps</span><span class="p">;</span>
<span class="n">path_</span> <span class="o">=</span> <span class="n">path</span><span class="p">;</span>
<span class="n">D3D11_TEXTURE2D_DESC</span> <span class="n">desc</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">Width</span> <span class="o">=</span> <span class="k">static_cast</span><span class="o"><</span><span class="n">UINT</span><span class="o">></span><span class="p">(</span><span class="n">width</span><span class="p">);</span>
<span class="n">desc</span><span class="p">.</span><span class="n">Height</span> <span class="o">=</span> <span class="k">static_cast</span><span class="o"><</span><span class="n">UINT</span><span class="o">></span><span class="p">(</span><span class="n">height</span><span class="p">);</span>
<span class="n">desc</span><span class="p">.</span><span class="n">MipLevels</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">ArraySize</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">SampleDesc</span><span class="p">.</span><span class="n">Count</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">SampleDesc</span><span class="p">.</span><span class="n">Quality</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">Format</span> <span class="o">=</span> <span class="n">DXGI_FORMAT_R8G8B8A8_UNORM</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">Usage</span> <span class="o">=</span> <span class="n">D3D11_USAGE_STAGING</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">BindFlags</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">CPUAccessFlags</span> <span class="o">=</span> <span class="n">D3D11_CPU_ACCESS_READ</span><span class="p">;</span>
<span class="n">desc</span><span class="p">.</span><span class="n">MiscFlags</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">device</span><span class="o">-></span><span class="n">CreateTexture2D</span><span class="p">(</span><span class="o">&</span><span class="n">desc</span><span class="p">,</span> <span class="nb">nullptr</span><span class="p">,</span> <span class="o">&</span><span class="n">ffmpeg_texture_</span><span class="p">)</span> <span class="o">!=</span> <span class="n">S_OK</span><span class="p">)</span>
<span class="n">tcerr</span> <span class="o"><<</span> <span class="s">L"Failed to create staging texture for recording"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="n">start_recording</span><span class="p">(</span><span class="kt">bool</span> <span class="n">compressed</span> <span class="o">=</span> <span class="nb">false</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">ffmpeg_texture_</span><span class="p">)</span>
<span class="k">return</span><span class="p">;</span>
<span class="n">std</span><span class="o">::</span><span class="kt">time_t</span> <span class="n">t</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">time</span><span class="p">(</span><span class="nb">nullptr</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">tm</span> <span class="n">tm</span><span class="p">;</span>
<span class="n">localtime_s</span><span class="p">(</span><span class="o">&</span><span class="n">tm</span><span class="p">,</span> <span class="o">&</span><span class="n">t</span><span class="p">);</span>
<span class="n">tstringstream</span> <span class="n">file</span><span class="p">;</span>
<span class="n">file</span> <span class="o"><<</span> <span class="n">path_</span> <span class="o"><<</span> <span class="s">"/record-"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">put_time</span><span class="p">(</span><span class="o">&</span><span class="n">tm</span><span class="p">,</span> <span class="s">L"%Y%m%d-%H%M%S"</span><span class="p">)</span> <span class="o"><<</span> <span class="s">L".mp4"</span><span class="p">;</span>
<span class="c1">// adapted from http://blog.mmacklin.com/2013/06/11/real-time-video-capture-with-ffmpeg/</span>
<span class="n">tstringstream</span> <span class="n">cmd</span><span class="p">;</span>
<span class="n">cmd</span> <span class="o"><<</span> <span class="s">L"ffmpeg -r "</span> <span class="o"><<</span> <span class="n">fps_</span> <span class="o"><<</span> <span class="s">" -f rawvideo -pix_fmt rgba "</span>
<span class="o"><<</span> <span class="s">L"-s "</span> <span class="o"><<</span> <span class="n">width_</span> <span class="o"><<</span> <span class="s">"x"</span> <span class="o"><<</span> <span class="n">height_</span> <span class="o"><<</span> <span class="s">" "</span>
<span class="o"><<</span> <span class="s">L"-i - "</span>
<span class="o"><<</span> <span class="s">L"-threads 2 -y "</span>
<span class="o"><<</span> <span class="s">L"-c:v libx264 "</span>
<span class="o"><<</span> <span class="p">(</span><span class="n">compressed</span><span class="p">)</span> <span class="o">?</span> <span class="s">L"-preset ultrafast -qp 0 "</span> <span class="o">:</span> <span class="s">L"-preset fast "</span>
<span class="o"><<</span> <span class="n">make_absolute_path</span><span class="p">(</span><span class="n">file</span><span class="p">.</span><span class="n">str</span><span class="p">())</span>
<span class="p">;</span>
<span class="n">tclog</span> <span class="o"><<</span> <span class="s">L"Recording video with: "</span> <span class="o"><<</span> <span class="n">cmd</span><span class="p">.</span><span class="n">str</span><span class="p">()</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="cp">#ifdef UNICODE
</span> <span class="n">ffmpeg_</span> <span class="o">=</span> <span class="n">_wpopen</span><span class="p">(</span><span class="n">cmd</span><span class="p">.</span><span class="n">str</span><span class="p">().</span><span class="n">c_str</span><span class="p">(),</span> <span class="s">L"wb"</span><span class="p">);</span>
<span class="cp">#else
</span> <span class="n">ffmpeg_</span> <span class="o">=</span> <span class="n">_popen</span><span class="p">(</span><span class="n">cmd</span><span class="p">.</span><span class="n">str</span><span class="p">().</span><span class="n">c_str</span><span class="p">(),</span> <span class="s">"wb"</span><span class="p">);</span>
<span class="cp">#endif
</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">ffmpeg_</span><span class="p">)</span>
<span class="n">tcerr</span> <span class="o"><<</span> <span class="s">L"Failed to initialize ffmpeg"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="n">stop_recording</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">ffmpeg_</span><span class="p">)</span>
<span class="n">_pclose</span><span class="p">(</span><span class="n">ffmpeg_</span><span class="p">);</span>
<span class="n">ffmpeg_</span> <span class="o">=</span> <span class="nb">nullptr</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="n">add_frame</span><span class="p">(</span><span class="n">ID3D11DeviceContext</span><span class="o">*</span> <span class="n">context</span><span class="p">,</span> <span class="n">ID3D11RenderTargetView</span><span class="o">*</span> <span class="n">rtv</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ID3D11Resource</span><span class="o">*</span> <span class="n">resource</span><span class="p">;</span>
<span class="n">rtv</span><span class="o">-></span><span class="n">GetResource</span><span class="p">(</span><span class="o">&</span><span class="n">resource</span><span class="p">);</span>
<span class="n">add_frame</span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">resource</span><span class="p">);</span>
<span class="n">safe_release</span><span class="p">(</span><span class="n">resource</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="n">add_frame</span><span class="p">(</span><span class="n">ID3D11DeviceContext</span><span class="o">*</span> <span class="n">context</span><span class="p">,</span> <span class="n">ID3D11Resource</span><span class="o">*</span> <span class="n">resource</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">context</span><span class="o">-></span><span class="n">CopyResource</span><span class="p">(</span><span class="n">ffmpeg_texture_</span><span class="p">,</span> <span class="n">resource</span><span class="p">);</span>
<span class="n">D3D11_MAPPED_SUBRESOURCE</span> <span class="n">msr</span><span class="p">;</span>
<span class="n">UINT</span> <span class="n">subresource</span> <span class="o">=</span> <span class="n">D3D11CalcSubresource</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="n">context</span><span class="o">-></span><span class="n">Map</span><span class="p">(</span><span class="n">ffmpeg_texture_</span><span class="p">,</span> <span class="n">subresource</span><span class="p">,</span> <span class="n">D3D11_MAP_READ</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="o">&</span><span class="n">msr</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">msr</span><span class="p">.</span><span class="n">pData</span> <span class="o">&&</span> <span class="n">ffmpeg_</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">fwrite</span><span class="p">(</span><span class="n">msr</span><span class="p">.</span><span class="n">pData</span><span class="p">,</span> <span class="p">(</span><span class="n">width_</span> <span class="o">*</span> <span class="n">height_</span> <span class="o">*</span> <span class="mi">4</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="n">ffmpeg_</span><span class="p">);</span>
<span class="n">context</span><span class="o">-></span><span class="n">Unmap</span><span class="p">(</span><span class="n">ffmpeg_texture_</span><span class="p">,</span> <span class="n">subresource</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="n">destroy</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">stop_recording</span><span class="p">();</span>
<span class="n">safe_release</span><span class="p">(</span><span class="n">ffmpeg_texture_</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span></code></pre></figure>
<p>Essentially the <em>video_encoder</em> class creates an open pipe to ffmpeg using the libx264 codec with zero compression. <strong>add_frame()</strong> copies a resource behind a RenderTargetView to a staging texture, which is then CPU read and fed to the pipe.</p>
<p>The crucial line is where the cmd stringstream is compiled: here you may want toggle <strong>compressed</strong> to have direct compression enabled and choose custom settings for <em>-qp</em> and <em>-crf</em> to control the compression rate. Either way, be warned that the resulting video is <strong>NOT</strong> compatible with MP4 support in browsers, because most of them only play content in YUV 4:2:0 color space and these settings output RGBA. I don’t have access to D3D11.1 where the staging texture can have <strong>DXGI_FORMAT_NV12</strong>, but if you do you may want to try this.</p>
<p>If you need maximum performance though, save the video uncompressed and compress it afterwards.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>ffmpeg <span class="nt">-i</span> record-xxxx-xxxx.mp4 <span class="nt">-c</span>:v libx264<span class="se">\</span>
<span class="nt">-crf</span> 23 out.mp4</code></pre></figure>
<p>This will give you a nice, compressed MP4 which modern browser can directly display without plugins. You can upload it to your webpage and embed the video with an HTML5 video tag or put it on a Dropbox folder and people can directly watch it. If you plan to cut and edit the video though, skip this step.</p>
<p>Embedding videos into a webpage is easily done with the following code.</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><video</span> <span class="na">style=</span><span class="s">"width:100%"</span> <span class="na">controls</span> <span class="na">autoplay</span><span class="nt">></span>
<span class="nt"><source</span> <span class="na">src=</span><span class="s">"http://foo/bar.mp4"</span> <span class="na">type=</span><span class="s">"video/mp4"</span><span class="nt">></span>
Your browser does not support the video tag.
<span class="nt"></video></span></code></pre></figure>
<p>The video should run on any modern browser without any plugins.</p>
<h1 id="editing">Editing</h1>
<p>When it comes to video editing my weapon of choice is Final Cut 7, which I’ll use to illustrate an important point: most if not all bigger editing tools use a special codec for cutting a video. Apple tools such as FCP and iMovie use the ProRes codec, which comes in several forms (Proxy, LT, HQ). If you value your time, don’t want your editing tool to arbitrarily recompress your snippets and maintain full control consider re-encoding your raw FFmpeg dump manually with such a codec. In case of FCP, this will also allow you to use blending operations and other effects without FCPs further interference.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2014_07_mpegstreamclip.png"><img src="https://www.tobias-franke.eu/layout/logcache/2014_07_mpegstreamclip.png" alt="" /></a></p>
<p>At this point I cannot overstate <a href="http://www.squared5.com/">the magnificent gloriosity of MPEGStreamClip</a>: it’s an easy to use free frontend for the Quicktime compressor on Windows and Mac OS X and it has a GUI for batch processing. I usually dump all my videos into the batch list and encode them with <em>Apple ProRes 422 (HQ)</em> at 100% quality. You can also use FFmpeg and simply script this behavior if you’re good with the command line. If you have a funny resolution, MPEGStreamClip also has some options to crop and scale the video. A very similar aweseome tool is <a href="http://handbrake.fr/">Handbrake</a> which comes with some neat settings for common enddevices such as smartphone and tablets. Yet another tool for simple cutting, scaling and cropping operations is <a href="http://www.virtualdub.org/">Virtual Dub</a>, which however doesn’t handle the encoding side as nicely.</p>
<h1 id="converting-to-the-final-format">Converting to the final format</h1>
<p>When you are done with the clip, the video is exported into its final format. Here you should take care to use a standard container (no, not WMV/ASF) with a standard video and audio codec (no, not Fraps or Cinepak or DivX or …) so that the video will work in Safari, Chrome, Firefox and Opera directly and play with the default video player of the operating system, even on mobile devices. You may use the command above for FFMpeg (and add audio settings if necessary), or use MPEGStreamclip like this:</p>
<ul>
<li>Open your uncompressed, edited video</li>
<li>Go to File -> Export to MPEG-4</li>
<li>Choose H.264 compression</li>
<li>Select 100% quality</li>
<li>Limit the data rate to 3000-8000 kbps</li>
<li>Disable sound if there is none in the video</li>
<li>Export the video</li>
</ul>
<p>And that’s it, you’re done. Prepare some template Photoshop intro and outro images (which contain paper title, author names, logos of the conference) and creating new videos from recorded material should be an effort of minutes!</p>
<h1 id="the-checklist">The checklist</h1>
<ul>
<li>All videos have the same resolution and framerate</li>
<li>All videos are either 720p or 1080p</li>
<li>You have re-encoded them with the intermediate codec of your editing tool</li>
<li>You have created a project in said editing tool with the same resolution and framerate of your videos</li>
<li>The final output is H.264</li>
</ul>
<p>Another helpful resource is <a href="https://support.google.com/youtube/answer/1722171?hl=en">the Youtube advanced encoding page</a>, which has two tables for recommended bitrates for standard and high quality movies.</p>
<h1 id="references">References</h1>
<ol>
<li><a href="http://www.ffmpeg.org/">FFmpeg</a></li>
<li><a href="http://www.squared5.org/">MPEGStreamClip</a></li>
<li><a href="http://www.fraps.com/">Fraps</a></li>
<li><a href="http://exkode.com/dxtory-features-en.html">DXTory</a></li>
<li>NVIDIA, <a href="http://www.geforce.com/geforce-experience/shadowplay">Shadowplay</a></li>
<li>Miles Macklin, <a href="http://blog.mmacklin.com/2013/06/11/real-time-video-capture-with-ffmpeg/">Real-Time Video Capture with FFmpeg</a></li>
<li>Google, <a href="https://support.google.com/youtube/answer/1722171?hl=en">Youtube Advanced Encoding</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Sun, 06 Jul 2014 00:00:00 +0200
https://www.tobias-franke.eu/log/2014/07/06/on-publication-videos.html
https://www.tobias-franke.eu/log/2014/07/06/on-publication-videos.htmlVideoPublicationNotesNotes on importance sampling
<p>Some tutorials on importance sampling specular terms that are out in the wild have what I found to be an information gap: the step from the PDF to the actual sampling function is missing. Hopefully this step-by-step guide can help out one or two other confused readers. I checked all integrals with <a href="http://maxima.sourceforge.net/">Maxima</a> and <a href="http://www.wolframalpha.com/">Wolfram Alpha</a>. If you just have a free account at Wolfram Alpha, you might run into issues with exceeding the computation time, which is why I will also write down the commands to do everything in Maxima. If your result doesn’t look like the one noted here, try simplifying it first (in the Maxima menu <em>Simplify -> Simplifiy Expression</em>) or put it into Wolfram Alpha and check the <em>Alternate form</em> section!</p>
<h1 id="phong">Phong</h1>
<p>The PDF for a normalized Phong BRDF with the specular power notation is:</p>
<script type="math/tex; mode=display">\begin{equation}
p(\theta, \phi) = \frac{\left(n+1\right)}{2 \pi} \cos^n \theta \sin \theta
\end{equation}</script>
<p>We generate two functions to sample theta and phi independently. To do this, <a href="http://www.wolframalpha.com/input/?i=integrate_0%5E%282pi%29+%28n%2B1%29%2F%282*pi%29+*+cos%28t%29%5En+*+sin%28t%29+dp">we first integrate $p(\theta, \phi)$ along the domain of $\phi$</a>:</p>
<blockquote>
<p>integrate((n+1)/(2 * %pi) * cos(t)^n * sin(t), p, 0, 2 * %pi)</p>
</blockquote>
<script type="math/tex; mode=display">\begin{equation}
p(\theta) = \int^{2 \pi}_0 p(\theta, \phi) d\phi = \left(n+1\right) \cos^n \theta \sin \theta
\end{equation}</script>
<p>This is the <em>marginal density function</em> of $\theta $. Note that the extra $\sin \theta$ at the end is there because we are dealing with differential solid angles in spherical coordinates. We can retreive the PDF for $\phi$ with the conditional probability $p(\phi | \theta)$, the <em>conditional density function</em>:</p>
<script type="math/tex; mode=display">\begin{equation}
p(\phi \| \theta) = \frac{p(\theta, \phi)}{p(\theta)} = \frac{1}{2 \pi}
\end{equation}</script>
<p>For isotropic NDFs this is always the case, no matter which PDF you will integrate over the domain of $\phi$, so all you need to calculate for the next PDF is the function for $\theta$.</p>
<p>Now that we have two functions for each variable, we can integrate both to generate a CDF each. <a href="http://www.wolframalpha.com/input/?i=integrate_0%5Es+1%2F%282pi%29+dp">The case of $\phi$ is trivial</a>:</p>
<blockquote>
<p>integrate(1/(2 * %pi), p, 0, s)</p>
</blockquote>
<script type="math/tex; mode=display">\begin{equation}
P(s_\phi) = \int_0^{s_\phi} p(\phi)d\phi = \int_0^{s_\phi} \frac{1}{2 \pi} d\phi = \frac{s_\phi}{2 \pi}
\end{equation}</script>
<p>If we set $P(s_\phi)$ to a random variable $\xi_\phi$ and solve for $s_\phi $, we get:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
P(s_\phi) & = & \frac{s_\phi}{2 \pi} = \xi_\phi \\
s_\phi & = & 2 \pi \xi_\phi \\
\end{eqnarray} %]]></script>
<p>Again, sampling around the azimuthal angle of an isotropic NDF is always the same because the material is rotationally invariant along this angle, so you don’t need to derive this formula again.</p>
<p>We now repeat the process and <a href="http://www.wolframalpha.com/input/?i=integrate_0%5Es+%28n%2B1%29+*+cos%28t%29%5En+*+sin%28t%29+dt">create a CDF for $p(\theta)$</a>:</p>
<blockquote>
<p>integrate((n+1) * cos(t)^n * sin(t), t, 0, s)</p>
</blockquote>
<script type="math/tex; mode=display">\begin{equation}
P(s_\theta) = \int_0^{s_\theta} p(\theta) d\theta = 1 - cos^{n+1} s_\theta
\end{equation}</script>
<p>Just like in the case of $\phi $, we set $P(s_\theta)$ to a random variable again and solve for $s $:</p>
<blockquote>
<p>solve(1 - cos(s)^(n+1) = x, s)</p>
</blockquote>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
P(s_\theta) & = & 1 - cos^{n+1} s_\theta = \xi_\theta \\
s_\theta & = & cos^{-1}\left( \left( 1 - \xi_\theta \right)^\frac{1}{n+1} \right) \\
\end{eqnarray} %]]></script>
<p>Therefore, a GLSL shader which generates important directions for a Phong NDF from random, uniform values looks like this:</p>
<figure class="highlight"><pre><code class="language-glsl" data-lang="glsl"><span class="kt">vec2</span> <span class="nf">importance_sample_phong</span><span class="p">(</span><span class="kt">vec2</span> <span class="n">xi</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">float</span> <span class="n">phi</span> <span class="o">=</span> <span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span> <span class="o">*</span> <span class="n">PI</span> <span class="o">*</span> <span class="n">xi</span><span class="p">.</span><span class="n">x</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">theta</span> <span class="o">=</span> <span class="n">acos</span><span class="p">(</span><span class="n">pow</span><span class="p">(</span><span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span> <span class="o">-</span> <span class="n">xi</span><span class="p">.</span><span class="n">y</span><span class="p">,</span> <span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span><span class="o">/</span><span class="p">(</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span><span class="p">)));</span>
<span class="k">return</span> <span class="kt">vec2</span><span class="p">(</span><span class="n">phi</span><span class="p">,</span> <span class="n">theta</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>Note that since $\xi_\theta \in [0,1)$ the expression $1 - \xi_\theta$ is also a random variable in the same range.</p>
<p>You can for instance use <a href="https://en.wikipedia.org/wiki/Constructions_of_low-discrepancy_sequences">a low-discrepancy sequence</a> as a source for uniformly distributed random values <em>xi</em> and generate important samples which aren’t clumped.</p>
<h1 id="trowbridge-reitz-aka-ggx">Trowbridge-Reitz aka GGX</h1>
<p>The GGX normal distribution function is part of a microfacet model, which gives you the probability of micro-surface normals oriented along a certain direction. The term has to be normalized though brefore integrating over the hemisphere to account for the projected micro-surface area:</p>
<script type="math/tex; mode=display">\begin{equation}
\int_\Omega D(\mathbf{m})(\mathbf{n} \cdot \mathbf{m}) d\mathbf{m} = 1
\end{equation}</script>
<p>The NDF itself is defined as:</p>
<script type="math/tex; mode=display">\begin{equation}
D(\mathbf{m}) = \frac{\alpha^2}{\pi \left( \cos^2 (\mathbf{n} \cdot \mathbf{m}) \left(\alpha^2 - 1 \right) +1 \right)^2}
\end{equation}</script>
<p>Just like above, we start out with the PDF for GGX:</p>
<script type="math/tex; mode=display">\begin{equation}
p(\theta, \phi) = \frac{\alpha^2}{\pi \left( \cos^2 \theta \left(\alpha^2 - 1 \right) +1 \right)^2} \cos\theta \sin\theta
\end{equation}</script>
<p>As in the case of Phong, we create two functions for $\theta$ and $\phi $. First <a href="http://www.wolframalpha.com/input/?i=integrate_0%5E%282pi%29+%28a%5E2*cos%28t%29*sin%28t%29%29%2F%28pi*%28%28a%5E2%E2%88%921%29*cos%28t%29%5E2%2B1%29%5E2%29+dp">let’s create $p(\theta) $</a>:</p>
<blockquote>
<p>integrate((a^2 * cos(t) * sin(t))/(%pi * ((a^2−1) * cos(t)^2+1)^2), p, 0, 2 * %pi)</p>
</blockquote>
<script type="math/tex; mode=display">\begin{equation}
p(\theta) = \int^{2 \pi}_0 p(\theta, \phi) d\phi = \frac{2 \alpha^2}{\left( \cos^2 \theta \left( \alpha^2 -1 \right) + 1 \right)^2} \cos \theta \sin \theta
\end{equation}</script>
<p>The integration for $\phi$ is the same as above, so we skip it and instead now <a href="http://www.wolframalpha.com/input/?i=integrate_0%5Es+%28a%5E2*cos%28t%29*sin%28t%29%29%2F%28pi*%28%28a%5E2%E2%88%921%29*cos%28t%29%5E2%2B1%29%5E2%29+dt">create the CDF for $p(\theta) $</a>:</p>
<blockquote>
<p>integrate((2 * a^2 * cos(t) * sin(t))/((a^2−1) * cos(t)^2+1)^2, t, 0, s)</p>
</blockquote>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
P(s_\theta) & = & \int_0^{s_\theta} p(\theta) d\theta \\
& = & 2 \alpha^2 \left( \frac{1}{ \left( 2 \alpha^4 - 4 \alpha^2 + 2 \right) \cos^2 s_\theta + 2 \alpha^2 - 2} - \frac{1}{2 \alpha^4 - 2 \alpha^2 } \right) \\
\end{eqnarray} %]]></script>
<p>Setting the CDF to a random variable and solving for $s$ yields:</p>
<blockquote>
<p>solve(2 * a^2 * (1/((2 * a^4−4 * a^2+2) * cos(s)^2+2 * a^2−2)−1/(2 * a^4−2 * a^2)) = x, s)</p>
</blockquote>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
P(s_\theta) & = & \xi_\theta \\
s_\theta & = & cos^{-1} \left( \sqrt{ \frac{1 - \xi_\theta}{\left( \alpha^2 -1 \right) \xi_\theta + 1}} \right) \\
\end{eqnarray} %]]></script>
<p>A simple GLSL function to generate important directions looks like this:</p>
<figure class="highlight"><pre><code class="language-glsl" data-lang="glsl"><span class="kt">vec2</span> <span class="nf">importance_sample_ggx</span><span class="p">(</span><span class="kt">vec2</span> <span class="n">xi</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">float</span> <span class="n">phi</span> <span class="o">=</span> <span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span> <span class="o">*</span> <span class="n">PI</span> <span class="o">*</span> <span class="n">xi</span><span class="p">.</span><span class="n">x</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">theta</span> <span class="o">=</span> <span class="n">acos</span><span class="p">(</span><span class="n">sqrt</span><span class="p">((</span><span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span> <span class="o">-</span> <span class="n">xi</span><span class="p">.</span><span class="n">y</span><span class="p">)</span><span class="o">/</span>
<span class="p">((</span><span class="n">a</span><span class="o">*</span><span class="n">a</span> <span class="o">-</span> <span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span><span class="p">)</span> <span class="o">*</span> <span class="n">xi</span><span class="p">.</span><span class="n">y</span> <span class="o">+</span> <span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="n">f</span><span class="p">)</span>
<span class="p">));</span>
<span class="k">return</span> <span class="kt">vec2</span><span class="p">(</span><span class="n">phi</span><span class="p">,</span> <span class="n">theta</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<h1 id="conclusion">Conclusion</h1>
<blockquote>
<p>The ultimate goal of mathematics is to eliminate any need for intelligent thought.</p>
</blockquote>
<p>Other NDFs can be used to create important samples with the exact same procedure: given a hemispherical PDF (i.e. the specular part of the BRDF), create two independent PDFs for $\theta$ and $\phi$, integrate both from $0$ to $s$ (i.e. create a CDF for each), set the result to $\xi$ and solve for $s$.</p>
<h1 id="references">References</h1>
<ol>
<li>Walter et al., <a href="http://www.cs.cornell.edu/~srm/publications/EGSR07-btdf.html">Microfacet Models for Refraction through Rough Surfaces</a></li>
<li>Wikipedia, <a href="https://en.wikipedia.org/wiki/Constructions_of_low-discrepancy_sequences">Constructions of low-discrepancy sequences</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Sun, 30 Mar 2014 00:00:00 +0100
https://www.tobias-franke.eu/log/2014/03/30/notes_on_importance_sampling.html
https://www.tobias-franke.eu/log/2014/03/30/notes_on_importance_sampling.htmlPBSImportanceSamplingNotesPhysically based AR
<h1 id="introduction">Introduction</h1>
<p>A particularly brutal aspect of real-time rendering in AR environments is that your object is directly exposed to the physical reality around it for comparison. Unlike your typical game-engine, masking or hiding physical inaccuracies in the model used to simulate real reflection of light on your virtual object is not going to work very well when all the other objects on screen somehow behave differently. If the augmented object doesn’t match up with reality, a knee-jerk reaction is to edit the responsible shader and simply tweak the output with a bunch of random multipliers until all looks right again. However, before all the magic-number adjustments get out of hand, it might be time to review the basics and avoid a non-physically-based-shading-post-traumatic-stress-disorder™.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2014_02_dragon.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2014_02_dragon.jpg" alt="" /></a><br />
<em>Image based lit Stanford dragon: diffuse SH + specular environment map sampling.</em></p>
<p>Many real-time engine writers have concentrated on getting their equations right. The most recent result is the <em>Siggraph Physically Based Shading Course</em> (<a href="http://blog.selfshadow.com/publications/s2012-shading-course/">2012 Edition</a>, <a href="http://blog.selfshadow.com/publications/s2013-shading-course/">2013 Edition</a>). I will roughly lay out the basic principles of PBS and refer to other very good online blogs which have covered most of the details that do not need to be repeated ad absurdum.</p>
<p>Consider the standard form of the rendering equation:</p>
<script type="math/tex; mode=display">L\left(x, \mathbf{\omega_o}\right) = \int_\Omega f_r \left(x, \mathbf{\omega_i}, \mathbf{\omega_o}\right) L\left(x, \mathbf{\omega_i}\right) \cos \theta_i d\mathbf{\omega_i}</script>
<p>When shading your object, it is important that light reflection off the surface behaves physically plausible, that is the BRDF $ f_r $ has to respect certain conditions:</p>
<ul>
<li><strong>Positivity</strong>: the value of the BRDF is always positive.</li>
</ul>
<script type="math/tex; mode=display">f_r \left(x, \mathbf{\omega_i}, \mathbf{\omega_o} \right) \geq 0</script>
<ul>
<li><strong>Energy conservation</strong>: the total amount of energy reflected over all directions of the surface must be less or equal to the total amount of energy incident to it. In practical terms this means that the visible energy (i.e. reflected light) can at best decrease after bouncing off a surface, while the rest turns to heat or some other form which isn’t part of the simulation. A non-emissive surface however cannot emit more light than it received.</li>
</ul>
<script type="math/tex; mode=display">M = \int_\Omega L \left(x, \mathbf{\omega_o}\right) \cos \theta_o d\mathbf{\omega_o} \le \int_\Omega L \left(x, \mathbf{\omega_i}\right) \cos \theta_i d\mathbf{\omega_i} = E</script>
<script type="math/tex; mode=display">\forall \mathbf{\omega_o}, \int_\Omega f_r(x, \mathbf{\omega_o}, \mathbf{\omega_i}) cos \theta_i d\mathbf{\omega_i} \leq 1</script>
<ul>
<li><strong>Helmholtz reciprocity</strong>: the standard assumption in geometric optics is that exchanging in- and outgoing light direction $ \mathbf{\omega_o} $ in the BRDF doesn’t change the outcome.</li>
</ul>
<script type="math/tex; mode=display">f_r \left(x, \mathbf{\omega_i}, \mathbf{\omega_o} \right) = f_r \left(x, \mathbf{\omega_o}, \mathbf{\omega_i} \right)</script>
<ul>
<li><strong>Superposition</strong>: the BRDF is a linear function. Contribution of different light sources may be added up independently. There is a debate about whether this is a property of the BRDF or light.</li>
</ul>
<p>It is usually the energy conservation in the specular term where things go wrong first. Without normalization Blinn-Phong for instance, a popular and easy way to model specular reflection, can easily be too bright with small specular powers and quickly loose too much energy when increasing the term.</p>
<p>But there is more! Many renderers assume that there are materials which are perfectly diffuse, i.e. they scatter incident light in all directions equally. There is no such thing. John Hable <a href="http://filmicgames.com/archives/547">demonstrated this by showing materials</a> which would be considered to be diffuse reflectors. You can read more in his article <a href="http://filmicgames.com/archives/557">Everything has Fresnel</a>.</p>
<p>So here we are, with a BRDF that can output too much energy, darkens too quickly and can’t simulate the shininess of real world objects because the model is built with unrealistic parameters. How do we proceed?</p>
<p>One solution is to <a href="http://www.thetenthplanet.de/archives/255">find a normalization factor for Blinn-Phong</a> to fix the energy issues with the model and add a Fresnel term. There are also several other reflection models to choose from: Oren-Nayar, Cook-Torrance, Ashikhmin-Shirley, Ward…</p>
<h1 id="microfacet-models">Microfacet Models</h1>
<p>Physically based BRDF models are built on the theory of microfacets, which states that the surface of an object is composed of many tiny flat mirrors, each with its own orientation. The idea of a microfacet model is to capture the appearance of a macro-surface not by integration over its micro-factets, but by statistical means.</p>
<p>A microfacet model looks like this (with the notation for direct visibility):</p>
<script type="math/tex; mode=display">f_\mu \left( \mathbf{l}, \mathbf{v} \right) = \frac{k_d}{\pi} + \frac{ F \left( \mathbf{l}, \mathbf{h} \right) G \left( \mathbf{l}, \mathbf{v}, \mathbf{h} \right) D \left( \mathbf{h} \right) } { 4 \left( \mathbf{n} \cdot \mathbf{l} \right) \left( \mathbf{n} \cdot \mathbf{v} \right) }</script>
<p>where</p>
<script type="math/tex; mode=display">\mathbf{h} = \left| \mathbf{l} + \mathbf{v} \right|</script>
<p>This specular part of the BRDF has three important components:</p>
<ul>
<li>the <em>Fresnel factor</em> $ F $</li>
<li>a <em>geometric term</em> $ G $</li>
<li>a Normal Distribution Function (NDF) $ D $</li>
</ul>
<p>The Fresnel term $ F $ simulates the Fresnel behavior and can be implemented with the <a href="http://en.wikipedia.org/wiki/Schlick's_approximation">Schlick approximation</a>.</p>
<p>The geometric term $ G $ models the self-occlusion behavior of the microfacets on the surface and can be thought of as a visibility factor for a micro-landscape which simply depends on one parameter for the surface roughness.</p>
<p>The NDF $ D $ is the term that gives you the distribution of microfacet normals across the surface from a certain point of view. If more microfacets are oriented in the half-vector direction $ \mathbf{h} $, the specular highlight will be brighter. $ D $ is the <em>density</em> of normals oriented in direction $ \mathbf{h} $.</p>
<h1 id="conclusion">Conclusion</h1>
<p>To sum up, the idea here is to start out with the correct shading model to avoid inconsistencies that might turn up later (I’m speaking out of negative experience tweaking arbitrary parameters of my old AR renderer). Turning to such a model might not produce visible differences immediately, but will later be noticeable once GI is added to the system where the light bounces multiply any error one made right at the start.</p>
<p>A neat way to check how you are on the reality-scale is <a href="http://www.disneyanimation.com/technology/brdf.html">Disney’s BRDF Explorer</a>, which comes with GLSL implementations of several microfacet terms and other BRDFs (have a look at the brdf/ subdirectory and open one of the .brdf files).</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2014_02_brdfexp.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2014_02_brdfexp.jpg" alt="" /></a> <br />
<em>Disney’s BRDF explorer</em></p>
<p>You can download measured materials <a href="http://www.merl.com/brdf/">from the MERL database</a>, load them in the BRDF Explorer and compare them to see how well analytical BRDFs match up to reality.</p>
<h1 id="more-references">More References</h1>
<ol>
<li>Hill et al., <a href="http://blog.selfshadow.com/publications/s2012-shading-course/">Siggraph PBS 2012</a></li>
<li>Hill et al., <a href="http://blog.selfshadow.com/publications/s2013-shading-course/">Siggraph PBS 2013</a></li>
<li>Brian Karis, <a href="http://graphicrants.blogspot.de/2013/08/specular-brdf-reference.html">Specular BRDF Reference</a></li>
<li>John Hable, <a href="http://filmicgames.com/archives/557">Everything has Fresnel</a></li>
<li>Rory Driscoll, <a href="http://www.rorydriscoll.com/2009/01/25/energy-conservation-in-games/">Energy Conservation In Games</a></li>
<li>Rory Driscoll, <a href="http://www.rorydriscoll.com/2013/11/22/physically-based-shading/">Physically-Based Shading</a></li>
<li>Simon Yeung, <a href="http://simonstechblog.blogspot.de/2011/12/microfacet-brdf.html">Microfacet BRDF</a></li>
<li>Sébastien Lagarde, <a href="http://seblagarde.wordpress.com/2012/01/08/pi-or-not-to-pi-in-game-lighting-equation/">PI or not to PI in game lighting equation</a></li>
<li>Wikipedia, <a href="http://en.wikipedia.org/wiki/Schlick's_approximation">Schlick’s approximation</a></li>
<li>Christian Schüler, <a href="http://www.thetenthplanet.de/archives/255">The Blinn-Phong Normalization Zoo</a></li>
<li>D3D Book, <a href="http://content.gpwiki.org/index.php/D3DBook:(Lighting)_Cook-Torrance">Cook-Torrance</a></li>
<li>Disney, <a href="http://www.disneyanimation.com/technology/brdf.html">BRDF Explorer</a></li>
<li>MERL, <a href="http://www.merl.com/brdf/">MERL BRDF Database</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Sun, 02 Feb 2014 00:00:00 +0100
https://www.tobias-franke.eu/log/2014/02/02/physically-based-ar.html
https://www.tobias-franke.eu/log/2014/02/02/physically-based-ar.htmlIntroductionPBSA magic lantern
<h1 id="prologue">Prologue</h1>
<p>When adding a virtual object into a real scene, the renderer has to shade the object depending on the incident light it receives. Furthermore, the renderer might also take care of changes on reality which were caused by the object, such as a drop-shadow from it. But before we proceed, let’s not reinvent the wheel here: great inspiration can be drawn from the knowledge gathered in the film industry on how to augment reality properly, and by taking a step back and learn from history.</p>
<h1 id="introduction-to-the-macabre">Introduction to the macabre</h1>
<p>The earliest form of an augmented reality can be found shortly after the introduction of the <a href="https://en.wikipedia.org/wiki/Magic_lantern">Lanterna Magica</a> in the mid 16th century.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2014_01_ml.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2014_01_ml.jpg" alt="" /></a></p>
<p>Giovanni Fontana, Christian Huygens and/or Athanasius Kircher invented the first apparatus to project images onto a flat surface with a type of lantern that focuses light through a painted glass image. The glass can be moved through a slit on the side of the projection apparatus, much like a more modern slide projector.</p>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2014_01_woo.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2014_01_woo.jpg" alt="" /></a><br />
<em>From F. Marion, The Wonders Of Optics, 1869</em></p>
<p>It wasn’t long until the device became known as the “lantern of fear” when it was used to conjure up the occult. Apparitions of the dead projected onto smoke via rapidly exchanging images convinced an entire generation that some stage performers had actually attained a special connection to the afterlife. Combined with the bizarre fascination for spiritualism at that time, these shows caused such commotion that eventually the authorities stepped in to end it all.</p>
<p>What remained however was a new kind of artist: a cross-breed between magician and scientist.</p>
<h1 id="moving-images">Moving images</h1>
<p>The French illusionist and filmmaker Georges Méliès accidentally discovered what became known as the <em>stop-trick</em>, whereby a movie shot is halted and the filmed scene is substituted with something different. A great example is his short movie <em>The Hilarious Poster</em>, filmed in 1906. A regular poster on a street takes on a life of its own with horseplay directed at passengers which happen to walk by.</p>
<div class="video default-size">
<img src="https://www.tobias-franke.eu/layout/thumbcache/dPPP2MrlYr8.jpg" />
<a class="fa fa-5x default-size" href="https://www.youtube.com/watch?v=dPPP2MrlYr8"></a>
</div>
<p><em>The Hilarious Poster by Georges Méliès</em></p>
<p>In the modern film industry, early effects such as the light saber glow seen in the first Star Wars movies where inserted via <a href="https://en.wikipedia.org/wiki/Rotoscoping">rotoscoping</a>, a technique where the movie is basically edited and enhanced frame-by-frame with a new painted overlay. Initially, the laser sword was a rod with scotchlite attached to it (the material used for traffic sign reflectors), but it turned out that it didn’t reflect enough light for the movie, so the effect was added later.</p>
<p>Rotoscoping however is tedious, and surely there are better, more automated ways to add effects to a movie after it has been shot. A curious case is William’s and Chou’s “Interface”, shown at SIGGRAPH 1985 featuring a shiny robot kissed by a woman. The textured-mapped robot however was rendered and added afterwards, while the reflection of this romantic scene on its polished surfaces was filmed using a mirroring ball. Lance Williams <a href="http://dl.acm.org/citation.cfm?id=801126">published a paper two years earlier</a> which introduced mip-mapping to the world of computer graphics. In an almost Fermat-like manner, the paper contains a paragraph which reads:</p>
<blockquote>
<p>If we represent the illumination of a scene as a two-dimensional map, highlights can be effectively antialiased in much the same way as textures. Blinn and Newell [I] demonstrated specular reflection using an illumination map. The map was an image of the environment (a spherical projection of the scene, indexed by the X and Y components of the surface normals) which could be used to cast reflections onto specular surfaces. The impression of mirrored facets and chrome objects which can be achieved with this method is striking; Figure (16) provides an illustration. Reflectance mapping is not, however, accurate for local reflections. To achieve similar results with three dimensional accuracy requires ray-tracing.</p>
</blockquote>
<p><a href="https://www.tobias-franke.eu/layout/logcache/2014_01_lw.jpg"><img src="https://www.tobias-franke.eu/layout/logcache/2014_01_lw.jpg" alt="" /></a><br />
<em>Figure 16, courtesy of Lance Williams</em></p>
<p>Using Jimm Blinn’s modified <a href="https://en.wikipedia.org/wiki/Reflection_mapping">environment mapping technique</a> with real images instead of computer generated ones did the trick. <a href="http://www.pauldebevec.com/ReflectionMapping/illumap.pdf">Course notes from Miller and Hoffman at SIGGRAPH 1984</a> additionally added the idea of pre-convolving these images to simulate reflection off other glossy or diffuse materials. Eventually, the concept art became a reality when the technique was used in <em>The Flight of the Navigator</em> in 1986, followed by <em>The Abyss</em> in 1989 and <em>Terminator 2</em> in 1991.</p>
<div class="video default-size">
<img src="https://www.tobias-franke.eu/layout/thumbcache/8LlcqO3mq9Q.jpg" />
<a class="fa fa-5x default-size" href="https://www.youtube.com/watch?v=8LlcqO3mq9Q"></a>
</div>
<p><em>Trailer: The Flight of the Navigator</em></p>
<p>What makes this technique so elegant is that it circumvents the need to track light sources in your scene. One could also extract single point light sources from a hemispherical image, but small reflected highlights from shiny objects can easily distract the algorithm and cause flickering when using low sampling rates. A reflection map instead represents the entire lighting configuration and is therefore physically more plausible than a bunch of point lights.</p>
<p>There is however no inherent necessity to augment reality with objects that have to look physically plausible. First and foremost, it is important that whatever has been added to the scene is at its proper place at all times (i.e. <em>geometrically registered</em>). One grandiose example is Jessica Rabbit’s <em>Why don’t you do it right</em>:</p>
<div class="video default-size">
<img src="https://www.tobias-franke.eu/layout/thumbcache/yy5THitqPBw.jpg" />
<a class="fa fa-5x default-size" href="https://www.youtube.com/watch?v=yy5THitqPBw"></a>
</div>
<p><em>A scene from: Who Framed Roger Rabbit</em></p>
<p>As you can see, Jessica sometimes vanishes behind other objects or people as if she were actually at the correct position in the 3D scene. Of course being a comic character, Jessica’s painters took care of this, but one can use invisible impostors in a renderer to create a virtualized depth buffer of the real scene if none exists.</p>
<p>Note that the clip features another important property, which is scene consistency: Jessica’s reflection on the real catwalk can be seen 14 seconds into the clip. The editors know about the reflective material and can draw the reflection that appears with the viewing angle of the camera. The shadow of her hand is also visible on some dude’s face at exactly minute one. Jessica’s presence in the scene changes reality (and not just by making the audience go nuts)!</p>
<p>Of course, ever since stop-motion was declared extinct by Phil Tippet while producing the animations for Jurassic Park, having computer-generated effects in movies has become the norm. But irrespective of the technique used to augment something into a real scene, capturing reality correctly and having consistent light interaction is key to create the proper illusion of a mixed reality. Real and virtual light sources, direct and indirect, influence the virtual and real space. The importance of this is nowhere more clear and apparent than when watching a bad, low-budget sci-fi movie.</p>
<p>Because your typical SFX director may go overboard with the special effects in a movie scene, it might happen that the only real thing left is the actor himself. Humans are sensitive to facial features, and it is therefore necessary to edit the appearance of skin reflections to match the scene. You don’t want to end up with a green aura from the studio set engulfing the actor. SIGGRAPH 2000 brought a revolution to the film industry with the introduction of <a href="http://gl.ict.usc.edu/LightStages/">the Light Stage</a>. In it, the actors face is captured multiple times under varying illumination. After enough samples have been gathered, for each pixel on the actors face one can effectively invert the rendering equation and extract the BSDF of the skin and reproduce it under <em>different</em> illumination.</p>
<div class="video default-size">
<img src="https://www.tobias-franke.eu/layout/thumbcache/piJ4Zke7EUw.jpg" />
<a class="fa fa-5x default-size" href="https://www.youtube.com/watch?v=piJ4Zke7EUw"></a>
</div>
<p><em>The Digital Emily Project</em></p>
<p>This method doesn’t just work with faces, but with materials in general. There is a broad range of reconstruction algorithms for materials, but the basic method still includes a dome which captures samples of a surface under varying illumination conditions. Every time simulated light is supposed to interact with a real surface, its properties have to be known upfront.</p>
<h1 id="light-and-magic">Light and Magic</h1>
<p>So in essence, the evolution of the film industry from a simple projector used in 17th century theater to the intricate details produced for the <em>The Lord of the Rings</em> movies already provides enough knowledge on the problem of correct augmentation. Over time, artists figured out several methods to capture real light, honor light interaction with real and virtual surfaces and improve overall consistency by reconstructing the world in a more elaborate manner.</p>
<p>In the next articles, I will go into details on how to translate most of these methods to real-time algorithms. We’ll start off creating a basic pipeline for an AR renderer and then proceed to create shaded virtual geometry first before relighting reality.</p>
<h1 id="references">References</h1>
<ol>
<li>Fulgence Marion, <a href="https://archive.org/details/wondersofoptics00mariiala">The Wonders Of Optics</a></li>
<li>Jimm Blinn, <a href="http://dl.acm.org/citation.cfm?id=360353">Texture and reflection in computer generated images</a></li>
<li>Wikipedia, <a href="https://en.wikipedia.org/wiki/Rotoscoping">Rotoscoping</a></li>
<li>Lance Williams, <a href="http://dl.acm.org/citation.cfm?id=801126">Pyramidal parametrics</a></li>
<li>Disney, <a href="https://www.youtube.com/embed/8LlcqO3mq9Q">The Flight of the Navigator (Trailer)</a></li>
<li>Paul Debevec, <a href="http://www.pauldebevec.com/ReflectionMapping/">The Story of Reflection Mapping</a></li>
<li>Georges Méliès, <a href="https://www.youtube.com/watch?v=dPPP2MrlYr8">The Hilarious Poster</a></li>
<li>Wikipedia, <a href="https://en.wikipedia.org/wiki/Georges_M%C3%A9li%C3%A8s">Georges Méliès</a></li>
<li>Robert Zemeckies, <a href="https://www.youtube.com/watch?v=yy5THitqPBw">Who Framed Roger Rabbit</a></li>
<li>USC, <a href="http://gl.ict.usc.edu/LightStages/">The Light Stages at UC Berkeley and USC ICT</a></li>
</ol>
<br/> <hr/> <br/> <em>Did you like this post? You can leave a tip.</em> <br/><br/> ETH: <a title="Ethereum" href="https://www.tobias-franke.eu/download/qr/0xc38267f5C592277cD153c5FE81375f7eEd56A2C2.png">0xc38267f5C592277cD153c5FE81375f7eEd56A2C2</a><br/> BTC: <a title="Bitcoin" href="https://www.tobias-franke.eu/download/qr/19Xq6bXif7keevSNyc6deSw5cQm36QLzxm.png">19Xq6bXif7keevSNyc6deSw5cQm36QLzxm</a><br/> BCH: <a title="Bitcoin Cash" href="https://www.tobias-franke.eu/download/qr/qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734.png">qpggmzulrudqm7rkk05rv6pea3me4rkmku8jq76734</a><br/> XLM: <a title="Stellar Lumens" href="https://www.tobias-franke.eu/download/qr/GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A.png">GCELMWABQUOK764IRNVHID7HVCPMUBHV2U7PFZWRCMJQM5TUG4EZ7X6A</a><br/> LTC: <a title="Litecoin" href="https://www.tobias-franke.eu/download/qr/ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv.png">ltc1qsksu8qsvxwphpp2zh9qvgtwx54j2tx7pl7ngsv</a>
Tue, 07 Jan 2014 00:00:00 +0100
https://www.tobias-franke.eu/log/2014/01/07/a-magic-lantern.html
https://www.tobias-franke.eu/log/2014/01/07/a-magic-lantern.htmlGeneralRelightingIntroduction