My name is Ozymandias

In one of the more bizarre hiring encounters I’ve experienced, I sit together with my family at lunch when I receive a Twitter message followed by an email that can be paraphrased as Saw your online profile, would you be interested in joining Unity? A couple of weeks later I am in Copenhagen at the office. The atmosphere is relaxed, family-like even, and I am getting interviewed by someone called Joe. I am ignorant of course and do not realize this isn’t Joe Shmoe, but the Joachim Ante, one of the founders and CTO of Unity.

First day at the office in Copenhagen

My first day at Unity is 1st of January 2016, following my departure at Fraunhofer IGD after 2 years of doing a PhD. I’m getting on-boarded by David Helgason as the only graphics programmer in a room full of mathematicians. David shows a keen interest in everyone and their story, and wants to know one special quirk about everyone. It’s a wild mixture under one company roof, and an analogy. His point of course is that one of the hard problems Unity needs to solve is catering to a wide variety of games using one single engine, something quite different from my former research environment where I used to have a very narrow and highly specified problem space in front of me and just wrote single-use code.

Today is my last day at Unity. I am very happy I got to work on this incredible engine. Having pushed code to millions of devices is simultaneously terrifying and exhilarating. In this post I will cover three things I learned during my time at Unity that - in hindsight - may seem obvious, but ultimately freed me from the bubble I came from. I had the privilege of working alongside most excellent people that helped me get there (in fact, many more than on this Twitter-only list): Kuba, Jesper, Kasper, Rasmus, David, Robert, Florent, Ionut, Paul, Arnaud, Veselin, Martin, Göksel, Arthur, Nicholas, Cyril, Adrien, Antoine, Sebastien, Francesco, Ben, Anis, Thomas, Laurent, Thomas, Julien, Christophe, Eric, Kemal, Laurent, Julien, Felipe, Evgenii, Peter, Natasha, Aras, Kleber, Rej and Baldur.

Generate Lighting

Every feature of a modern game engine is a cover on top of a bottomless pit full of problems to solve. For me, that cover was a single button, or so I thought. Initially it felt great to hide behind it: Here is that one feature where you can truly capsule off from everyone else. That button precomputes all static lighting using one of the global illumination backends the user can select. There are a bunch of settings to adjust for like ray counts etc., but all in all it seemed like a very contained world. However, complexity tends to leak.

I had dedicated a good amount of time to a conundrum: As the name suggests, global illumination is a global phenomenon, but for the sake of memory consumption many scenes are often tiled in some form or another, and then computed independently. The disconnect, i.e. the global lighting not being so global anymore, needs to be managed later somehow. A lot of user-complaints and tickets I went through had the same core issue: Two tiles of a scene were precomputed independently, but with - accidental or not - differing settings. Sometimes there was a different skybox involved, sometimes the GI backend was flipped or the textures stored directional lighting in one but not the other tile. For half a year, I wrote a lot of messages explaining that when putting those tiles back together, the contradicting settings will cause all sorts of issues, not just visually.

Even though everyone understood the message, the tickets kept coming. Eventually I sat in front of a project so convoluted that I too made the very same mistakes I tried to educate everyone about, and so I added warning labels to everything. Initially they were quite unpopular, but the flood of tickets came to a grinding halt.

In essence, I learned the hard way that no amount of documentation, videos or talks will iron out the complexities of your software. If there are no visual indicators attached to parameters that can be used wrong very easily, everyone will keep using them wrong until the end of time, not because of ignorance or because it is hard to use, but simply because the software is complex and mistakes are easy to make.

Physically based ork-skin

One of the things that used to baffle me before working in the games industry was the apparent excitement and ignorance artists and graphics programmers alike displayed when it came to physically based rendering. At SIGGRAPH and other conferences, courses dealing with rendering and lighting in games celebrated the fact they’ve adhered even slightly to the rules of reality and took away all their artists favorite sliders. Zero tolerance for shenanigans like lights with negative energy or materials that violate the conservation of energy! No longer would it be possible to abuse the lighting system, no longer would different game scenes have wildly different behavior and everything would be nice, coherent and predictable.

But did they not get the memo? The literature on physically based rendering was already ancient by graphics standards. Why the victory parade?

Unity’s platform support is huge. Unlike tailored engines, games made with Unity often run on many combinations of graphics APIs, operating systems, CPU architectures and end-devices. Additionally, the type of graphics required for a certain look pile on top of the already complex situation: 2D, 3D, realistic, open-world, top-down, minimalistic, stylized, massive, impossible… Then there is the project itself: A game, a prototype, a demo, a movie (real-time or a glorified frame capturer), a UI or some interactive educational tool. All of these factors are just technicalities though. Yes you might end up on a device that is still unable to use linear intensities, yes you may not have RTX support, yes the platform could be unable to stream all your data and yes the game may not even be next-gen.

I remember a brief chat with one of the tech-artists about the perfect renderer, that in my opinion simply required to follow a strict set of physically-guided rules and was augmented by tools that helped artists deal with the assets they need, like the built-in photogrammetry tool.

Sounds good, I just need that sample of ork-skin and we’ll get the cameras.

I sometimes tend to forget that most game settings are not situated in this universe. While tinkering with rendering architectures all day long where I focus to get one thing right - the transport of light - and dealing with all the technical issues - memory consumption, serialization, shader architecture, ray-throughput, threading etc. - it did not occur to me that the artist might actually want to break the rules of physics on purpose. In all cases the engine is both tool and stumbling block, either enabling a vision or preventing it. The engine programmer writing the underlying architecture is trapped between two extremes: On one side the desire to write code that behaves predictable and consistent across the wide variety of projects, styles and platforms, on the other to enable the freedom of the artists and their often conflicting and contradictory ideas of what they want see on screen.

At the end of the day, nothing is as important as the vision of the game though. Just like an artist can take a brush, an empty canvas and paint anything, be it an impression of the real world, a fantasy setting or an Escher-like mind-bender, so too should an engine deliver the flexibility to enable the game designer to achieve their goal, even if it is completely weird, unorthodox and against all reason.

And so I learned that in many cases the default implementation should just be easily replaceable. If the implementation does not work for the game, then it should be possible to use a custom one. Games are just different from other software in that regard, because they are pieces of art.

The engine is eternal

Invent yourself and then reinvent yourself is the slogan of the BMW demo showcasing the running version of the high-defintion render pipeline using DXR ray tracing, and it perfectly captures what goes on under the hood of Unity. The demo was the end-result of a long development cycle that started with a tiny hackweek project using the brand-new DXR API and its semi-correct documentation to render ambient occlusion. Like all paradigm shifts, the new way of doing things did not really fit nicely into the existing structures, at least in the beginning. Where should the DXR renderer live? Should it still work if the project is setup to use Vulkan, now that we’re using two APIs rather than one? Can the shaders run in parallel with the deferred renderer, and if so how do we synchronize them? How are the materials translated? Who is in control of the reflection probes? There were quite a few questions like these that required some back and forth, but in the end the renderer would fit in.

Adapting code to make things fit is sometimes not enough. After many years of service, the built-in render pipeline was simply too monolithic to be used across vastly different devices and could not provide the flexibility that many games needed, be it custom adaptations to the renderer or simply the option to rip out entire features a game is not using. Hence one night over some pizza at the office with Joe and a colleague, he casually pondered how Unity could look like if the render pipeline would be written in C#, as opposed to being built-in. One hackweek later and the Scriptable Render Pipeline was born. Though that was just a prototype. Now we suddenly had to decide who is in control of the flow. Should the renderer push its state down from C# to the rest of the base code in C++, or should it work the other way around? How are custom features - implemented in the a ScriptableRenderPipeline - reflected back, such as a whacky material with some new sliders that the global illumination backends are not aware of? Can the C++ code inject render calls, and if so when?

In 2018 we shipped the GPU Lightmapper, an enhancement of the Progressive Lightmapping backend that so far only ran on the CPU. If the development machine has a capable GPU in store, the GPU backend will render at significantly more rays per second. After a long process of abstracting away most jobs running inside the Lightmapper, we went with OpenCL as the compute solution, as it would run on all supported Editor platforms. And then, at WWDC2018, whilst Book of the Dead was shown on stage to demo the new eGPUs for MacBooks, Twitter began to notice that Apple had just deprecated OpenCL on macOS in favor of Metal.

When I compare projects I do for fun with the code I write for work, I can see a strong divergence between the two. In my hobby projects all code is following a very strict order, and if it does not then it is usually time to fix that. Throughout the code identical concepts are used. Helper functions abstract away repeating patterns and everything looks nice and tidy. The code seems to have a long life ahead, but only because it serves a single purpose.

In comparison, game engine code looks a bit like tectonic plates shifting at the speed of sound. Entire systems exist multiple times over, the control flow is not obvious, and features can get ripped out from the codebase rather abruptly. Code is rewritten much more frequently than in other software. To maintain a bleeding-edge status, developers are forced to constantly re-evaluate their code. Because of that, the balance when planning out features shifts quite dramatically to the near rather than the far future, and that plan is always one round away of loosing the API-, Paper- or Dependency-Russian-Roulette.

Now it might all sound like a lot of duct tape and a constant struggle to keep a patient alive while exchanging all guts at once and simultaneously doing brain surgery. One could get the impression that nothing is here to stay, and that eventually it is all but Unity in name.

But I learned that Unity earned its reputation as one of the most intuitive game engines out there not because it has been around forever unchanged, but because it is constantly adapted and modified to make current and new technology easily accessible to everyone. There may be a lot going on under the hood to keep that reputation, but this is the reason why a decade from now I’ll still be able to recognize the motor I helped build, and why the engine is forever.