3D
relighting in compositing - yet another approach
an exercise in 3D mathematics (and a useful Fusion macro plugin, too)
an exercise in 3D mathematics (and a useful Fusion macro plugin, too)
UPDATE: the plugin is here, and it's FAST! Get it here.
I found myself in need of a novel approach to 'compositing time' relighting of 3D rendered scenes. What one usually has, either as an integrated or a third-party tool, is the capability of shading objects based on their surfaces' normals, which mimics the behaviour of a directional light. During some (admittedly deeply ugly) light-generating particle work I needed a quick way to simulate the lighting characteristics of a point light, instead, with intensity decaying according to distance etc.
In a matter of few hours Macro v001 was born. It was quick indeed, both to construct and in use, relying on quite an impressive range of mandatory inputs: a normal pass, a 'world position' pass (a picture, below, is worth more than these ten words) and the imported 3D camera. With such abundance of information, it was only a matter of subtractions (to get some vectors) and a bunch of CustomTools (how could I ever forget those) to get more vectors and calculate the lighting via Blinn-Phong reflection model.
Test scene: Fusion renderer (via FBX import) on the left, macro on the right - it could use some normal smoothing, or a glow, or something
Voici la (v002) macro, voila les diffuse (red) and specular (green) components
How to get a 'world coordinates' shader in Maya - a floating point format is needed, of course
Fair enough. After a few minutes playing with my tool, I realized that there had to be redundance somewhere, and that my reasoning had been to simply mimic an (utterly unsatisfactory) 3D renderer. But, we compositors live and prosper in a 2D world, don't we? And also: to be provided with a normal pass is, evidently, normal, but what about the world position pass? Even more: a 3D renderer takes the 3D world position of a point and transforms it, according to the camera position, orientation and lens, into a 2D point on the screen; is it possible to do the opposite, going from 2D render - which we're supposed to have - to 3D position knowing the camera attributes?
Well, the answer is no if the rendered image is strictly 2D, but all we need is a Z channel - and nowadays, if your render doesn't already come with a Z channel, you have the constitutional right to complain about the 3D department (we all must thank the incessant lobbying of the Compositors Guild) - besides, you can always get a bunch of stock Z channels for a couple of bucks on Ebay.
An RGBAZ image, if seen as <X2d,Y2d,Z> point positions, gives us a 3D 'projected' space - not plain 3D (literally) but enough to get us started. The first thing we need to do is 'unproject' the space into an ordinary camera space (i.e. a plain 3D space with origin at the camera); then we 'unrotate' the camera, that is, we perform the inverse of the renderer camera rotation transformation, which happens to be the inverse of a regular rotation of an object - in the end we just rotate the world with the camera angles; and finally we translate the world back to the origin.
Unproject, rotate and traslate all in one go
That (right) is what a world position shader looks like in RGB, and this (left) is the same thing calculated from Z channel and camera attributes (confused?)
After having gone through all this hard work to get the world position (I had to start with 3D Basics for Really Clueless Dummies and 2nd Grade Maths for Former Italian Journalists Looking For Another Opportunity) I felt quite unsatisfied of the somewhat limited result; I soon realized that I could also derive normals just by doing a cross product of two vectors lying on the surface, and, now that I had world position, getting these two vectors (or an approximation of them) for any given point was only a matter of subtracting it from two nearby points - a simple convolution matrix.
Normals from world position from Z+camera (left), normals from mental ray (right) - notice the noticeable tessellation artifacts of the poly spheres
Then things started to go awfully wrong; I won't even say what I was trying to accomplish - these were the preliminar results:
Mmm, let's try with a different software - I didn't particularly enjoy the interface anyway
Quaternions sound cool, too bad there's a compatibility issue with my math skills
Actually, after
repeatedly hitting my
head with a wall (the other way round
would not be compliant with my discontent) I could obtain one out of
the three numbers that I
seeked, then the other one, then the last one, only not at the same
time. But I'm confident I'll eventually figure it out, maybe in a
couple of years.
Enough. I finally threw in some more niceties (ehm... like a bump mapping looking so perfecly fake I decided to keep it, and a 'spotlight' mode - basically another dot product) and wrapped everything in macro v002 - which you can find at the top of this very page. The thing is so slow (especially when only fed with the Z channel) that I'm thinking of doing a plugin of it, then immediatly changing my mind. Weird.
Enough. I finally threw in some more niceties (ehm... like a bump mapping looking so perfecly fake I decided to keep it, and a 'spotlight' mode - basically another dot product) and wrapped everything in macro v002 - which you can find at the top of this very page. The thing is so slow (especially when only fed with the Z channel) that I'm thinking of doing a plugin of it, then immediatly changing my mind. Weird.