Lightwrap with CustomTool
tutorial for Fusion5

This tutorial owes its existence (and our doggy hero) to an interesting discussion in the PigsFly forums. I hope it will benefit a broad range of readers: novices, beginners, and newbies alike. I myself am barely into Fusion, but after too many months with insert here the name of that other application that starts with an 'A' I was so excited with the novelty that I had an urge to share my profoundly benign feelings about this piece of software. But I managed to contain it and decided instead to make this crappy tutorial about the almighty CustomTool using lightwraps as an excuse.
Okay, enough with kidding - let's say that I'm just trying to show how to actually use CustomTools to build more compact and efficient comps. The general principle, of course, is also valid for Shake (channel expressions, I know they are there, somewhere) and the like.

Take your time - this stuff is not so trivial after all if you never did 'math compositing', but I'm sure you'll figure out what's happening at each step.
If you never happened to use a CustomTool before, you should first read eyeon's excellent page on it to understand its principle of operation and basic syntax - and the page about the Merge tool is another good source of information - okay, close this useless tab and go read the whole vfxpedia thing.
Update: I kinda realized that the abovementioned page is actually a bit cryptic for people that keep at safety distance from anything math/scripting (or just beginners), and since these people are the miscredents I want to convert I added quite a bit of general info/explanation here and there. Sorry for the inconvenience (it being the uncomfortable length of this page).
Here's our foreground dog along with his alpha channel
Let's start with a simple comp: our first very basic lightwrap is achieved by multiplying our matte to a blurred and inverted version of itself, and then multiplying the whole thing by 2 to rescale the range to 0-1.  (otherwise the outer edge of the resulting matte will have a value of 0.5 because of the blurring). I'll try to explain better this last step. Let's take a pixel at the very edge of our matte; for simplicity suppose its alpha value is exactly 1 (our matte is quite hard-edged). What we do when we 'simplified-version-of-the-story'-blur this pixel is to take a certain amount of white (matte) pixels from one side of it and average them with the same amount of black (non-matte) pixels on the other side. 20 white pixels averaged with 20 black pixels make a grey pixel, that means alpha value of 0.5. If we multiply this 0.5 pixel with the unblurred itself, that was 1.0, we get 0.5. For the very nature of the operation we're doing this is the maximum value we can get in our new matte, but it just makes no sense to work in the 0-0.5 range; so by multiplying by 2 we renormalize our matte (that is, we put it back to a nice 0-1 range). Speaking of colorcorrecion terms, gain is a simple multiplication (gain of 2.0 = x*2) that only preserves blacks while gamma is an exponential (gamma of 3.4 = x^1/3.4) that leaves the value in the 0 - 1 range, thus preserving both blacks and whites.
The initial flow and the resulting matte

Note: when I created the lightwrap I actually had the custom tools there from the very beginning; I tried to recreate the flow using many simple nodes for the sake of clarification.
Let's get into the juicy stuff: make a CustomTool and connect its Image1 input to the foreground image and the Image2 input to the Blur. We'll start by replicating the functionality of these MatteControl, Merge and ColorGain within one CustomTool (not the blur, though - convolution kernels are unpractical to achieve with non-tiny matrices and painfully slow).
 So how do CustomTools work? They evaluate frame by frame, pixel by pixel, the per-channel expressions provided in the Channels tab. These expressions can simply be numbers (if you type '0.5' in the Alpha expression, the output will have a solid semitransparent alpha channel); or they can reference channels in the input images by following the scheme 'channel letter+input number ' (the default 'r1' - that stands for 'red channel of input Image1' - in the Red expression takes whatever is in the red channel of the node connected to Image1 and passes it unaltered to the red channel of the output); or they can build new images by combining numbers and channels in the inputs within simple mathematical expressions.
 In the channels expressions we can now refer to the original foreground matte as a1, while the blurred matte is a2. The resulting Alpha Expression will be pretty straightforward:
a1 * (1-a2) * 2
Let's go thru this again - refer to the little flow with the notes above: we took our matte, blurred it (and that, once connected to Image2, becomes known as a2) then inverted it (1-a2), then merged it with the original matte via an operation In to multiply the two alphas (so (1-a2) * a1) then gained (multiplied) the resulting alpha by 2.

There comes our CustomTool, still shy yet ready to show its prowess

It's composite time! For now we'll keep things simple, so get a background, blur it and merge the result over our dog using our new matte; then merge the whole thing over the background:

So far, so good

What we want now is to have the background light wrapping only where it shines most, so we'll throw in two background luma thresholds: a low threshold (say .2) below which we want no lightwrap at all, and a high threshold (we'll make this .8) above which we want full lightwrap - and a linear blend inbetween. For this we would use the levels inside a BrightnessContrast or a ColorCorrector, but to better explain how to achieve the same result with the CustomTool I'm going to use two basic tools.

Slowly getting there

So, what's happening here? Let's follow the adventures of our .2 pixel (call it LTp) and .8 pixel (HTp) thru the flow. We start by converting our background to BW, since we're interested in luma; we then put a Brightness of -0.2, so our LTp and his even lower fellows get clamped to 0 and HTp goes to .6 (HT - LT); now we want HTp to become 1.0, hence 0.6 * x = 1.0, hence x = 1/0.6 = 1.66 - again, we use a ColorGain for multiplication.  We then multiply this to our matte with another Merge (Operation In).
Leaving the defaults of the ColorSpace, this turns out to be ((0.299r + 0.587g + 0.114b) - LT) * 1/(HT-LT). So now we can pipe our blurred background into Image3 of our CustomTool, set the desired low and high thresholds into Number In 1 and 2 in the controls tab - we refer to them as n1 and n2 in expression lingo - and update our Alpha Expression as follows:
a1*(1-a2)*2 * (max ((0.299*r3 + 0.587*g3 + 0.114*b3) - n1),0) / (n2-n1)
We need the max because inside an expression there's no clamping until the final result, and we placed it in such a fashion that it clamps the result of our cheap 'brightness math'. In this case actually it is not (yet) needed, since negative values will stay negative throughout the calculation and eventually get clamped anyway, but I feel safer to put min's and max's wherever it makes sense, just so we won't quiz for otherwise productive minutes on why we're getting absurd (though nice) results. Conversely if we don't clamp values over 1.0 - and we don't - this whole thing will work beautifully also with HDR images.

CustomTool fears no defeat

This matte looks promising

Using the very same principle we'll add now a foreground threshold to our matte, so that bright areas of the foreground will eat back the lightwrap, as we all know that in the eternal battle between Light and Light, Light wins. Prove me wrong.

Dear ol' flow's getting messy

By now you should already know that the selected tools over there are nothing more than a mere
max (1 - max((1- (0.299*r1 + 0.587*g1 + 0.114*b1) - n3),0) / n3), 0)
that gives us a nice clamped matte and that goes as another factor into our CustomTool's Alpha Expression, provided that you put the desired value into Number In 3.  This value, however, is not actually a threshold (we'd get a nasty bitmask out of it) but the value above which the resulting alpha starts decreasing, with a tighter falloff curve at lower values and a broader one at high values - and this makes sense anyway. If you need finer control, just provide the high threshold alongside and change /n3 to /(n4-n3) just like we did for the background thresholds.

We're almost done. The last touch is to give our lightwrap a nice matte gamma control and a blend control - trivial by now, isn't it? So let's put everything that is lightly'n'lazily lying in our Alpha Expression - which incidentally is everything we have typed so far, since expression chunks are notoriously lazy - in parentheses and add at the very end
^ (1/n5) * n6
being n5 and n6 the gamma value and the blend value that we dutifully type in Number In 5 and Number In 6, of course. We are saying to take everything and raise it to the power of the inverse of the gamma value, exactly what 'gammaing' actually does. As for blend, I still can't figure it out.

Is it really worth the effort?

After we've been through all this thrive to get our matte, could we let two ugly merges do the compositing? Of course not! So as a finishing touch let's throw in our own custom composite tool, in this case a screen tool - but it is easy to pack up an add or an over. Connect Image1, 2 and 3 of our new CustomTool to the matte CustomTool, the foreground and the background. Wait a moment! We have bg, fg and lightwrap matte, but what about the actual lightwrap pixels - e.g. the blurred background? Too bad CustomTools have only 3 inputs - but there's a clever solution: since we had the blurred background as an input to the matte CustomTool, we can just pass along its RGB values to the composite CustomTool. So we just need to type c3 in the Red, Green and Blue Expressions of the matte CustomTool - this will take values from the corresponding channel (r, g and b) of input Image3 - which was the blurred background - and pass them along to the output - which now feeds input Image1 of the composite CustomTool.

Easy setup

On with the composite: we want to screen the lightwrap (RGB and matte in Image1) over the foreground (Image2)
(1 - ((1 - (c1 * a1)) * (1-c2) ) ) * a2
and then put everything over the background (Image3)
+ c3 * max((a3-a2),0)
We can also feed in another 'premultiplied' control, that when activated will skip the processing of the foreground by its matte; making it 0 for 'unpremultiplied' and 1 for 'premultiplied' and putting it on n1, we'll end up having
(1 - ((1 - (c1*a1)) * (1 -c2))) * max(a2,n1) + c3 * max((a3-a2),0)

I can live with or without you

A few more general tips:
  1. Name your custom tools! If you are like me, as soon as your flow reaches a certain complexity you’ll be adding custom tools one after another. Since they are so versatile it is much more important to name them in a meaningful fashion for the task they accomplish. When you come back to adjust you’ll figure out the expression in a glimpse, if you know beforehand what it is supposed to do.
  2. Merging consecutive custom tools is not always going to speed up the flow, as opposed to what one may think. In the example above, tweaking the matte CustomTool to produce an already lightwrapped foreground would require us to insert the matte calculation in all three RGB channels, making the whole thing slower.
  3. Experiment! You can push things so far as to throw a bunch of still evaluating nonsense in your CustomTool (chances are your supervisor won't ask anyway) and wait for serendipity, or Godot, it depends, you know.

Comments, critics, explanation requests, mercilessly pointing out conceptual or even linguistic errors, all of this is tolerated welcome mandatory. If you really have to Feel free to Do it at pigsfly here.