Edge Enhancement Shader
During my third project at The Game Assembly the group I was in knew they wanted a cartoon-esque style for our game and we agreed that we needed a outline for our game. Initially I was thinking we could use the method where you draw the models twice and invert the normal of the second mode to create an outline but since the game needed in on most objects in the scene, including intersections between the floor and the walls, this meant we needed a edge detection shader.
The biggest challenge of this was that we used textures atlases for the textures for most of the objects, while other objects had normal maps, this meant that I could not only use normal information since it would result in the shader either creating too much noise in certain objects or missing certain intersections. After some testing I decided to use a the method that William Chyr, developer of Manifold Garden, described and tested on his development blog (read about it here). I would also like to thank Wizard Loaf’s graphics programmer, Sebastian Grape, who helped me iterate and implement the shader in our game.
Manifold Garden edge detection
The basic idea of the shader is to sample the normal data[Figure 1] and depth data[Figure 2] from for the pixel and then lerp between the original color and the color of the edge based on the length of the original color minus the sampled value. The step is there to be able to control the threshold of the edge and control how sensitive the edge detection is.
return lerp(aOriginalColor, aEdgeColor, 40.f * step(0.01f, length(myDepthNormal - mySampledColor)));
The variable mySampledValue is a float4 with the normal data in the RGB-channels and in the A-channel the depth is stored.
After changing the bias of the depth and the normal in the in the lerp it produces a good outline that is consistent in it’s thickness [See Figure 3].