Unity Stencil Buffer

I recently participated in a project for the Agricultural XR app, and I am in charge of UI and Interaction Design in Unity. This article will talk a bit about the stencil mask I created to display the crop objects while users investigate their fields. I am basically creating all of my shaders using ASE, so this shader is also ASE.

First, I want the box and corn will be masked by the plane, so the user can only inspect the corn through the plane. Thus, we have to assign a material to the plane (Create -> Amplify Shader -> Unlit), and on the left side panel, turn on the stencil buffer (Subshader -> Check on Stencil Buffer). 

* In my case, I wrote the stencil buffer in Pass, not the Subshader. Check more on this document.

In my case, a reference value of 1 is assigned to the stencil mask. The ’Comparison’ is set to Always, and the ’Front-face pass’ is set to Replace. 

- Comparison defines when the stencil operation passes.  The value is Always, which means that no matter what reference value we use, the object will always be drawn as I want the mask can always be seen.

- Front-Face Pass is set to ’Replace’. This will write the reference value (1) into the stencil buffer, marking the plane as visible. The rest of the operators we can leave as default. For more information, check the Unity official document & this video. 

— > The purpose of this setting is that we need to make sure the mask is always visible, yet the object inside the mask should be equal to 1 to be visible through the mask. * Note that the Front-face pass is only for the front side, as now we have set our material to ‘Cull Back’, if we set it to both sides,  it will show more customization options as the image shows below.

Last but not least, we need to adjust the render queue for the stencil mask. By setting it to ‘-1’, the mask’s render queue becomes ‘1999’, which ensures it renders before the geometry (which starts at 2000). This ordering is crucial as we need the mask to be rendered before other elements in the system.

Now we need to mask out the objects that should appear inside the mask. In my case, this includes the corn and the background box.

–> First, after creating a new shader and its corresponding materials, we’ll configure the stencil buffer settings similar to what we did with the mask object.

Remember how we gave the mask a reference number of 1?  –> We’ll need to use this same number (1) for the objects that should appear inside the mask. The key difference is in the Comparison setting - set this to ‘Equal’. This means the pixel/object will only render if its reference value matches the current value in the stencil buffer.

–> Then, set the Pass Front to ‘Keep’ (unlike the mask where we used ‘Replace’). Since the mask has already set the stencil buffer value to 1, using ‘Keep’ maintains this value and ensures the objects remain visible within the masked area.

Boom! Just like that, our stencil mask is ready to go! Now you can see the objects are properly masked - they’re only visible through the mask shape.

Tricky parts come right after…When we rotate our viewport or camera, we can see that the objects outside of the mask can also be seen via the mask, as shown in the image below. Thus, our initial attempt to change the render queue needs to be revised. 

Let’s go back to the render queue to check what other options we have to solve this. From the image below, we can get to know that geometry will be renderer first, see the second image (screenshot of frame debugger), then it renders the object and finally goes to the box container for the object. Which is not what we want for rendering orders. 

–> The first step is to adjust the render queue for all stencil-related objects (both the mask and the objects inside it). We want regular, non-stencil objects to render first, followed by our stencil buffer objects.

To achieve this, I’ve set the render queue to ‘Alpha Test’, which handles opaque objects but renders later than geometry. This ordering works well with our earlier modifications. However, you can manually set the render queue values as long as you maintain the correct rendering order:

The specific queue values or render tags don’t matter as much as maintaining this sequence!

–> Next, what we need to do here is to turn on its Depth Test for the ‘Box Container’–> ’Zwrite Mode’ to ‘ON’ and ‘ZTest Mode’ to ‘Always’. 

 The purpose of these settings is to:

• 1. Configure the box so that it first passes a stencil test—thereby preserving its stencil relationship. 
• 2. Then do a depth test to cull objects occluded behind it, following the normal rendering process. Setting “ZTest Mode” to “Always” forces the box container to bypass standard depth comparisons, guaranteeing it always passes the depth test. 

All Set!

See the image below for the final settings and outcome.