Masking

Post-processing doesn’t always mean altering the entire screen. In our dissolve tutorial, we looked at using a mask to control how the object would dissolve by applying a noise texture or gradient to manipulate the shader. Screen shaders can do that too.

Let’s set up a simple X-ray shader as an example. It uses multiple viewports and a mask.

Setting the scene up

We need three viewports: what you see when the X-ray isn’t on, what the x-ray reveals, and what shape the X-ray is. We’ll be combining three textures in one final shader so we’ll use viewport textures.

1. Make a new User Interface scene.
2. Add three ViewportContainers to the root of the scene. Call them MainView, XRayView, and MaskView.
   • Set their Stretched property to enabled, and their Right and Bottom anchors to 1 to take up the screen. Set each container’s Self Modulate alpha to 0 to make them invisible.
3. Add a Viewport to each container.
4. Add a TextureRect to the root of the scene and call it Presentation.
   1. Assign a new ViewportTexture to its Texture property, and point it to MainView/Viewport.
   2. Set its Flip V property to true.

Main View

The MainView is where you’d keep your gameplay scene and actors.

In our example, the MainView has a simple scene as a child of the Viewport named MainScene. It only has a sprite and a point of origin for our X-ray scope which is animated using an animation player.

We also use RemoteTransform2D nodes which control the position of objects in the XRayView. These synchronize our two worlds together across different viewports by giving them paths to another node in the hierarchy.

XRay View

The x-ray view has everything you see when underneath the mask.

In our example, we just have an animated gear train. The RemoteTransform2D which controls its position is a child of the Robi sprite in the MainScene.

To have both the MainView and the XRayView visible inside the x-ray area, turn on the Viewport’s Transparent Bg property. Otherwise, you’ll replace the main view entirely, and the background color will be part of the texture.

Mask View

Masks that have one purpose are typically black and white images. The mask view shows only where the mask is white, and hides itself where it’s black.

Our example consists of a single Node2D that implements _draw() to draw a white cone pointed to the right named Cone. It’s a child of the MaskView Viewport. Making it’s script a tool shows it in the editor.

tool
extends Node2D

func _draw() -> void:
    draw_polygon(
        [Vector2.ZERO, Vector2(2000, 500), Vector2(2000, -500)], [Color.white]
    )

The RemoteTransform2D attached to the Watcher in the MainView controls the Cone’s position and rotation.

Make sure you turn on Transparent Bg on the Viewport because we want anything that isn’t the mask to be hidden so it should be black or transparent.

Note: The other way of making the viewport’s background black is to have a full screen black ColorRect as the top most node.

Presentation

The TextureRect is where we bring it all together. If you recall, we named it Presentation.

Add a new ShaderMaterial and assign it a new Shader. Save it to your filesystem as xray_mask.shader.

We need space for the x-ray and mask textures, so declare those two uniforms.

shader_type canvas_item;

uniform sampler2D alternate_viewport;
uniform sampler2D mask_viewport;

Dimming the rest of the screen helps contrast where the scope is highlighting. We’ll control how dark it should get with another uniform with a range from 0% to 100% dimmed.

uniform float dim_main_view : hint_range(0, 1) = 0.2;

The first step is to sample all three textures because they have all the information we need. The mask is black and white, so we extract one of the colors - red in this case - and have it as a floating point value instead of a color.

void fragment() {
    vec4 main_color = texture(TEXTURE, UV);
    float mask_color = texture(mask_viewport, UV).r;
    vec4 alternate_color = texture(alternate_viewport, UV);
}

To figure out the final color, we need to decide on the output:

• Where the mask isn’t, it should be the main color, but darker.
• Where the mask and the alternate color both co-exist, it should be the alternate color.
• Where the mask is but where there is no alternate color, it should be the main color.

We can use basic math to get this result. First, we calculate the darker main color where the mask isn’t. The mask is between 0.0 and 1.0, so we invert it with 1.0 - mask_color, and multiply it with the main color. Everything being dim is done by building a color and multiplying it with the mask inverse.

This might bring parts of the color to below 0, so we’ll clamp it at the very end.

vec4 out_color = main_color * (1.0 - mask_color);
vec4 dim_color = vec4(vec3(dim_main_view), 0.0) * (1.0 - mask_color);

COLOR = clamp(out_color - dim_color, 0, 1);

We then add the alternate color to the main color.

out_color += alternate_color * mask_color;

But we also want to show the main color and not replace it. To do this we only mask where the alternate color exists. Since it’s a transparent texture, we can use the alpha for that. Instead of taking the inverse of the mask, we take the inverse of the product of the alternate color’s alpha and the mask.

vec4 out_color = main_color * (1.0 - alternate_color.a * mask_color);

Which brings the final code:

shader_type canvas_item;

uniform sampler2D alternate_viewport;
uniform sampler2D mask_viewport;
uniform float dim_main_view : hint_range(0, 1) = 0.2;

void fragment() {
    vec4 main_color = texture(TEXTURE, UV);
    float mask_color = texture(mask_viewport, UV).r;
    vec4 alternate_color = texture(alternate_viewport, UV);

    vec4 out_color = main_color * (1.0 - alternate_color.a * mask_color);
    out_color += alternate_color * mask_color;
    vec4 dim_color = vec4(vec3(dim_main_view), 0.0) * (1.0 - mask_color);

    COLOR = clamp(out_color - dim_color, 0, 1);
}