Stencil / Masking in 3D

This is a shader, where you based on colors, can make stencil/ masking effects. The example is 3D, but a similar effect can achieved easy in 2D (same concept).

Possible Use-Cases

Visibility Control
UI Effects (like the Persona games)
Portals / Windows
Stylized Transitions
More!

Setup

ℹ️ In this example I’m going to use Rendering Layer 16 to 20 for Masking effects. You can change that in your project. Also you can only use one Rendering Layer for everything but with different colors.

Your Main Camera(s)

Choose your Camera3D object
In Cull Mask disable all Layers from 16 to 20

Masking Object(s)

Create/ Instance your Mesh and open Material
Create New StandardMaterial3D
In Shading > Shading Mode set it to Unshaded
In Albedo > Color choose a Color of #FF00FF
Under VisualInstance3D activate only Layer 16 (all other should be deactivated)

Create Stencil Mask Viewport

Create Node3D and name it Stencil
Add a Child Element to Stencil from Type SubViewport and name it MaskViewport
- Set Size to what you have in your project settings (like 1280 x 720) – will be overwritten by script later
- Optional: If you want a “see through” effect, in Cull Mask only enable Layer 16
Add a Child Element to MaskViewport from Type Camera3D and name it MaskCamera
- In Environment create a New Environment
- For Background > Mode in Environment set it to Custom Color (and keep standard values)
On Stencil attach a new Script stencil.gd (see Shader code)
- For Target Camera choose your main camera
- For Mask Camera choose MaskCamera
- For Mask Viewport choose MaskViewport

Create Stencil Content

Create a TextureRect and name it StencilContent
In Layout > Anchors Preset choose Full Rect
In Texture add the texture you want to show (like a Noise-Texture or a Viewport Texture)
In Material create New ShaderMaterial
In Resource > Local to Scene set to true
Create New Shader using stencil.gdshader (see Shader code)
- Target Color – The color which is used for Stencil (e.g. #FF00FF)
- Tolerance – To play a litte for the tolerance of color for stencil (try as low as possible)
- Invert Mask – To invert, that not the object self is masked out, but everything else
- Stencil Tex – Stencil texture to use – choose MaskViewport

Inspirations

Shader code

/****************************
stencil.gdshader
****************************/

shader_type canvas_item;

uniform sampler2D stencil_tex;
uniform vec4 target_color : source_color = vec4(1.0, 0.0, 1.0, 1.0);
uniform float tolerance : hint_range(0.0, 1.0);
uniform bool invert_mask = false;

void fragment() {
	vec4 tex_color = texture(stencil_tex, UV);
	float diff = distance(tex_color.rgb, target_color.rgb);

	if (diff <= tolerance) {
		COLOR.a = invert_mask ? 0.0 : 1.0;
	} else {
		COLOR.a = invert_mask ? 1.0 : 0.0;
	}
}

/****************************
stencil.gd
****************************/

@tool
extends Node3D

@export var target_camera: Camera3D = null
@export var mask_camera : Camera3D = null
@export var mask_viewport : SubViewport

var shader_material: ShaderMaterial = null

func _process(delta: float) -> void:
	if Engine.is_editor_hint():
		var editor_viewport = EditorInterface.get_editor_viewport_3d(0)
		var editor_camera = editor_viewport.get_camera_3d();
		
		mask_camera.global_transform = editor_camera.global_transform
		mask_camera.fov = editor_camera.fov
		mask_viewport.size = editor_viewport.size
		
	elif target_camera != null:
		mask_camera.global_transform = target_camera.global_transform
		mask_camera.fov = target_camera.fov
		mask_viewport.size = get_viewport().size