Fake stencil silhouette/outline (object-based but without depth test)

As opposed to most 3D outlines, this one doesn’t care about normals, as evidenced by the jagged model I’m testing it on. It pretty much perfectly outlines the outer edge of the model (or several at once if you so choose). It’s based on Mark Raynsford’s approach to creating the outline from a mask; but since I haven’t found a way to access a depth/stencil buffer only for a given viewport, I’m using Leafshade’s trick: a special layer with no lights to create that mask. This approach has the same downside as Leafshade’s original shader: it completely disrespects depth. So use it if you don’t mind X-Ray outlines.

Setup: See the scene tree on Screenshot 2. You only need to set this up once, regardless of the number of models you want to outline.

– Set camera cull mask to Layer 6 (my fake “stencil buffer”; choose any layer that you aren’t using). Make sure that no lights have that Layer selected. Also add an empty Environment to the camera (to block out your default Environment).

– In Viewport settings, set background to transparent. You’ll probably also want to turn off inputs, and 2D HDR can’t hurt either. This Viewport will essentially become a fake stencil buffer (Screenshot 1).

– On your ColorRect, set Anchors to Full and Input to Ignore. Make this shader its Material. Assign the ViewportTexture from StencilViewport as stencilMask (Godot will ask you to make the material unique). Tweak lineWeight and outlineColor however you like.

– In any node that has access to your viewport (could be its parent, or the viewport itself), add a script to synchronize it with your main camera and viewport. No SubviewportContainers are needed. Here’s my version (from jazzfool’s outline example):

@onready var stencil_viewport : SubViewport = $StencilViewport
@onready var stencil_camera : Camera3D = $StencilViewport/Camera3D

func _process(_delta : float) -> void:
	var viewport := get_viewport()
	var current_camera := viewport.get_camera_3d()

	if stencil_viewport.size != viewport.size:
		stencil_viewport.size = viewport.size

	if current_camera:
		stencil_camera.fov = current_camera.fov
		stencil_camera.global_transform = current_camera.global_transform

Whenever you want an object to be highlighted, add layer 6 (or whichever layer you’re using) to its Layers mask. It doesn’t matter where in the scene tree your model is, but you might want to attach a “test model” to the camera to tweak the outline in-editor and instantly see the results. Highlighting can be done from a script as well, e.g. when you’re selecting that object:

my_mesh_instance.set_layer_mask_value(6, true) # Makes the "stencil subviewport" read this layer, effectively turning the outline on

Hopefully I’ll figure out the spatial version (projecting this on a Quad?) and then perhaps a depth-respecting outline will be possible. Suggestions are welcome. For now, I believe this version should also have its uses.

art credit: Konst. Evans, Toklian

Shader code
// Fake stencil outline by Sithoid
// Viewport logic by Leafshade Interactive https://www.youtube.com/watch?v=yh1Kdr37RmI
// Outline logic by Mark Raynsford https://io7m.com/documents/outline-glsl/

// Put this on a ColorRect that covers the entire screen
// As a StencilMask, pass a texture from a Viewport that only receives a specific layer

shader_type canvas_item;

uniform float lineWeight : hint_range(0.5, 10.0) = 3.0; // How thick the outline is
uniform vec4 outlineColor : source_color = vec4(3.0, 0.8, 0.0, 0.8); // Can be > 1
uniform sampler2D stencilMask : source_color;

void fragment() {
	float dx = (SCREEN_PIXEL_SIZE.x) * lineWeight;
	float dy = (SCREEN_PIXEL_SIZE.y) * lineWeight;
	
	vec2 uvCenter   = vec2(SCREEN_UV.x - dx * 0.5, SCREEN_UV.y + dy * 0.5); // Shift by line size makes line expansion uniform
	vec2 uvRight    = vec2(uvCenter.x + dx, uvCenter.y);
	vec2 uvTop      = vec2(uvCenter.x,      uvCenter.y - dx);
	vec2 uvTopRight = vec2(uvCenter.x + dx, uvCenter.y - dx);
	
	float mCenter   = texture(stencilMask, uvCenter).a;
	float mTop      = texture(stencilMask, uvTop).a;
	float mRight    = texture(stencilMask, uvRight).a;
	float mTopRight = texture(stencilMask, uvTopRight).a;
	
	float dT  = abs(mCenter - mTop);
	float dR  = abs(mCenter - mRight);
	float dTR = abs(mCenter - mTopRight);
	
	float delta = 0.0;
	delta = max(delta, dT);
	delta = max(delta, dR);
	delta = max(delta, dTR);
	
	vec4 outline = vec4(outlineColor.r, outlineColor.g, outlineColor.b, sign(delta) * outlineColor.a);
	COLOR = outline;
}
Tags
highlight, outline, silhouette, Stencil
The shader code and all code snippets in this post are under CC0 license and can be used freely without the author's permission. Images and videos, and assets depicted in those, do not fall under this license. For more info, see our License terms.

More from Sithoid

3D Color Range

Related shaders

Edge detection without depth texture (so you can use it in 2D)

Outline Silhouette Shader

Stencil / Masking in 3D

Subscribe
Notify of
guest

7 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Alexcellent
Alexcellent
16 days ago

Hey! For what version of godot is this shader for ? Testing in 4.3 and this line gives me trouble. :

code
if stencil_viewport.size != viewport.size:
stencil_viewport.size = viewport.size
Isn’t viewport.size readonly in GD 4.X? I’ve tried stencil_viewport.set_size_override(viewport.get_size()) but still problematic
would love to use that shader!

Last edited 16 days ago by Alexcellent
Alexcellent
Alexcellent
11 days ago
Reply to  Sithoid

Oh yeah! Now it totally works – Thanks for your help 🙂

Bry
Bry
10 days ago

I get console error spam when I attach the Viewport texture to the ColorRect:

Attempted to use the same texture in framebuffer attachment and a uniform (set: 1, binding: 1), this is not allowed.

Godot never asked to make the material unique as stated.

Also, it isn’t really clear what the TestModel node is used for, or if it’s even needed at all.

I couldn’t get it to work following the instructions, running on 4.3.

Bry
Bry
10 days ago
Reply to  Bry

Got it working. Mistakenly put the ColorRect inside of the StencilViewport, which caused the error above. As far as I can tell, the “TestModel” node isn’t needed, so I deleted it.

Bry
Bry
10 days ago

For my case I get slightly better results by changing line 38 in the shader to:

COLOR = mix(outline, vec4(outlineColor.r, outlineColor.g, outlineColor.b, 0.0), texture(stencilMask, SCREEN_UV).a);

Doing this prevents some of the overlap of the outline on top of the mesh.