Fake stencil silhouette/outline (object-based but without depth test)
As opposed to most 3D outlines, this one doesn’t care about normals, as evidenced by the jagged model I’m testing it on. It pretty much perfectly outlines the outer edge of the model (or several at once if you so choose). It’s based on Mark Raynsford’s approach to creating the outline from a mask; but since I haven’t found a way to access a depth/stencil buffer only for a given viewport, I’m using Leafshade’s trick: a special layer with no lights to create that mask. This approach has the same downside as Leafshade’s original shader: it completely disrespects depth. So use it if you don’t mind X-Ray outlines.
Setup: See the scene tree on Screenshot 2. You only need to set this up once, regardless of the number of models you want to outline.
– Set camera cull mask to Layer 6 (my fake “stencil buffer”; choose any layer that you aren’t using). Make sure that no lights have that Layer selected. Also add an empty Environment to the camera (to block out your default Environment).
– In Viewport settings, set background to transparent. You’ll probably also want to turn off inputs, and 2D HDR can’t hurt either. This Viewport will essentially become a fake stencil buffer (Screenshot 1).
– On your ColorRect, set Anchors to Full and Input to Ignore. Make this shader its Material. Assign the ViewportTexture from StencilViewport as stencilMask
(Godot will ask you to make the material unique). Tweak lineWeight
and outlineColor
however you like.
– In any node that has access to your viewport (could be its parent, or the viewport itself), add a script to synchronize it with your main camera and viewport. No SubviewportContainers are needed. Here’s my version (from jazzfool’s outline example):
@onready var stencil_viewport : SubViewport = $StencilViewport
@onready var stencil_camera : Camera3D = $StencilViewport/Camera3D
func _process(_delta : float) -> void:
var viewport := get_viewport()
var current_camera := viewport.get_camera_3d()
if stencil_viewport.size != viewport.size:
stencil_viewport.size = viewport.size
if current_camera:
stencil_camera.fov = current_camera.fov
stencil_camera.global_transform = current_camera.global_transform
Whenever you want an object to be highlighted, add layer 6 (or whichever layer you’re using) to its Layers mask. It doesn’t matter where in the scene tree your model is, but you might want to attach a “test model” to the camera to tweak the outline in-editor and instantly see the results. Highlighting can be done from a script as well, e.g. when you’re selecting that object:
my_mesh_instance.set_layer_mask_value(6, true) # Makes the "stencil subviewport" read this layer, effectively turning the outline on
Hopefully I’ll figure out the spatial version (projecting this on a Quad?) and then perhaps a depth-respecting outline will be possible. Suggestions are welcome. For now, I believe this version should also have its uses.
art credit: Konst. Evans, Toklian
Shader code
// Fake stencil outline by Sithoid
// Viewport logic by Leafshade Interactive https://www.youtube.com/watch?v=yh1Kdr37RmI
// Outline logic by Mark Raynsford https://io7m.com/documents/outline-glsl/
// Put this on a ColorRect that covers the entire screen
// As a StencilMask, pass a texture from a Viewport that only receives a specific layer
shader_type canvas_item;
uniform float lineWeight : hint_range(0.5, 10.0) = 3.0; // How thick the outline is
uniform vec4 outlineColor : source_color = vec4(3.0, 0.8, 0.0, 0.8); // Can be > 1
uniform sampler2D stencilMask : source_color;
void fragment() {
float dx = (SCREEN_PIXEL_SIZE.x) * lineWeight;
float dy = (SCREEN_PIXEL_SIZE.y) * lineWeight;
vec2 uvCenter = vec2(SCREEN_UV.x - dx * 0.5, SCREEN_UV.y + dy * 0.5); // Shift by line size makes line expansion uniform
vec2 uvRight = vec2(uvCenter.x + dx, uvCenter.y);
vec2 uvTop = vec2(uvCenter.x, uvCenter.y - dx);
vec2 uvTopRight = vec2(uvCenter.x + dx, uvCenter.y - dx);
float mCenter = texture(stencilMask, uvCenter).a;
float mTop = texture(stencilMask, uvTop).a;
float mRight = texture(stencilMask, uvRight).a;
float mTopRight = texture(stencilMask, uvTopRight).a;
float dT = abs(mCenter - mTop);
float dR = abs(mCenter - mRight);
float dTR = abs(mCenter - mTopRight);
float delta = 0.0;
delta = max(delta, dT);
delta = max(delta, dR);
delta = max(delta, dTR);
vec4 outline = vec4(outlineColor.r, outlineColor.g, outlineColor.b, sign(delta) * outlineColor.a);
COLOR = outline;
}
Hey! For what version of godot is this shader for ? Testing in 4.3 and this line gives me trouble. :
stencil_viewport.size = viewport.size
would love to use that shader!
Hmm! Sorry for not noticing the comment earlier. It is indeed for 4.3. My viewport size was set manually so this check never passed, but I’ve double-checked it:
Output:
The docs state “If the parent node is a SubViewportContainer and its SubViewportContainer.stretch is true, the viewport size cannot be changed manually.” So it is read-only only under those conditions. My subviewport is parented to a plain Node, that must be why it’s working.
Oh yeah! Now it totally works – Thanks for your help 🙂
I get console error spam when I attach the Viewport texture to the ColorRect:
Attempted to use the same texture in framebuffer attachment and a uniform (set: 1, binding: 1), this is not allowed.
Godot never asked to make the material unique as stated.
Also, it isn’t really clear what the TestModel node is used for, or if it’s even needed at all.
I couldn’t get it to work following the instructions, running on 4.3.
Got it working. Mistakenly put the ColorRect inside of the StencilViewport, which caused the error above. As far as I can tell, the “TestModel” node isn’t needed, so I deleted it.
Glad it worked out! You’re right, TestModel is only used to see & tweak the outline in the editor. I’ve edited the description to make it clearer.
For my case I get slightly better results by changing line 38 in the shader to:
COLOR = mix(outline, vec4(outlineColor.r, outlineColor.g, outlineColor.b, 0.0), texture(stencilMask, SCREEN_UV).a);
Doing this prevents some of the overlap of the outline on top of the mesh.