Variations on 2D outlines

Sometimes, you may need or want your outlines to be inside your sprites instead of expanding outward. Vector drawing programs often offer the following options: outer, inner, or in-and-out stroke.

In this lesson, we’re going to see how to produce the last two types of outlines.

Inner outline

To draw a stroke inside of the sprite instead of the outside, we have to use texture samples differently. Instead of taking their total, we instead take the inverse of their product. This gives us an alpha value where pixels along edges become highlighted and any that are in the middle become transparent.

float outline = texture(TEXTURE, UV + vec2(-size.x, 0)).a;
outline *= texture(TEXTURE, UV + vec2(0, size.y)).a;
outline *= texture(TEXTURE, UV + vec2(size.x, 0)).a;
outline *= texture(TEXTURE, UV + vec2(0, -size.y)).a;
outline *= texture(TEXTURE, UV + vec2(-size.x, size.y)).a;
outline *= texture(TEXTURE, UV + vec2(size.x, size.y)).a;
outline *= texture(TEXTURE, UV + vec2(-size.x, -size.y)).a;
outline *= texture(TEXTURE, UV + vec2(size.x, -size.y)).a;
outline = 1.0 - outline;

We grab the original sprite color and interpolate it with the line color. Though instead of taking the difference between the outline and the original alpha, we take its product to clip any pixels that lie outside the sprite.

vec4 color = texture(TEXTURE, UV);
vec4 outlined_result = mix(color, line_color, outline * color.a);

But if we output this color as the result and make the line color transparent, it would inline our sprite with a transparent color and hide parts of it. To fix this, we interpolate the result with the original color a second time, using the result’s alpha as the delta.

COLOR = mix(color, outlined_result, outlined_result.a);

This gives us the following final shader code:

shader_type canvas_item;

uniform vec4 line_color : hint_color = vec4(1);
uniform float line_thickness : hint_range(0, 10) = 1.0;

void fragment() {
    vec2 size = TEXTURE_PIXEL_SIZE * line_thickness;

    float outline = texture(TEXTURE, UV + vec2(-size.x, 0)).a;
    outline *= texture(TEXTURE, UV + vec2(0, size.y)).a;
    outline *= texture(TEXTURE, UV + vec2(size.x, 0)).a;
    outline *= texture(TEXTURE, UV + vec2(0, -size.y)).a;
    outline *= texture(TEXTURE, UV + vec2(-size.x, size.y)).a;
    outline *= texture(TEXTURE, UV + vec2(size.x, size.y)).a;
    outline *= texture(TEXTURE, UV + vec2(-size.x, -size.y)).a;
    outline *= texture(TEXTURE, UV + vec2(size.x, -size.y)).a;
    outline = 1.0 - outline;

    vec4 color = texture(TEXTURE, UV);
    vec4 outlined_result = mix(color, line_color, outline * color.a);
    COLOR = mix(color, outlined_result, outlined_result.a);
}

In-and-out outline

Having both inline and outline is simple in concept; we mix the two results. But to avoid sampling twice as many times, we need to use the samples in different ways.

We separate the texture samples from the math, putting them into individual variables. Because we’re dealing with two lines instead of one, we also divide the thickness by two so that an in-outline is the same thickness as an inline or outline of the same value.

vec2 size = TEXTURE_PIXEL_SIZE * line_thickness / 2.0;

float l = texture(TEXTURE, UV + vec2(-size.x, 0)).a;
float u = texture(TEXTURE, UV + vec2(0, size.y)).a;
float r = texture(TEXTURE, UV + vec2(size.x, 0)).a;
float d = texture(TEXTURE, UV + vec2(0, -size.y)).a;
float lu = texture(TEXTURE, UV + vec2(-size.x, size.y)).a;
float ru = texture(TEXTURE, UV + vec2(size.x, size.y)).a;
float ld = texture(TEXTURE, UV + vec2(-size.x, -size.y)).a;
float rd = texture(TEXTURE, UV + vec2(size.x, -size.y)).a;

vec4 color = texture(TEXTURE, UV);

The math for both remains the same, just summed up into separate lines. An outer stroke is the sum of all those samples, clamped to 1.0. An inner one is the inverse product of those samples.

We also change the way we use the original color’s alpha for each operation here. The result gives us an alpha value for both the outline and the inline. Unless both are 0.0, one will be 1.0 while the other will be 0.0, so we can sum them together to get a combination.

float outline = min(1.0, l+r+u+d+lu+ru+ld+rd) - color.a;
float inline = (1.0 - l * r * u * d * lu * ru * rd * ld) * color.a;

vec4 outlined_result = mix(color, line_color, outline + inline);

Like the inline shader, if we use this color as the result, a transparent line color will make the sprite’s inner edge transparent instead of replacing the color, so we have to mix this color with the result’s alpha.

COLOR = mix(color, outlined_result, outlined_result.a);

Here is the final shader code:

shader_type canvas_item;

uniform vec4 line_color : hint_color = vec4(1);
uniform float line_thickness : hint_range(0, 10) = 1.0;

void fragment() {
    vec2 size = TEXTURE_PIXEL_SIZE * line_thickness / 2.0;

    float l = texture(TEXTURE, UV + vec2(-size.x, 0)).a;
    float u = texture(TEXTURE, UV + vec2(0, size.y)).a;
    float r = texture(TEXTURE, UV + vec2(size.x, 0)).a;
    float d = texture(TEXTURE, UV + vec2(0, -size.y)).a;
    float lu = texture(TEXTURE, UV + vec2(-size.x, size.y)).a;
    float ru = texture(TEXTURE, UV + vec2(size.x, size.y)).a;
    float ld = texture(TEXTURE, UV + vec2(-size.x, -size.y)).a;
    float rd = texture(TEXTURE, UV + vec2(size.x, -size.y)).a;

    vec4 color = texture(TEXTURE, UV);
    float outline = min(1.0, l+r+u+d+lu+ru+ld+rd) - color.a;
    float inline = (1.0 - l * r * u * d * lu * ru * rd * ld) * color.a;

    vec4 outlined_result = mix(color, line_color, outline + inline);
    COLOR = mix(color, outlined_result, outlined_result.a);
}