cd /overengineering-a-stream-overlay/

How I overengineered a stream overlay

· by jill

What I realized was, the shader design of running code for every single pixel of an image isn't only applicable when your GPU allows you to do it.

So back in September this year, I decided to stream music to one of my friend groups. The plan of it was: I create a funny-looking overlay in OBS, then go in a Discord VC and stream the layout and music through a fake audio input. It worked well, but the part that this post is about is the layout.

It looked something like this:

A stream layout. It's split into 3 segments: one is a classic vertical bar music visualizer, another is a notepad with timestamps for different albums, and the third one is covered in black with question marks on top. There are also many GIFs on top of the layout, most of them are dancing ones.
It's a mess. I love it.

Ignoring all the mess of dancing GIFs, you'll notice it's literally just tmux with a music visualizer, nano and.. a black spot?

Yeah, so you'll notice all of the software used here isn't original. I slapped some Linux software together in a tmux shell and called it a day. But I wasn't satisfied with it, so I decided to code up a tiny script for the top-right segment.

My mental image of it was this: it'd show the currently playing song, and have some extra fancy visuals if possible. I knew this was possible with MPRIS - a standard D-Bus interface which allows you to control music playing on the system, read its metadata (which is the part we're interested in!) and do this all through any music player on the system.

And since I was already going to broadcast the audio through a music player, this was the prefect solution:

It's a standard-looking music player, specifically Rhythmbox. There are tons of songs up for display.

So of course, logically, I started with the most optional part: the background.

A terminal display. In the background are vague-looking graphics effects comprised out of standard ASCII characters.

The background might look like it's really hard to make, but surprisingly, not really?

So, in short, to render anything on a GPU you need a shader, specifically a pixel shader. What a pixel shader does is go one-by-one on each pixel, runs some code, and puts the resulting color (in form of a 3D vector) on the spot it ran the function for.

For example, in a pseudo-GLSL, you could do something like this:

void main() {
  // uv is a variable that converts the screen into
  // a 2D vector from 0.0 to 1.0
  vec2 uv = gl_FragCoord.xy / resolution.xy;
  return vec3(uv.x, uv.y, 1.0);

And this pseudo-GLSL code will render something like this:

A gradient! In the top left, there's cyan, in the bottom left there's blue, in the bottom right there's purple and in the top right there's white.
It makes sense if you think about it - the bottom-left corner is fully blue because the color value there is 0, 0, 1 - exactly the color of a solid blue.

This may seem weird and unintuitive, but in reality shaders are a lot more powerful than you think. For example, my homepage's background is made entirely through a single shader! It makes sense if you think about it: you start with something simple like this:

Weird-looking purple shapes.

Then, you add a bit of UV distortion, making the image use "wrong" positions for the image:

The weird-looking purple shapes, now heavily distorted and pixelated.

Add some more lines on top, do some color correction, and you're done!

It no longer resembles the shapes, but is still purple, and looks very.. interesting.

I won't dwell on shaders too much, but my point is - they're sick. People have made really impressive stuff with them, and there's an entire website dedicated to archiving, creating and sharing shaders.

You may think, "well, this applies to GPUs, cool, but how does this relate to the music visualizer script"? Well, what I realized was, this design of running code for every single pixel of an image isn't only applicable when your GPU allows you to do it.

A terminal only has a few "pixels" in it - in reality, if you have, say, a 60x30 terminal, that means it'd be able to display 1,800 characters, which means it's a very small canvas for rendering actual images. You could try, but all you get is vague shapes..

I ran viu on an image of She. It doesn't look too good.
Even by cheating and shoving 2 pixels in 1 character by using background and foreground colors, it's.. not a very clear image.

Turns out, vague shapes are exactly what we need for a weird-looking funky background! And since a terminal is so tiny, you can do this pretty efficiently! So I got to work, and made a script that does just that in Node.JS (I still wish I could learn a better language overnight).

The shader I chose was a classic plasma shader - I won't go too in-depth into how they're done, but you can check out this really good article for a better explanation of how one is made.

After you render out a plasma shader (like seen above) to a tiny terminal, you can now convert each color to a character. If you have an array of characters like so:

const plas = [' ', '.', '*', '/', '0'];

You can now assign every single "brightness" value its own character. If an image is fully dark, it'll display blank, letting the background color of the terminal show up. If an image is fully bright, it'll displays as zeroes, which take up more space than the other characters, it'll seem as if the color is brighter, as there's more of the terminal foreground color showing up.

So we end up with a very demoscene-looking terminal display! Exactly what we wanted. And once we slap on the rest of the display, we end up with this!

The plasma shader in this screenshot actually uses 2 layers of plasma shaders and puts them on top of each other!

And that's pretty much it. The MPRIS, metadata handling part of the code isn't very fun to discuss, as it's just D-Bus jank, so I've left it out.


As with all of my other projects, you can check out its source code! I definitely don't think it's very readable or nice-looking, but it gets the job done. The text rendering specifically was a very very jank system to implement, despite being the thing terminals were made to display...

I hope you learnt something fun about computer graphics today. I like them computer graphics. They make me happy. :)