What is this?

I wanted to use OBS to overlay my webcam on a RTSP stream but I didn't want to spend money on a green screen for chroma keying. So I found a way to use all this expensive computer hardware that is lying around instead of spending money on frabric.

But really what is this?

A simple webpage that will take webcam input, use tensorflow+bodypix to segment the image in two parts (background and person) and then set the background to a generic green screen for chroma keying later. It's not perfect, but it's pretty good.

See references below for a detailed explaination on how this works.

This is similar to how other apps achieve the same effect (Zoom, Teams, Nvidia Broadcast RTX thing, Xsplit, Chromacam) but free and run entirely in browser. Nvidia Broadcast is a popular option but this works on non-rtx cards and on macos.

The intended use for this was to feed in to OBS and utilize the chroma key plugin to overlay video there.

Stand alone app

There is a stand alone app version here which allows you to place this on the desktop and drag it around, the transparent background works right on the desktop. With the stand alone app running you could just capture the whole desktop image in your tool of choice (obs, zoom, etc...) without having to chroma key the video out later.

Select GPU

You can increase the speed in Windows hosts with multiple GPUs by selecting the faster device with this guide. With the default settings on my 2018 XPS 15 (integrated GPU + Nvidia 1050), segmentation speed will increase from 15fps to 30fps. The same guide can be followed for the standalone app or the web browser used to open this page.

How to use in OBS

Step 1: Open this website and make it a small window around the video, press 'b' to cycle through backgrounds to green or blue. On another screen or window open the app you'd like to capture like so:

alt text

Step 2: Open OBS

Step 3: Create a new window or desktop capture source to capture for your video background:

Step 4: Create a new window capture source, use the chromabro window (this window) as the source:

Step 5: Add a crop filter to the chromabro window capture:

Step 6: Add a chroma key filter to the chromabro window capture:

You should now see it working within OBS. Take note of the Notes below however.

Step 7: Set your layout

Step 8: Start streaming, recording, or virtual camera-ing! It should look like this

Step 9: Profit

Notes

The mask is slower to render than the video so it will lag on sudden movements.
Part of the browser running this must be visible on screen for the webcam to keep working. You can move it most of the way off screen, but not all the way or cover it up with another window. It will work for a few seconds and then pause.
Some backgrounds can confuse it, it's best to have a boring background like a wall (in my case it confuses a jacket hanging behind me with a bit of me, but it's pretty good)
Window capture in OBS is slow in macos (works pretty well in windows in my test) so you might want to capture the webcam via a desktop capture and crop that

Questions?

Use the links at the bottom of my website to contact me for any questions/issues/updates.

If you find yourself using it maybe you could swing some coffee money my way...

References

https://blog.francium.tech/edit-live-video-background-with-webrtc-and-tensorflow-js-c67f92307ac5

https://github.com/tensorflow/tfjs-models/blob/master/body-pix/README-v1.md

https://blog.tensorflow.org/2019/11/updated-bodypix-2.html

https://jameshfisher.com/2020/09/23/running-bodypix-on-a-video-stream/