OmnimatteZero: Training-Free Video Matting with Diffusion
The Problem
Video editing is a pain. You want to remove an object from a video, and suddenly you're dealing with frame-by-frame rotoscoping, shadow cleanup, and alignment issues. Even if you're fancy and use machine learning, most tools require you to train a custom model, which is a hard "no" for anyone who doesn't have a server farm in their garage. OmnimatteZero tries to fix that by using pre-trained video diffusion models to handle object removal, foreground extraction, and layer composition—all without training.
What This Does
OmnimatteZero is a set of Python scripts built around pre-trained video diffusion models. It works with three core tasks: objectremoval, foregroundcomposition, and generating attention-based masks with selfattentionmap.py. Everything is training-free, so it’s plug-and-play as long as you already have your input videos and masks.
The repo is simple: input data goes into examplevideos/, results come out in results/. For example, if you want to remove an object from a video, you’d throw the video along with an object mask (objectmask.mp4) and a total mask (totalmask.mp4) into examplevideos/. Then, you run python objectremoval.py. The script uses the diffusion model to inpaint the masked regions, outputs a clean video, and dumps it into results/.
If you don’t have a totalmask (which includes shadows/reflections), there’s selfattentionmap.py. This script generates it by extracting attention maps from the diffusion model's transformer layers. It’s not magic, but it’s a solid hack.
Real-World Use
Let’s say you’re editing a promotional video but your client suddenly hates the swan in the background (creative decisions, am I right?). You’d organize your files like this:
Run:
You’ll get results/swanlake.mp4 with the swan and its reflection gone, no training or manual editing. If you only have the objectmask, run:
It’ll generate the total_mask for you.
The Bottom Line
OmnimatteZero is great if you want high-quality video matting without the headache of training models. That said, it’s GPU-heavy (32GB VRAM recommended) and not exactly beginner-friendly—this is for people who are already comfortable with Python, video processing, and diffusion models. If that’s you, this repo will save you a lot of time. If not, maybe stick to After Effects.