The Problem

Video matting and object removal traditionally require task-specific training datasets and model fine-tuning. This repository addresses the gap by enabling training-free video manipulation using pre-trained video diffusion models, leveraging their learned priors for object removal, foreground extraction, and layer composition without any task-specific training.

What This Does

OmnimatteZero implements three core capabilities using pre-trained video diffusion models:

  • Object Removal (object_removal.py): Removes objects and their effects (shadows, reflections) from videos by inpainting masked regions through the diffusion process. The function round_to_nearest_resolution_acceptable_by_vae ensures latent dimensions align with VAE constraints.
  • Foreground Extraction (self_attention_map.py): Extracts foreground layers with associated effects. Key functions include _find_attention_modules and _create_hook which register hooks to capture attention maps, and pre_hook which processes attention before model forward passes.
  • Layer Composition (foreground_composition.py): Composites extracted foreground layers onto new backgrounds. The MyAutoencoderKLLTXVideo class and forward/forward_encode methods handle video encoding/decoding through the autoencoder.

The repository requires CUDA-capable GPU (32GB+ VRAM recommended) and PyTorch 2.4+. Dependencies are declared in requirements.txt without a lockfile, meaning builds are not reproducible without manual pinning.

How It Is Wired

Execution flows through a structured call graph of 33 internal edges across 5 Python modules. The entry point is OmnimatteZero.py, which defines the OmnimatteZero class and my_call function. Key routing paths:

  • register_hooks (called from 2 places) → _find_attention_modules → _create_hook → apply_guidance → compute_guidance_matrix → get_guidance_scale_for_step
  • precompute_correspondences → compute_correspondences → _tracks_to_latent_positions (called twice)
  • remove_hooks is called from 2 places for cleanup

The module with the widest blast radius is attention_guidance.py (18 functions, 4 classes), which defines load_model, compute_correspondences, _generate_grid_query_points, and _fallback_correspondences. This file is imported by OmnimatteZero.py and serves as the primary guidance computation hub. foreground_composition.py is another high-traffic route with its forward and forward_encode methods handling video tensor transformations through the autoencoder.

Notably, there are no circular dependencies across the 5 internal modules, and attention_guidance has zero incoming import edges (Ca=0), meaning it is a root module that other components depend on but does not import from them.

How To Use It

Setup: Clone the repository and install dependencies:

git clone https://github.com/moses-y/OmnimatteZero.git
cd OmnimatteZero
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Configuration: No environment variables are documented. Configuration parameters are embedded directly in the source files — object_removal.py contains base_dir, expected_height, expected_width, and num_inference_steps; self_attention_map.py accepts --video_folder CLI argument.

Running it:

  • Object removal: python object_removal.py — processes example videos from example_videos/ and saves results to results/
  • Self-attention map generation: python self_attention_map.py --video_folder ./example_videos/your_video_name — generates total_mask from object_mask using diffusion model attention patterns
  • Full pipeline: The OmnimatteZero class in OmnimatteZero.py orchestrates the complete matting/compositing workflow

Data preparation: Input videos must be organized under example_videos/ with video.mp4, object_mask.mp4, and total_mask.mp4 per folder. Masks can be generated using SAM2 if not already available.

Real-World Use

In a production pipeline, this repo could serve as a training-free prototype for object removal in video post-production, where quick iteration matters more than perfect consistency. For example, a compositor could remove an unwanted object from footage and composite a replacement layer onto a new background without collecting and training on custom matting data. The training-free approach reduces data collection overhead but inherits whatever biases and artifacts exist in the underlying diffusion model — particularly around temporal coherence in generated inpainting.

Code Health & Issues

Static analysis identified two substantive findings:

  • [HIGH/cognitive_load] Deep nesting across 3 files (OmnimatteZero.py, attention_guidance.py, foreground_composition.py): Max indentation depth 9 makes control flow hard to follow. Fix: flatten with early returns/guard clauses and extract inner blocks.
  • [MEDIUM/resilience] Broad exception handling in object_removal.py: Bare or Exception-wide except clauses swallow errors indiscriminately. Fix: catch specific exceptions; re-raise or log the rest.

SDLC observations from the file structure: no test files, no CI/CD pipeline, no lockfile (requirements.txt only), and no Dockerfile. A license file is present but Dependabot/Renovate is not configured, leaving dependency updates manual and advisories unpatched until manual audit.

The Bottom Line

This is a well-scoped, technically credible implementation of training-free video matting that leverages existing diffusion model priors effectively. The code is functional but suffers from deep nesting that raises cognitive load when modifying guidance or composition logic, and the absence of a lockfile or CI gate means dependency updates carry regression risk. It's suitable for researchers and prototyping teams that have access to 32GB+ VRAM and need quick video manipulation without training pipelines, but production teams should address the nesting and resilience issues before integrating into broader workflows.