The Problem

Video understanding typically treats frames as independent images, losing the continuous motion that defines a scene. Reconstructing 4D structure (3D + time) from video usually requires multi-stage pipelines—optical flow, depth estimation, and point tracking run separately and are hard to fuse. Trace Anything collapses this into a single forward pass by predicting a continuous, parametric 3D trajectory for every pixel across all frames.

What This Does

Trace Anything introduces a trajectory field: a continuous function that maps each pixel to its 3D position over time. The model outputs B-spline coefficients that define these trajectories, enabling dense, temporally-coherent 4D reconstruction from a video, an image pair, or an unstructured image set.

The repo is a PyTorch inference implementation. The core model lives in trace_anything/trace_anything.py, with building blocks in trace_anything/layers/ (patch embedding, positional encoding, transformer blocks, DPT decoder). The scripts/ directory contains three entry points: infer.py for batch inference, view.py for the interactive 3D viewer, and user_mask.py for user-guided mask refinement. Example inputs are under examples/input/.

How It Is Wired

Execution starts at main in scripts/infer.py:272, which reaches 47 functions. The shortest path to leaving the process is main -> run (at infer.py:186), which writes files via os.makedirs and runs the model via model.forward. The pipeline is: load config (_load_cfg), load images (_load_images), build the model (_build_model_from_cfg), run inference, then post-process outputs.

The most-connected module is trace_anything/trace_anything.py (2 importers, 4 imports, instability 0.67). It contains the core forward pass and B-spline evaluation (evaluate_bspline_conf, _compute_bspline_basis). trace_anything/layers/blocks.py is a stable leaf (2 importers, 0 imports) holding reusable transformer blocks. The highest blast radius for changes is _pretty (called from 3 places) and ensure_dir (2 places) in the scripts—both are trivial helpers but widely used.

File-by-file:

  • scripts/infer.py — orchestrates the full inference run.
  • scripts/view.py — interactive 3D visualization; owns to_numpy, ensure_dir, hsv_colormap.
  • scripts/user_mask.py — mask-guided refinement; heavy branching (45 branch points over 143 lines).
  • trace_anything/trace_anything.py — core model, B-spline trajectory prediction.
  • trace_anything/heads.py — output heads for dense depth and confidence.
  • trace_anything/layers/ — building blocks: blocks.py, dpt_block.py, patch_embed.py, pos_embed.py.

The module graph has no cycles, which keeps refactoring straightforward.

How To Use It

git clone https://github.com/moses-y/TraceAnything
cd TraceAnything
conda create -n trace_anything python=3.10
conda activate trace_anything
pip install einops omegaconf pillow opencv-python viser imageio matplotlib torchvision

Download the pretrained model from Hugging Face and place it at checkpoints/trace_anything.pt. Run inference with python scripts/infer.py --config configs/eval.yaml. The configs/eval.yaml file controls scene paths and output settings. Note: the README specifies a single GPU with ≥48 GB VRAM, so this is not a laptop-friendly workload.

Real-World Use

A VFX studio needs to track a moving object across a handheld shot. Feed the video frames into scripts/infer.py, get dense 3D trajectories for every pixel, then import the output into a compositing tool to attach 3D elements that stay locked to the footage. The interactive viewer in scripts/view.py lets an artist inspect and correct trajectories before export.

Code Health & Issues

Static analysis (not opinion) found 6 issues: 1 high, 5 medium.

  • High – Deep nesting – scripts/user_mask.py, scripts/view.py: max indentation depth 10 makes control flow hard to follow. Flatten with early returns.
  • Medium – Duplicated code blocks – 11 repeated 6-line blocks across 11 files. Extract shared helpers.
  • Medium – Broad exception handling – scripts/user_mask.py, scripts/view.py: bare except swallows errors. Catch specific exceptions.
  • Medium – High branching density – scripts/user_mask.py: 45 branch points over 143 lines. Consider table-driven dispatch.

SDLC observations from the file structure: no test suite (11 source files, 0 test files), no CI pipeline (no .github/ or CI config), and one large binary (assets/interactive_monkeys.gif, 8.6 MB) that should be moved to Git LFS.

The Bottom Line

This is a clean, focused inference implementation of a novel research method with no circular dependencies and a small, understandable codebase. It is not production-ready: no tests, no CI, and heavy VRAM requirements limit it to research or high-end GPU environments. Use it if you need to evaluate or extend the trajectory-field approach; harden it before putting it behind any service.