The Problem
If you’ve ever tried to reconstruct dense 4D geometry and motion from basic videos, you know the agony: most solutions need tons of post-processing, calibration, or multi-view setups. Getting a clean, frame-by-frame point map and scene flow from a single video is a mess—especially if you want everything lined up in a consistent world coordinate system.
What This Does
MotionCrafter tackles the headache head-on. Feed in a monocular video (yep, just one camera), and it spits out dense geometry and motion estimates for every frame—all mapped together, no Frankenstein post-optimization required. Under the hood, the motioncrafter/geometrymotionvae.py file is the brains for the 4D VAE, while the motioncrafter/unet.py deals with diffusion-based prediction. The main script, run.py, glues the pipeline together: you call it, it loads models, runs inference, and saves results.
Configs live in the configs/ folder, split by training type (unettrain for diffusion, vaetrain for geometry, etc.). Dataset handling happens in datasets/video.py and datasets/videotransforms.py, which means you don’t have to write your own loader unless you’re a masochist. Visualization is handled separately—just call visualize/visualize.py if you want to see your outputs without hacking matplotlib.
Real-World Use
Let’s say you’ve got a pile of drone footage and want to reconstruct moving objects for simulation or AR. Install the dependencies (requirements.txt), throw your video at run.py:
python run.py --videopath myvideo.mp4 --save_folder output
Need to tweak things? Change resolution, number of frames, or swap in your own model weights via command line flags. Training is pretty standard: organize your dataset as outlined in DATASET.md, then use train.py. The configs in configs/ are ready for tweaking—no hiding settings in some obscure class.
The Bottom Line
MotionCrafter does one thing well: turn boring videos into dense geometry and motion maps, no fuss. Documentation is straightforward, code isn’t buried under layers of abstraction, and it’s not beginner-friendly unless you know your way around PyTorch and video data. If you need high-quality 4D reconstructions, this is solid. If you just want simple optical flow, look elsewhere—this is overkill.