The Problem
Standard 3D reconstruction pipelines require either dense ground-truth annotations or expensive pretrained priors from large-scale datasets. This limits applicability in domains where labeled 3D data is scarce or unavailable, and hand-crafted priors may not generalize to arbitrary object categories.
What This Does
NAS3R implements a self-supervised feed-forward framework that jointly learns explicit 3D geometry and camera parameters from image sequences alone. The core architecture resides in src/model/, featuring encoder backbones (croco, vggt, maskedcroco) that feed into a splatting-based decoder (src/model/decoder/cudasplatting.py, src/model/decoder/decodersplattingcuda.py). View sampling logic is modular across src/dataset/viewsampler/, with configurations for bounded, arbitrary, and evaluation samplers. The training loop is orchestrated from src/main.py with experiment configs in config/experiment/. A C++/CUDA extension (src/model/encoder/backbone/croco/curope/) handles patch embedding operations, built via src/model/encoder/backbone/croco/curope/setup.py.
How To Use It
Setup: git clone --recurse-submodules git@github.com:ranrhuang/NAS3R.git cd NAS3R conda create -n nas3r python=3.11 -y && conda activate nas3r pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121 pip install -r requirements.txt --no-build-isolation pip install -e submodules/diff-gaussian-rasterization --no-build-isolation
Training (2-view, re10k): python -m src.main +experiment=nas3r/random/re10k wandb.m
Pretrained checkpoints (re10knas3r.ckpt, multiview variants) are hosted on Hugging Face under RanranHuang/NAS3R. Configure experiments via YAML files in config/experiment/nas3r/.
Real-World Use
A robotics perception pipeline could ingest monocular video streams and reconstruct dense 3D geometry without camera calibration labels. The framework's novel view synthesis capability enables view-dependent rendering from sparse inputs, useful for sensor fusion or simulators requiring diverse viewpoints. The eval_pose.py and src/evaluation/ modules provide quantitative pose estimation metrics on held-out datasets.
Code Health & Issues
No CI/CD pipeline - .github/ directory and CI config are absent; no automated build/test gate exists to validate changes on push. Dependencies declared without lockfile - requirements.txt pins package versions but lacks a requirements.lock or similar, risking non-reproducible builds across environments. C++ extension build dependency - the curope CUDA extension requires manual compilation via setup.py; prebuilt binaries are present in the repo but may not match all system configurations. 4 test files present - tests/ directory contains basic test coverage, but scope and integration depth appear limited.
The Bottom Line
NAS3R delivers a functional self-supervised 3D reconstruction pipeline that avoids ground-truth dependencies, with modular view sampling and splatting-based rendering. The codebase is structurally coherent and well-organized, but the absence of CI, a dependency lockfile, and documented build procedures for the CUDA extension raises operational friction for production adoption. Best suited for research prototyping or teams comfortable managing manual build steps.