SpatialLM

The Problem

SpatialLM aims to train large language models that can ingest 3‑D point‑cloud data (from monocular video, RGBD, or LiDAR) and output structured scene understanding – walls, doors, windows, and oriented object bounding boxes with semantic labels. The pain point is that most existing 3D‑LLM pipelines either require specialized collection hardware or produce unstructured outputs, forcing downstream users to do costly manual annotation.

What This Does

SpatialLM provides a multimodal architecture that bridges unstructured 3‑D geometry and high‑level semantic representation. Key components include:

• Point‑cloud encoding – spatiallm/model/sonata_encoder.py (772 loc, defines offset2bincount, batch2offset, and serialization logic) and spatiallm/pcd/transform.py (29 functions, point‑cloud preprocessing).
• Layout & entity modeling – spatiallm/layout/layout.py defines grid generation and entity extraction; spatiallm/layout/entity.py holds rotate, translate, scale, and normalize_and_discretize.
• Tuning & training pipeline – train.py:29 is the entry point; it calls main which reaches 68 functions, ultimately invoking spatiallm/tuner/trainer.py and spatiallm/tuner/framework/loader.py for data loading and model inference.
• Configuration & hparams – configs/spatiallm_sft.yaml and the hparams modules under spatiallm/tuner/hparams/ (model_args.py, training_args.py, etc.) supply training arguments; spatiallm/tuner/hparams/parser.py parses CLI args via read_args.

The repo ships 46 Python source files, 22 declared dependencies (e.g., transformers>=4.41.2,<=4.46.1, pandas^2.2.3, numpy^1.26), and a pyproject.toml manifest with no lockfile.

How It Is Wired

Execution starts at train.py:29 → main (the sole entry point). From there the call graph fans out:

The most connected hub is spatiallm/tuner/framework/__init__.py (Ca = 14, Ce = 0, instability 0) – 14 modules import from it, making it a blast‑radius point; any change ripples widely.

Circular imports affect spatiallm/tuner/hparams/__init__.py, spatiallm/tuner/data/loader.py, and spatiallm/tuner/hparams/model_args.py (import‑cycle member x8).

Paths that leave the process: main → run_exp → read_args reads sys.argv[1] via Path(...).absolute().read_text; spatiallm/tuner/framework/loader.py calls a model for inference; a few functions read/write files (e.g., logging.py _write_log).

How To Use It

Setup

# Clone the repository
git clone https://github.com/moses-y/SpatialLM
cd SpatialLM

# Create a virtual environment and install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -e .   # installs from pyproject.toml

No lockfile is committed; after install, generate one:

pip freeze > requirements.txt   # or use `pip install --dry-run` then lock

Configuration Edit configs/spatiallm_sft.yaml for run‑time settings (data paths, model name, output directory). Environment variables referenced in spatiallm/tuner/hparams/parser.py (_set_env_vars) control logging verbosity and CUDA devices.

Running it

python train.py --config configs/spatiallm_sft.yaml

The train.py entry point triggers the full training pipeline, including data loading (spatiallm/tuner/data/loader.py), model instantiation (spatiallm/model/spatiallm_llama.py or spatiallm/model/spatiallm_qwen.py), and checkpointing (spatiallm/tuner/framework/loader.py).

Real‑World Use

A typical workflow: a field team captures a point‑cloud scan via a handheld LiDAR. The scan is pre‑processed with spatiallm/pcd/transform.py (e.g., _preprocess_point_cloud, _regularize_point_clouds). The resulting tensor is fed to sonata_encoder.py, which outputs a hierarchical graph. SpatialLM’s LLM head then generates a JSON description containing wall locations, door coordinates, and object categories. This output can be consumed by a building‑information‑modeling (BIM) tool for as‑built documentation, bypassing manual floor‑plan tracing.

Code Health & Issues

Static analysis (Measured Analysis) reports the following findings:

• [HIGH/soundness] Import cycle member x8 – files spatiallm/tuner/hparams/__init__.py, spatiallm/tuner/data/loader.py, spatiallm/tuner/hparams/model_args.py mutually import, risking runtime failures.
• [HIGH] No test suite – 50 source files, zero test files; any change ships without regression signal.
• [HIGH] No lockfile beside pyproject.toml – dependencies are unpinned, making builds non‑reproducible and vulnerable versions possible.
• [HIGH] No CI/CD pipeline – no automated build/test gate; every PR merges without having been exercised.
• [MEDIUM] Broad exception handling x4 – spatiallm/tuner/framework/utils.py, callbacks.py, patcher.py catch Exception indiscriminately, swallowing errors.
• [MEDIUM] Deep nesting x11 – spatiallm/layout/layout.py, spatiallm/tuner/data/converter.py, spatiallm/tuner/data/template.py max indentation depth 6, control flow hard to follow.
• [MEDIUM] Oversized file – spatiallm/model/sonata_encoder.py 772 loc, a single change ripples widely.
• [MEDIUM] Duplicated code blocks – 281 repeated 6‑line blocks across 21 files (collator, converter, formatter, etc.).
• [MEDIUM] Hub module – spatiallm/tuner/framework/__init__.py depends on 14 other modules; churn is high‑blast‑radius.
• [MEDIUM] No Dependabot/Renovate config – vulnerable dependencies may go unpatched.

SDLC observations (beyond the measured block): missing licence file in the root, no Dockerfile, no committed secrets, but licence text is present (LICENSE.txt).

The Bottom Line

SpatialLM delivers a functional pipeline for turning raw point clouds into structured 3‑D scene descriptions, which is valuable for indoor‑modeling use‑cases. However, the codebase suffers from import cycles, missing reproducibility tooling (lockfile, CI), and structural debt (oversized files, duplicated logic, deep nesting) that increase the risk of regressions and make future changes costly. Teams that can tolerate early‑stage software quality and have capacity to address the highlighted health items will find the model useful; others should treat it as prototype material until the SDLC gaps are closed.