The Problem
Organizations that need detailed, semantic 3‑D scene understanding (e.g., indoor mapping, robotics navigation, AR content creation) must currently stitch together separate point‑cloud processing pipelines, custom geometry parsers, and language models. The lack of a single model that consumes raw point clouds and directly emits structured scene descriptions forces engineers to maintain fragile glue code and duplicate data‑preparation logic.
What This Does
SpatialLM bundles a large language model with a point‑cloud encoder (the Sonata encoder in spatiallm/model/sonataencoder.py) and a series of serialization utilities (spatiallm/model/serialization/*.py). The core model classes—spatiallm/model/spatiallmllama.py and spatiallm/model/spatiallmqwen.py—extend LLaMA/Qwen back‑ends to accept a token stream that interleaves geometry tokens (produced by spatiallm/pcd/pcdloader.py) with textual prompts.
The repository also supplies a full finetuning stack under spatiallm/tuner/. Data handling lives in spatiallm/tuner/data/ (loader, collator, formatter) and the training loop is in spatiallm/tuner/trainer.py. Example scripts (train.py, eval.py, inference.py) demonstrate end‑to‑end training, evaluation, and inference on the released SpatialLM‑Dataset.
How To Use It
Setup
Install the package and its dependencies from pyproject.toml pip install . Or, for an editable dev environment pip install -e .
The pyproject.toml lists torch, transformers, datasets, and open3d among required libraries.
Configuration
Finetuning expects a YAML config; a template is provided at configs/spatiallmsft.yaml. Populate the fields (model checkpoint path, dataset name, training hyper‑parameters) before launching training.
Running
Training: python train.py --config configs/spatiallmsft.yaml Evaluation: python eval.py --config configs/spatiallmsft.yaml Inference (single point‑cloud file): python inference.py \ --modelpath path/to/checkpoint \ --pcd path/to/pointcloud.pcd \ --output out.json
The scripts read the configuration file directly; no additional environment variables are required.
Real‑World Use
A robotics team can integrate SpatialLM into a perception node:
from spatiallm.inference import runinference scenedesc = runinference( modelpath="models/spatiallmllama1b.pt", pcdpath="/tmp/currentscan.pcd" ) scenedesc now contains walls, doors, and object boxes ready for downstream planning.
The JSON output can be fed to a navigation stack that reasons about traversable space and object interactions without writing custom geometry parsers.
Code Health & Issues
Medium – No test suite – No tests/ directory or pytest files; untested code paths increase regression risk. Medium – No CI/CD – Repository lacks .github/workflows/ or other pipeline definitions; build and quality gates are manual. Low – Missing lockfile – Dependencies are declared only in pyproject.toml; reproducible builds rely on external PyPI snapshots. Low – Sparse documentation – README covers high‑level usage; detailed API docs for modules such as spatiallm/layout/ are absent. Low – License present but minimal – LICENSE.txt is included, but no explicit citation guidance for model weights. Low – Potential runtime errors – Point‑cloud loading (spatiallm/pcd/pcdloader.py) assumes Open3D availability but does not validate file existence before processing.
Overall, the code follows a clear package layout and uses standard PyTorch/Transformers patterns, but the lack of automated testing and CI means stability depends on manual verification.
The Bottom Line
SpatialLM delivers a ready‑to‑use 3‑D LLM that bridges raw point clouds and structured scene descriptions, with a complete training pipeline. It is suitable for teams that can allocate resources for manual testing and integration, especially when working with indoor mapping or robotics workloads. Smaller or production‑critical deployments should first add a test suite and CI to mitigate the current quality gaps.