rf detr

Abstract

RF-DETR is a real-time object detection and segmentation model architecture developed by Roboflow, SOTA on COCO, designed for fine-tuning. [ICLR 2026]

The Problem

RF-DETR addresses the need for a real-time transformer architecture that unifies object detection, instance segmentation, and keypoint detection with state-of-the-art COCO performance. The repository provides fine-tunable models built on DINOv2 vision transformer backbones, targeting researchers and engineers who need consistent APIs across detection, segmentation, and keypoint tasks without maintaining separate codebases.

What This Does

The core model architecture lives in src/rfdetr/models/, structured around a DINOv2 vision transformer backbone (src/rfdetr/models/backbone/dinov2.py) with task-specific heads for detection (src/rfdetr/models/heads/segmentation.py, src/rfdetr/models/heads/keypoints.py), and a bipartite matching criterion (src/rfdetr/models/criterion.py). Configuration is handled through YAML files in configs/ (e.g., configs/rfdetr_base.yaml, configs/rfdetr_seg_large.yaml), while dataset pipelines operate from src/rfdetr/datasets/ including transforms (src/rfdetr/datasets/transforms.py), COCO integration (src/rfdetr/datasets/coco.py), and Yolo format support (src/rfdetr/datasets/yolo.py). Export functionality is centralized in src/rfdetr/export/main.py with backends for ONNX (src/rfdetr/export/_onnx/exporter.py), CoreML (src/rfdetr/export/_coreml/converter.py), TensorRT (src/rfdetr/export/_tensorrt.py), and Executorch (src/rfdetr/export/_executorch/converter.py). Training entry points are available via src/rfdetr/training/cli.py and inference through src/rfdetr/inference.py.

How It Is Wired

Execution originates at the CLI entry points: src/rfdetr/__main__.py for package-level invocation, src/rfdetr/training/cli.py for training workflows, and src/rfdetr/export/main.py for model export. The training CLI loads a config YAML, instantiates the model via src/rfdetr/models/__init__.py, constructs datasets through src/rfdetr/datasets/_develop.py, and runs the training loop with evaluation hooks in src/rfdetr/evaluation/coco_eval.py. The export pipeline routes through src/rfdetr/export/_backend.py to format-specific converters, each implementing convert() methods that produce framework-specific artifacts. A notable coupling exists between src/rfdetr/models/backbone/dinov2.py and its JSON config files in src/rfdetr/models/backbone/dinov2_configs/ — any backbone dimension changes require corresponding config updates, and the windowed attention variant (src/rfdetr/models/backbone/dinov2_with_windowed_attn.py) introduces an additional code path that diverges from the standard DINOv2 flow. The evaluation module (src/rfdetr/evaluation/) is relatively decoupled, accepting prediction/ground-truth pairs without influencing model architecture, which limits its blast radius during architecture changes.

How To Use It

Setup: Install the published package with pip install rfdetr for Python >= 3.10, or install from source via pip install https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip. No lockfile is present in the repository — dependencies are declared in pyproject.toml only, which may lead to non-reproducible builds across environments.

Configuration: Model configuration is specified through YAML files in configs/. The configs/rfdetr_base.yaml file defines the default architecture, while segmentation variants use configs/rfdetr_seg_*.yaml. Environment variables or keys are not required for basic operation; configuration is file-driven.

Running it: To fine-tune on a detection dataset, the notebook docs/cookbooks/fine-tune_detection.ipynb demonstrates the workflow. For inference, src/rfdetr/inference.py exports a predict() function acceptimg an image path and returning detection results. Export to ONNX is triggered from the command line: python -m rfdetr.export main --weights PATH --output ONNX --format onnx, and similar flags exist for TensorRT, CoreML, and Executorch formats.

Real-World Use

A computer vision team needing to deploy a single model across camera-based detection, warehouse inventory segmentation, and human pose estimation would use RF-DETR's unified API. A typical workflow loads a checkpoint, runs rfdetr.inference.predict() on a batch of images, and exports the same weights to ONNX for edge deployment. The DINOv2 backbone ensures transfer learning from large-scale pre-training, while the NAS-informed model sizing (nano through xlarge) allows matching compute budgets to accuracy requirements. The export pipeline produces format-conversion-ready weights without retraining, useful for CI-integrated deployment pipelines.

Code Health & Issues

Low/Risk - Dependencies declared without a lockfile: pyproject.toml lists runtime dependencies without pinned versions or a requirements.txt lockfile, creating non-reproducible build risk. Verified at the root pyproject.toml.

Beyond this measured finding, the repository structure shows comprehensive test coverage (160 test files in tests/), extensive documentation (80 doc files), and CI workflows across GitHub Actions for tests, docs, and releases. No license or secrets issues are evident from the file graph.

The Bottom Line

RF-DETR delivers a well-structured, single-API architecture for detection, segmentation, and keypoint tasks with solid DINOv2-backed baselines and export flexibility across major deployment formats. The absence of a lockfile is the primary reproducibility concern, and the DINOv2 backbone configs introduce a modest coupling point for architecture modifications. Teams seeking a real-time, fine-tunable multi-task model will find it practical; those requiring strict dependency pinning should add a requirements.txt lockfile to their workflow.