DepthFM
DepthFM (CompVis, LMU Munich): relative depth from one image by flow matching on a Stable Diffusion UNet, in two steps. MIT code; OpenRAIL weights (commercially usable with use restrictions).
experimental Fork of CompVis/depth-fm@da632c8 · backends: portable, kernels
- Paper DepthFM: Fast Monocular Depth Estimation with Flow Matching AAAI 2025 · first released 20 Mar 2024
Checkpoint depthfm-v1
Paper 2024 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install fastforks-core
from fastforks import depth
model = depth.load("depthfm-v1") # kernels on a supported GPU, else portable PyTorch
result = model(image)
result.depth
result.backend # what ran: mode, device, dtype, kernels fastforks bench depthfm-v1 Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.
Input: synthetic, 994x1482 (the task’s standard size)
| Backend | dtype | Time per call (ms) | vs original | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | not measured | – | – | – | – |
| Original fp16 | fp16 | out of memory | – | – | – | – |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp16 | 5662 | – | 13.3 GiB/14.8 GiB | 3.6 | 6.00 |
| Kernels | fp16 | 3570 | – | 9,237 MiB/9,314 MiB | 4.6 | 3.73 |
| Portable + torch.compile | 3947 | – | 9,240 MiB/10.1 GiB | 102 (build) | – | |
| TensorRTONNX export: conversion failed | fails | – | – | – | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-15 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-17 | Portable | fp16 | out of memory | – | 994x1482 | |
| Portable | fp32 | out of memory | – | 994x1482 | ||
| Kernels | fp16 | not implemented | – | 994x1482 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- MIT LICENSELICENSE-MODELLICENSE-VAE
- Weights
- LicenseRef-CreativeML-OpenRAIL-PlusPlus-M AND LicenseRef-CreativeML-OpenRAIL-M source
- Training data
- Stable Diffusion 2.1 (LAION-5B subsets) fine-tuned on the synthetic Hypersim and Virtual KITTI 2 datasets per the DepthFM paper (arXiv 2403.13788)
- The UNet is fine-tuned from Stable Diffusion 2.1 (OpenRAIL++-M) and runs with the Stable Diffusion 1.5 VAE (OpenRAIL-M): commercial use is allowed but with use restrictions that must be passed on; the training data terms are not checked.
Our reading of the licences, not legal advice.
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
iBims-1
AbsRel / delta1
| Run | dtype | AbsRel ↓ | δ1 ↑ | Δ AbsRel vs fp32 original | images | ms / image |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 0.0652 | 0.9540 | reference | 100 | 2414 |
| Original fp16 | fp16 | 0.0652 | 0.9540 | ±0.0000 | 100 | 1418 |
| Portable fp32 | fp32 | 0.0650 | 0.9543 | +0.0002 | 100 | 2045 |
| Portable | fp16 | 0.0650 | 0.9543 | +0.0002 | 100 | 911 |
| Kernels | fp16 | 0.0650 | 0.9543 | +0.0002 | 100 | 530 |
AbsRel / delta1 after a least-squares scale + shift fit (in disparity space for inverse-depth models), mean over images. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-16; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 6d3bab64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 68b2f345, 2026-11-16.
NYUv2 test (first 100 images)
AbsRel / delta1
| Run | dtype | AbsRel ↓ | δ1 ↑ | Δ AbsRel vs fp32 original | images | ms / image |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 0.0735 | 0.9489 | reference | 100 | 2066 |
| Original fp16 | fp16 | 0.0731 | 0.9492 | +0.0004 | 100 | 1228 |
| Portable fp32 | not measured | |||||
| Portable | fp16 | 0.0732 | 0.9493 | +0.0003 | 100 | 789 |
| Kernels | fp16 | 0.0732 | 0.9493 | +0.0003 | 100 | 456 |
AbsRel / delta1 after a least-squares scale + shift fit (in disparity space for inverse-depth models), mean over images. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-16; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 68b2f345, 2026-11-16.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 1 % and p99 <= 3 % of the reference's median |value| (absolute error of the output depth).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.0001 (relative) | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.01 (relative) | 0.01169 (relative) |
| GPU default dtypes, p99 |diff| | ≤ 0.03 (relative) | 0.02968 (relative) |
Recorded for depthfm-v1 on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: cudnn_hacc
Details
DepthFM (Gui et al. 2024): affine-invariant monocular depth by flow matching from the image latent to the depth latent; a Stable Diffusion 2.1 UNet fine-tuned to the flow, a few Euler steps, ensemble of noise samples (v1).
- Input / output
- Input: one RGB image. Output: a depth or inverse-depth map.
What the numbers mean, and which models need the camera focal length
kind(a card field,result.kind) says what the numbers mean:metric= metres;scale_invariant,affine_invariant,affine_invariant_log(log depth) andaffine_invariant_inverseare relative (unknown scale or shift, not metres). Metric models that use the focal length takemodel(image, intrinsics=K)(K[3, 3], pixels of the input):metric3d-vit-largeandda3metric-largeneed it for correct metres (da3metric-large: metres = output x focal / 300; withoutintrinsics=a 300 px focal length is assumed, wrong for a real camera).unidepth-v2-vitl14,unik3d-vitl,depth-proand MoGe estimate the camera themselves (givenintrinsics=replacefx);zoedepth-nkand the relative models (da3mono-largetoo) take none. Normals-only models (kindnone) returnresult.normals[B, H, W, 3](unit vectors, camera frame: x right, y down, z forward). - Input size
- any H x W >= 1, resized or padded inside the model; tested up to 1024x1024 (10,078 MiB allocated)
- Options
steps=21 to 1000· changes the outputensemble=4>= 1· changes the outputlong_side=0>= 0· changes the outputseed=2024· changes the output
- Outputs
depth- CUDA kernels
cudnn_haccconvolutions with fp16 accumulation (cuDNN)
- Apple (MPS)
- fp32, fp16
Limitations
- Flow model on a noised latent: the seeded noise comes from a CPU generator and differs from the original code's device generator for the same seed.
- A traced or exported module reuses the noise of its traced shape.
- The network input sides are rounded to multiples of 64, so the aspect ratio changes slightly.
Full reference, generated from the repository
DepthFM (CompVis, LMU Munich): relative depth from one image by flow matching on a Stable Diffusion UNet, in two steps. MIT code; OpenRAIL weights (commercially usable with use restrictions).
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
depthfm-v1 | synthetic (994x1482) | original portable kernels portable + torch.compile TensorRT fp16 | OOM 5662 3570 3947 fails: ONNX export: conversion failed | – 13651 / 15150 9237 / 9314 9240 / 10292 – |
Details
- Original code: https://github.com/CompVis/depth-fm at
da632c8 - Code licence: MIT (permissive); files:
LICENSE,LICENSE-MODEL,LICENSE-VAE - Weights licence: LicenseRef-CreativeML-OpenRAIL-PlusPlus-M AND LicenseRef-CreativeML-OpenRAIL-M
- Licence evidence: https://github.com/CompVis/depth-fm/blob/da632c8118e41cc64ca191a13314e2db1a7722d9/README.md (fine-tuned from Stable Diffusion v2-1; code MIT); SD2 licence text from https://huggingface.co/sd2-community/stable-diffusion-2/blob/2511124fabf8bf30c0ca9dccd9729e7f0f2fa669/LICENSE-MODEL, SD1.5 VAE licence https://github.com/CompVis/stable-diffusion/blob/21f890f9da3cfbeaba8e2ac3c425ee9e998d5229/LICENSE
- Training data: Stable Diffusion 2.1 (LAION-5B subsets) fine-tuned on the synthetic Hypersim and Virtual KITTI 2 datasets per the DepthFM paper (arXiv 2403.13788)
- Licence note: The UNet is fine-tuned from Stable Diffusion 2.1 (OpenRAIL++-M) and runs with the Stable Diffusion 1.5 VAE (OpenRAIL-M): commercial use is allowed but with use restrictions that must be passed on; the training data terms are not checked.
- Options:
steps= 2 (1 to 1000; alsonum_steps),ensemble= 4 (>= 1; alsoensemble_size),long_side= 0 (>= 0; alsoprocessing_res),seed= 2024 - Kernel libraries:
cudnn_hacc - Variants:
depthfm-v1 - Outputs: depth; kind:
affine_invariant - Input size: any H x W >= 1, resized or padded inside the model
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
depthfm-v11024x1024 (9.8 GiB peak allocated, 15.0 reserved) depthfm-v1994x1482, first call: portable + torch.compile 102 s (cold Inductor cache).depthfm-v1load / first call: portable 3.58 s / 6.004 s, kernels 4.59 s / 3.734 s- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
AbsRel / delta1 after a least-squares scale + shift fit (in disparity space for inverse-depth models), mean over images. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | portable fp32 | kernels | note |
|---|---|---|---|---|---|---|---|
depthfm-v1 | iBims-1, AbsRel / delta1 | 0.0652 / 0.9540 | 0.0652 / 0.9540 | 0.0650 / 0.9543 | 0.0650 / 0.9543 | 0.0650 / 0.9543 | |
depthfm-v1 | NYUv2 test (first 100 images), AbsRel / delta1 | 0.0735 / 0.9489 | 0.0731 / 0.9492 | 0.0732 / 0.9493 | 0.0732 / 0.9493 |
Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 68b2f345, 2026-11-16; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 6d3bab64, 2026-11-17; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 68b2f345, 2026-11-16.
Parity
CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 1 % and p99 <= 3 % of the reference's median |value| (absolute error of the output depth).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
depthfm-v1 | 0.01169 / 0.02968 | RTX 5060 Ti, torch 2.14.1+cu130 | cudnn_hacc@cudnn92400 |
Limitations
- Flow model on a noised latent: the seeded noise comes from a CPU generator and differs from the original code's device generator for the same seed.
- A traced or exported module reuses the noise of its traced shape.
- The network input sides are rounded to multiples of 64, so the aspect ratio changes slightly.
Example output
No example rendered for this checkpoint yet.