Marigold V2
Relative depth from one photo by a very large image-generation model run for a single step: accurate but slow and memory-hungry (about 2 GB of GPU memory, 9 GB of host RAM).
experimental Fork of huawei-bayerlab/marigold-v2@main · backends: portable, kernels
- Paper Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation ACM TOG (SIGGRAPH Asia 2026) · first released 9 Sept 2026
Checkpoint marigold-v2-depth
Paper 2026 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install fastforks-core
from fastforks import depth
model = depth.load("marigold-v2-depth") # kernels on a supported GPU, else portable PyTorch
result = model(image)
result.depth
result.backend # what ran: mode, device, dtype, kernels fastforks bench marigold-v2-depth Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs bf16. Portable runs bf16 on CUDA.
| Backend | dtype | Time per call (ms) | vs original | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | not measured | – | – | – | – |
| Original fp16 | fp16 | out of memory | – | – | – | – |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | bf16 | 5093 | – | 1,408 MiB/1,698 MiB | 7.8 | 5.09 |
| Kernels | bf16 | 3886 | – | 1,498 MiB/1,780 MiB | 6.2 | 3.89 |
| Portable + torch.compiletimeout after 900 s (audit 2026-11-16, RTX 5060 Ti: compile | fails | – | – | – | – | |
| TensorRTplanned for 0.2.1 | fails | – | – | – | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-15. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-17 | Portable | fp32 | cannot run here | – | 994x1482 | |
| Kernels | bf16 | not implemented | – | 994x1482 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- Apache-2.0 LICENSENOTICE
- Weights
- Apache-2.0 source
- Training data
- Qwen-Image-Edit-2509 (Alibaba, Apache-2.0) fine-tuned (LoRA + VAE decoder) on synthetic depth data per the Marigold V2 paper (arXiv 2609.08084); the training data terms are not checked
- The converted weight file (NF4 transformer, LoRA, folded constants, 2-D VAE) is hosted by fastforks under the originals' Apache-2.0 licence; NOTICE keeps both attributions and lists the changes.
Our reading of the licences, not legal advice.
Weights
marigold-v2-depth-log-stage2-nf4.safetensors 9.0 GB· hosted on Hugging Facefastforks downloads and checks these files on first use (fastforks download marigold-v2-depth).
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
NYUv2 test
AbsRel / delta1
| Run | dtype | AbsRel ↓ | δ1 ↑ | images | ms / image |
|---|---|---|---|---|---|
| Original fp32 | not measured | ||||
| Original fp16 | bf16 | out of memory | |||
| Portable fp32 | fp32 | 0.0386 | 0.9840 | 654 | 1328 |
| Portable | bf16 | 0.0386 | 0.9840 | 654 | 1310 |
| Kernels | bf16 | 0.0386 | 0.9840 | 654 | 777 |
The original code was not run on this dataset, so the table cannot show the distance to it.
AbsRel / delta1 after a least-squares scale + shift fit (in disparity space for inverse-depth models), mean over images. Original = the authors’ code, same checkpoint and protocol. Measured on original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-16; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 6d3bab64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 37687519, 2026-11-14.
iBims-1
AbsRel / delta1
| Run | dtype | AbsRel ↓ | δ1 ↑ | images | ms / image |
|---|---|---|---|---|---|
| Original fp32 | not measured | ||||
| Original fp16 | not measured | ||||
| Portable fp32 | fp32 | 0.0291 | 0.9877 | 100 | 1384 |
| Portable | bf16 | 0.0291 | 0.9877 | 100 | 1362 |
| Kernels | bf16 | 0.0291 | 0.9877 | 100 | 820 |
The original code was not run on this dataset, so the table cannot show the distance to it.
AbsRel / delta1 after a least-squares scale + shift fit (in disparity space for inverse-depth models), mean over images. Measured on portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 6d3bab64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 37687519, 2026-11-14.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.02 vs the original. GPU, bfloat16 (the default), portable and kernels: mean |diff| <= 1 % and p99 <= 5 % of the reference's median |value| (absolute error of the output depth).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.02 (relative) | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.01 (relative) | 0 (relative) |
| GPU default dtypes, p99 |diff| | ≤ 0.05 (relative) | 0 (relative) |
Recorded for marigold-v2-depth on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: nf4
Details
Marigold V2 (Pavlovic, Wandel, Obukhov et al. 2026): affine-invariant log depth by one rectified-flow step of Qwen-Image-Edit-2509's 20 B transformer (NF4 4-bit, as the authors run it) with a rank-128 depth LoRA and a fine-tuned Qwen VAE decoder (depth/Log-stage2).
- Input / output
- Input: one RGB image. Output: a depth or inverse-depth map.
What the numbers mean, and which models need the camera focal length
kind(a card field,result.kind) says what the numbers mean:metric= metres;scale_invariant,affine_invariant,affine_invariant_log(log depth) andaffine_invariant_inverseare relative (unknown scale or shift, not metres). Metric models that use the focal length takemodel(image, intrinsics=K)(K[3, 3], pixels of the input):metric3d-vit-largeandda3metric-largeneed it for correct metres (da3metric-large: metres = output x focal / 300; withoutintrinsics=a 300 px focal length is assumed, wrong for a real camera).unidepth-v2-vitl14,unik3d-vitl,depth-proand MoGe estimate the camera themselves (givenintrinsics=replacefx);zoedepth-nkand the relative models (da3mono-largetoo) take none. Normals-only models (kindnone) returnresult.normals[B, H, W, 3](unit vectors, camera frame: x right, y down, z forward). - Input size
- any H x W >= 1, resized or padded inside the model; tested up to 1988x2880 (12.8 GiB allocated)
- Options
seed=2026· changes the outputlong_side=0>= 0· changes the outputexact_mask=false· changes the outputstream=true· changes the output
- Outputs
depth- CUDA kernels
nf4
Limitations
- 20 B transformer in 4 bit: 8.4 GB of weights; by default (`stream=True`, CUDA) it runs in ~1-4 GB of device memory from 8.4 GB of page-locked host RAM (load fails with an error when less is free); `stream=False` keeps it resident (about 9-11 GB of device memory, 16 GB cards: up to ~1024 px).
- Diffusion model: the VAE posterior sample comes from a CPU generator seeded with `seed` and differs from the original code's (unseeded, device) sample.
- The enlargement to a multiple of 16 is bicubic (the original: OpenCV Lanczos); inputs with sides that are multiples of 16 are not resized.
- The output is log depth up to scale and shift (`kind = "affine_invariant_log"`): align in log space, not with the other families' depth outputs.
- A traced or exported module reuses the noise of its traced shape; CPU inference is possible but slow (bf16 GEMMs of a 20 B model).
Full reference, generated from the repository
Relative depth from one photo by a very large image-generation model run for a single step: accurate but slow and memory-hungry (about 2 GB of GPU memory, 9 GB of host RAM).
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
marigold-v2-depth | synthetic (994x1482) | original portable kernels portable + torch.compile TensorRT fp16 | OOM 5093 3886 fails: timeout after 900 s (audit 2026-11-16, RTX 5060 Ti: compile planned for 0.2.1 | – 1408 / 1698 1498 / 1780 – – |
Weight streaming
Warm inference time and peak reserved GPU memory for the same input with both settings. Streamed weights use pinned host memory.
| Model | Mode | Input | GPU | PyTorch | Resident (stream=False) | Streamed (stream=True) |
|---|---|---|---|---|---|---|
| marigold-v2-depth | kernels | 518x770 | RTX 5060 Ti | 2.14.1+cu130 | 1,071 ms / 9,482 MiB | 1,087 ms / 1,238 MiB |
| marigold-v2-depth | kernels | 994x1482 | RTX 5060 Ti | 2.14.1+cu130 | 3,868 ms / 10,548 MiB | 3,887 ms / 2,106 MiB |
| marigold-v2-depth | portable | 518x770 | RTX 5060 Ti | 2.14.1+cu130 | 1,666 ms / 9,582 MiB | 1,683 ms / 1,190 MiB |
| marigold-v2-depth | portable | 994x1482 | RTX 5060 Ti | 2.14.1+cu130 | 5,071 ms / 10,250 MiB | 5,092 ms / 2,106 MiB |
Details
- Original code: https://github.com/huawei-bayerlab/marigold-v2 at
main - Code licence: Apache-2.0 (permissive); files:
LICENSE,NOTICE - Weights licence: Apache-2.0
- Licence evidence: https://huggingface.co/huawei-bayerlab/marigold-v2/blob/main/README.md (license: apache-2.0; base model Qwen/Qwen-Image-Edit-2509)
- Training data: Qwen-Image-Edit-2509 (Alibaba, Apache-2.0) fine-tuned (LoRA + VAE decoder) on synthetic depth data per the Marigold V2 paper (arXiv 2609.08084); the training data terms are not checked
- Licence note: The converted weight file (NF4 transformer, LoRA, folded constants, 2-D VAE) is hosted by fastforks under the originals' Apache-2.0 licence; NOTICE keeps both attributions and lists the changes.
- Options:
seed= 2026,long_side= 0 (>= 0),exact_mask= False,stream= True - Kernel libraries:
nf4 - Variants:
marigold-v2-depth - Not in this release (planned for 0.2.1):
marigold-v2-depthONNX / TensorRT export: the NF4 transformer expands its weights per call; no ONNX path yet - Outputs: depth; kind:
affine_invariant_log - Input size: any H x W >= 1, resized or padded inside the model
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
marigold-v2-depth1988x2880 (12.8 GiB peak allocated, 13.5 reserved) marigold-v2-depthload / first call: portable 7.8 s / 5.094 s, kernels 6.2 s / 3.887 s- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
AbsRel / delta1 after a least-squares scale + shift fit (in disparity space for inverse-depth models), mean over images. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original half | portable | portable fp32 | kernels | note |
|---|---|---|---|---|---|---|
marigold-v2-depth | NYUv2 test, AbsRel / delta1 | out of memory | 0.0386 / 0.9840 | 0.0386 / 0.9840 | 0.0386 / 0.9840 | |
marigold-v2-depth | iBims-1, AbsRel / delta1 | 0.0291 / 0.9877 | 0.0291 / 0.9877 | 0.0291 / 0.9877 |
Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 37687519, 2026-11-14; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 6d3bab64, 2026-11-17; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 37687519, 2026-11-14.
Parity
CPU, float32: max |diff| <= 0.02 vs the original. GPU, bfloat16 (the default), portable and kernels: mean |diff| <= 1 % and p99 <= 5 % of the reference's median |value| (absolute error of the output depth).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
marigold-v2-depth | 0.0 / 0.0 | RTX 5060 Ti, torch 2.14.1+cu130 | nf4@e236d80ba78b:sm_120 |
Limitations
- 20 B transformer in 4 bit: 8.4 GB of weights; by default (
stream=True, CUDA) it runs in ~1-4 GB of device memory from 8.4 GB of page-locked host RAM (load fails with an error when less is free);stream=Falsekeeps it resident (about 9-11 GB of device memory, 16 GB cards: up to ~1024 px). - Diffusion model: the VAE posterior sample comes from a CPU generator seeded with
seedand differs from the original code's (unseeded, device) sample. - The enlargement to a multiple of 16 is bicubic (the original: OpenCV Lanczos); inputs with sides that are multiples of 16 are not resized.
- The output is log depth up to scale and shift (
kind = "affine_invariant_log"): align in log space, not with the other families' depth outputs. - A traced or exported module reuses the noise of its traced shape; CPU inference is possible but slow (bf16 GEMMs of a 20 B model).
Example output
No example rendered for this checkpoint yet.