Models · Stereo matching

FoundationStereo

NVIDIA's zero-shot stereo foundation model. Non-commercial weights.

stable Fork of NVlabs/FoundationStereo@6e88068 · backends: portable, kernels

Checkpoint

Paper 2025 Added in 0.1.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 1513 ms 994x1482, fp16
vs the faster original 2.8× original fp16: 4237 ms
Peak memory 2,818 MiB original: 8,286 MiB
Accuracy vs original fp32 ±0.00 kernels bad px %, ETH3D two-view train (0.65 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install "fastforks-core[nc]"

from fastforks import stereo

model = stereo.load("foundationstereo-vit-l")  # kernels on a supported GPU, else portable PyTorch
result = model(left, right)
result.disparity
result.backend  # what ran: mode, device, dtype, kernels

The weights are non-commercial: fastforks downloads them only with FASTFORKS_NONCOMMERCIAL=1 set.

fastforks bench foundationstereo-vit-l

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32fp32 92280.46×11.1 GiB/11.4 GiB––
Original fp16fp16 4237–8,286 MiB/8,750 MiB––
Portable fp32fp32not measured ––––
Portablefp16 29901.4×3,141 MiB/3,436 MiB2.02.99
Kernelsfp16 15132.8×2,818 MiB/3,300 MiB1.21.52
Portable + torch.compile 25011.7×3,453 MiB/3,954 MiB226 (build)–
TensorRT fp16fp16 15132.8×3,432 MiBenginenot comparable611 (build)–

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-01 to 2026-11-03. Median of 5 after 3 warm-up, uint8 CUDA input (synthetic for bench_inputs sizes), card defaults; gc before each load; peak = max over the first call and the timed calls. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-13Originalfp32out of memory–994x1482
Portablefp1625647 13.4 GiB processmatches994x1482
Portablefp32out of memory–994x1482
Kernelsfp1624480 13.0 GiB processmatches994x1482
NVIDIA L4 CUDA, torch 2.14.1+cu130, 2026-11-07Portablefp163172 3,114 MiB/3,416 MiB–994x1482
Kernelsfp162035 2,791 MiB/3,040 MiB–994x1482
NVIDIA L40S CUDA, torch 2.14.1+cu130, 2026-11-07Portablefp16968 3,114 MiB/3,416 MiB–994x1482
Kernelsfp16605 2,791 MiB/3,300 MiB–994x1482
NVIDIA A10 CUDA, torch 2.14.1+cu130, 2026-11-07Portablefp162456 3,114 MiB/3,416 MiB–994x1482
Kernelsfp161674 2,788 MiB/3,040 MiB–994x1482
NVIDIA A100-SXM4-40GB CUDA, torch 2.14.1+cu130, 2026-11-07Portablefp16959 2,874 MiB/3,156 MiB–994x1482
Kernelsfp16680 2,911 MiB/3,300 MiB–994x1482
NVIDIA RTX PRO 6000 CUDA, torch 2.14.1+cu130, 2026-11-08Portablefp16496 3,138 MiB/3,436 MiB–994x1482
Kernelsfp16337 2,815 MiB/3,180 MiB–994x1482

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Non-commercial only

Code
LicenseRef-NVIDIA-Source-Code-License-NC AND Apache-2.0 LICENSE
Weights
LicenseRef-NVIDIA-Source-Code-License-NC source
Training data
Foundation Stereo Dataset (FSD, synthetic, NVIDIA) plus SceneFlow, Falling Things, Sintel, CREStereo, InStereo2K and others; no per-dataset terms checked
  • code: LicenseRef-NVIDIA-Source-Code-License-NC AND Apache-2.0 (noncommercial)
  • weights: LicenseRef-NVIDIA-Source-Code-License-NC
  • Code and weights are under the NVIDIA Source Code License-NC (non-commercial research use only). Our kernels for this family count as derived work and inherit the NC limit; the family ships in fastforks-nc.

Our reading of the licences, not legal advice.

Weights

23-51-11/model_best_bp2.pth 3.3 GB· hosted on Google Drive

fastforks downloads and checks these files on first use (fastforks download foundationstereo-vit-l). Non-commercial weights need FASTFORKS_NONCOMMERCIAL=1.

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

ETH3D two-view train

bad > 1 px % / EPEtraining data not stated: possibly seen

Rundtypebad px % ↓EPE ↓Δ bad px % vs fp32 originalpairsms / pair
Original fp32fp320.650.168 reference 27 2171
Original fp16fp160.650.168 ±0.00 27 1029
Portable fp32fp320.650.168 ±0.00 27 1808
Portablefp160.650.168 ±0.00 27 720
Kernelsfp160.650.168 ±0.00 27 376

Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-05; portable fp32, portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 468fba10, 2026-11-05.

Middlebury 2014 trainingH

bad > 2 px % / EPEtraining data not stated: possibly seen

Rundtypebad px % ↓EPE ↓Δ bad px % vs fp32 originalpairsms / pair
Original fp32fp322.480.562 reference 15 7928
Original fp16fp162.480.561 ±0.00 15 3630
Portable fp32fp322.490.562 −0.01 15 6683
Portablefp162.470.561 +0.01 15 2564
Kernelsfp162.480.561 ±0.00 15 1235

Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-05; portable fp32, portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 468fba10, 2026-11-05.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.02 px and p99 <= 0.25 px.

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.002 pxnot recorded
GPU default dtypes, mean |diff|≤ 0.02 px0.00035 px
GPU default dtypes, p99 |diff|≤ 0.25 px0.00121 px

Recorded for foundationstereo-vit-l on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-04, with these kernels: cudnn_haccfastbbfsfs_cutlass

Details

FoundationStereo (Wen et al. 2025): zero-shot stereo foundation model with Depth Anything ViT + CNN features, a 3-D regularised cost volume with a disparity transformer and selective ConvGRU updates, 32 iterations.

Input / output
Input: a rectified left / right image pair. Output: a disparity map (pixels, left view).
Input size
any H x W >= 32, padded to a multiple of 32; tested up to 1988x2880 (7,298 MiB allocated)
Options
  • iters=32>= 1· changes the output
Outputs
disparity
CUDA kernels
  • fsFoundationStereo: fused epilogues, geometry lookups, GRU gates
  • fs_cutlassFoundationStereo: CUTLASS convolutions with fused epilogues
  • cudnn_haccconvolutions with fp16 accumulation (cuDNN)
  • fastbbfast backbone kernels: flash attention, window attention, instance norm and rotary embeddings for ViT and similar backbones
Apple (MPS)
stereo_metalMetal: stereo kernels. fp32, fp16

Limitations

  • ONNX export needs torch >= 2.14 (older exporters fail on this network).
Full reference, generated from the repository

NVIDIA's zero-shot stereo foundation model. Non-commercial weights.

Commercial use: no (code: LicenseRef-NVIDIA-Source-Code-License-NC AND Apache-2.0 (noncommercial); weights: LicenseRef-NVIDIA-Source-Code-License-NC).

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
foundationstereo-vit-lMiddlebury 2014 half (994x1482)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
9228
4237
2990
1513
2501
1513
11323 / 11632
8286 / 8750
3141 / 3436
2818 / 3300
3453 / 3954
4253 / 4322
foundationstereo-vit-sMiddlebury 2014 half (994x1482)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
7129
3609
2514
1120
2044
1196
9991 / 15144
6987 / 7280
2493 / 2606
2169 / 2470
2822 / 3646
2794 / 2872
Details
  • Original code: https://github.com/NVlabs/FoundationStereo at 6e8806816b53
  • Code licence: LicenseRef-NVIDIA-Source-Code-License-NC AND Apache-2.0 (noncommercial); files: LICENSE
  • Weights licence: LicenseRef-NVIDIA-Source-Code-License-NC
  • Licence evidence: https://github.com/NVlabs/FoundationStereo/blob/6e8806816b533e4d13ddbb95ffa907b797060a62/readme.md#L158
  • Training data: Foundation Stereo Dataset (FSD, synthetic, NVIDIA) plus SceneFlow, Falling Things, Sintel, CREStereo, InStereo2K and others; no per-dataset terms checked
  • Licence note: Code and weights are under the NVIDIA Source Code License-NC (non-commercial research use only). Our kernels for this family count as derived work and inherit the NC limit; the family ships in fastforks-nc.
  • Options: iters = 32 (>= 1)
  • Kernel libraries: fs, fs_cutlass, cudnn_hacc, fastbb
  • Variants: foundationstereo-vit-l, foundationstereo-vit-s
  • Outputs: disparity
  • Input size: any H x W >= 32, padded to a multiple of 32
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): foundationstereo-vit-l 1988x2880 (7.1 GiB peak allocated, 11.4 reserved); foundationstereo-vit-s 1988x2880 (6.5 GiB peak allocated, 10.7 reserved)
  • foundationstereo-vit-l 994x1482, first call: portable + torch.compile 226 s (cold Inductor cache), TensorRT fp16 611 s (ONNX + engine).
  • foundationstereo-vit-s 994x1482, first call: portable + torch.compile 212 s (cold Inductor cache), TensorRT fp16 545 s (ONNX + engine).
  • foundationstereo-vit-l load / first call: original 10.86 s / 2.563 s (375x1242), portable 1.98 s / 2.99† s, kernels 1.17† s / 1.517† s
  • foundationstereo-vit-s load / first call: original 1.44 s / 1.058 s (375x1242), portable 1.03 s / 2.514† s, kernels 0.33† s / 1.12† s
  • † measured in the speed sweep after the portable load (or an earlier size) in the same process: file cache and kernels are warm, so the time is lower than a fresh start.
  • Protocol, median of 5 after 3 warm-up, uint8 CUDA input (synthetic for bench_inputs sizes), card defaults; gc before each load; peak = max over the first call and the timed calls

Measured

Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portableportable fp32kernelsnote
foundationstereo-vit-lETH3D two-view train, bad > 1 px % / EPE0.65 / 0.1680.65 / 0.1680.65 / 0.1680.65 / 0.1680.65 / 0.168possibly in the training data
foundationstereo-vit-lMiddlebury 2014 trainingH, bad > 2 px % / EPE2.48 / 0.5622.48 / 0.5612.47 / 0.5612.49 / 0.5622.48 / 0.561possibly in the training data
foundationstereo-vit-sETH3D two-view train, bad > 1 px % / EPE1.10 / 0.2021.10 / 0.2021.10 / 0.2021.10 / 0.2021.10 / 0.202possibly in the training data
foundationstereo-vit-sMiddlebury 2014 trainingH, bad > 2 px % / EPE3.32 / 0.6793.32 / 0.6813.32 / 0.6803.32 / 0.6793.30 / 0.680possibly in the training data

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 468fba10, 2026-11-05.

Parity

CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.02 px and p99 <= 0.25 px.

variantkernels error vs original, mean / p99GPU, torchkernels used
foundationstereo-vit-l0.00035 / 0.00121RTX 5060 Ti, torch 2.14.1+cu130cudnn_hacc@cudnn92400, fastbb@7d058322634d:sm_120, fs@f6f8de2cca1e:sm_120, fs_cutlass@de5f3c10aa1b:sm_120
foundationstereo-vit-s0.00029 / 0.0009RTX 5060 Ti, torch 2.14.1+cu130cudnn_hacc@cudnn92400, fastbb@7d058322634d:sm_120, fs@f6f8de2cca1e:sm_120, fs_cutlass@de5f3c10aa1b:sm_120

Limitations

  • ONNX export needs torch >= 2.14 (older exporters fail on this network).

Example output

One fixed input through this checkpoint, the same for every model of the task (kernels · float16 · NVIDIA GeForce RTX 5060 Ti).

Left view
Right view
Output
Motorcycle, Middlebury 2014 stereo (left and right view, downscaled 4x), left view
Left view · 100%

Input: Motorcycle, Middlebury 2014 stereo (left and right view, downscaled 4x), Scharstein et al., Middlebury College (research dataset, no licence stated).