Models · Optical flow

FlowFormer Small

FlowFormer Small (ECCV 2022): convolutional features and a latent cost-memory transformer for optical flow. Apache-2.0 code; the authors provide no separate checkpoint licence.

experimental Fork of drinkingcoder/FlowFormer-Official@6ba7ea8 · backends: portable, kernels

Checkpoint

Paper 2022 Added in 0.2.0 Evaluated with 0.2.0

Portable, RTX 5060 Ti 16 GB 337 ms 375x1242, fp32
vs the faster original 0.9× original fp16: 303 ms
Peak memory 1,404 MiB original: 2,382 MiB
Accuracy vs original fp32 ±0.000 portable EPE, KITTI 2015 train (10.952 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install "fastforks-core[timm]"

from fastforks import flow

model = flow.load("flowformer-small-chairs")  # kernels on a supported GPU, else portable PyTorch
result = model(frame1, frame2)
result.flow
result.backend  # what ran: mode, device, dtype, kernels
fastforks bench flowformer-small-chairs

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp32. Portable runs fp32 on CUDA.

Input: synthetic, 375x1242

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32upstream FlowFormer Small, iters=32, full frame padded to multiples of 8fp32 3520.86×3,122 MiB/3,718 MiB1.80.47
Original fp16torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8fp16 303–2,382 MiB/2,970 MiB0.20.37
Portable fp32fp32not measured ––––
Portablefp32 3370.90×1,404 MiB/1,698 MiB1.60.56
Kernelsnot measured ––––
Portable + torch.compile 2501.2×2,438 MiB/3,196 MiB99 (build)–
TensorRT fp16fp16 2021.5×2,006 MiBenginenot comparable118 (build)–

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-11 to 2026-11-15. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14Originalfp32out of memory–1080x1920
Portablefp16out of memory–1080x1920
Portablefp32out of memory–1080x1920
Kernelsfp32not implemented–1080x1920

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Commercial use allowed

Code
Apache-2.0 AND BSD-3-Clause AND WTFPL LICENSELICENSE-RAFTLICENSE-GMA
Weights
Apache-2.0 AND BSD-3-Clause AND WTFPL (assumed from code)(not stated by the authors; assumed to follow the code) source
Training data
  • weights licence not stated by the original authors; assumed to be the code licence

Our reading of the licences, not legal advice.

Weights

flowformer-small-chairs.pth 25.0 MB· inside a 69.0 MB archive· hosted on Google Drive

fastforks downloads and checks these files on first use (fastforks download flowformer-small-chairs).

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

KITTI 2015 train

EPE / Fl-all %

RundtypeEPE ↓Fl-all % ↓Δ EPE vs fp32 originalpairsms / pair
Original fp32fp3210.95236.72 reference 200 352
Original fp16fp1611.10236.80 −0.149 200 303
Portable fp32fp3210.95236.72 ±0.000 200 351
Portablefp3210.95236.72 ±0.000 200 351
Kernelsnot measured

Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-11; portable fp32, portable: RTX 5060 Ti, torch 2.14.1+cu130, fastforks a6ea0725+dir, 2026-11-11.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float32 (the default), portable and kernels: mean |diff| <= 0.0021 and p99 <= 0.013 (absolute error of the output flow).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.0001 not recorded
GPU default dtypes, mean |diff|≤ 0.0021 0.00039
GPU default dtypes, p99 |diff|≤ 0.013 0.00146

Recorded for flowformer-small-chairs on RTX 5060 Ti, torch 2.10.0+cu130, 2026-11-17, with these kernels: flow_upsample

Details

FlowFormer Small (ECCV 2022): convolutional features, a latent cost-memory transformer and recurrent flow decoding. The authors' Small checkpoints, with plain float32 inference.

Input / output
Input: two RGB images (consecutive frames). Output: per-pixel motion from the first to the second (dx, dy in pixels).
Input size
any H x W >= 64, padded to a multiple of 8; tested up to 1024x1024 (11.2 GiB allocated)
Options
  • iters=32>= 1· changes the output
Outputs
flow
CUDA kernels
  • flow_upsample
Apple (MPS)
fp32, fp16

Limitations

  • Only the Small architecture is included.
  • Portable runs float32 (the validated dtype): 338 ms and 1410 MB at 375x1242 against 303 ms and 2382 MB for the fp16-autocast original (EPE identical to the float32 original); switching the default to fp16 needs a full accuracy re-validation.
  • Full-frame cost memory grows quadratically with input area; upstream tiled evaluation is not included. Full-frame 1080p exceeded the RTX 5060 Ti 16GB memory in portable and original fp32/fp16 runs.
Full reference, generated from the repository

FlowFormer Small (ECCV 2022): convolutional features and a latent cost-memory transformer for optical flow. Apache-2.0 code; the authors provide no separate checkpoint licence.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
flowformer-small-chairssynthetic (375x1242)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
352
303
337
–
250
202
3122 / 3718
2382 / 2970
1404 / 1698
–
2438 / 3196
2087 / 2138
flowformer-small-thingssynthetic (375x1242)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
352
303
337
–
250
202
3122 / 3678
2382 / 3010
1404 / 1698
–
2438 / 3196
2087 / 2138
flowformer-small-sintelsynthetic (375x1242)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
352
303
336
–
250
202
3122 / 3718
2382 / 2990
1404 / 1698
–
2438 / 3196
2087 / 2138
Details
  • Original code: https://github.com/drinkingcoder/FlowFormer-Official at 6ba7ea82b453
  • Code licence: Apache-2.0 AND BSD-3-Clause AND WTFPL (permissive); files: LICENSE, LICENSE-RAFT, LICENSE-GMA
  • Weights licence: Apache-2.0 AND BSD-3-Clause AND WTFPL (assumed from code)
  • Licence evidence: https://github.com/drinkingcoder/FlowFormer-Official/blob/6ba7ea82b45394d3a7a25808399695427c9febd8/README.md#license
  • Training data: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa) (The authors' Small checkpoints follow the chairs, things and sintel training stages; per-variant training data follows configs/small_*.py.)
    • flowformer-small-chairs: FlyingChairs (Freiburg)
    • flowformer-small-things: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa)
    • flowformer-small-sintel: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), MPI Sintel optical flow, KITTI stereo / optical flow 2012 / 2015, HCI HD1K optical flow benchmark
  • Options: iters = 32 (>= 1)
  • Kernel libraries: flow_upsample
  • Variants: flowformer-small-chairs, flowformer-small-things, flowformer-small-sintel
  • Outputs: flow
  • Input size: any H x W >= 64, padded to a multiple of 8
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): flowformer-small-chairs 1024x1024 (11.2 GiB peak allocated, 12.3 reserved); flowformer-small-things 1024x1024 (11.2 GiB peak allocated, 12.3 reserved); flowformer-small-sintel 1024x1024 (11.2 GiB peak allocated, 12.3 reserved)
  • flowformer-small-chairs 375x1242, first call: portable + torch.compile 99 s (cold Inductor cache), TensorRT fp16 118 s (ONNX + engine).
  • flowformer-small-things 375x1242, first call: portable + torch.compile 99 s (cold Inductor cache), TensorRT fp16 118 s (ONNX + engine).
  • flowformer-small-sintel 375x1242, first call: portable + torch.compile 99 s (cold Inductor cache), TensorRT fp16 117 s (ONNX + engine).
  • flowformer-small-chairs original (fp32) 375x1242: upstream FlowFormer Small, iters=32, full frame padded to multiples of 8
  • flowformer-small-chairs original 375x1242: torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8
  • flowformer-small-things original (fp32) 375x1242: upstream FlowFormer Small, iters=32, full frame padded to multiples of 8
  • flowformer-small-things original 375x1242: torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8
  • flowformer-small-sintel original (fp32) 375x1242: upstream FlowFormer Small, iters=32, full frame padded to multiples of 8
  • flowformer-small-sintel original 375x1242: torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8
  • flowformer-small-chairs load / first call: original 0.16 s / 0.366 s (375x1242), portable 1.59 s / 0.561 s (375x1242)
  • flowformer-small-things load / first call: original 0.15 s / 0.304 s (375x1242), portable 1.42 s / 0.559 s (375x1242)
  • flowformer-small-sintel load / first call: original 0.15 s / 0.304 s (375x1242), portable 1.56 s / 0.581 s (375x1242)
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portablenote
flowformer-small-chairsKITTI 2015 train, EPE / Fl-all %10.952 / 36.7211.102 / 36.8010.952 / 36.72
flowformer-small-thingsKITTI 2015 train, EPE / Fl-all %6.884 / 17.927.034 / 17.916.884 / 17.92
flowformer-small-sintelKITTI 2015 train, EPE / Fl-all %4.657 / 6.354.206 / 6.304.657 / 6.35trained on this dataset's public images; not a zero-shot result

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks a6ea0725+dirty, 2026-11-11.

Parity

CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float32 (the default), portable and kernels: mean |diff| <= 0.0021 and p99 <= 0.013 (absolute error of the output flow).

variantkernels error vs original, mean / p99GPU, torchkernels used
flowformer-small-chairs0.00039 / 0.00146RTX 5060 Ti, torch 2.10.0+cu130flow_upsample@5eca8c1a6d3b:sm_120
flowformer-small-things0.00037 / 0.00168RTX 5060 Ti, torch 2.10.0+cu130flow_upsample@5eca8c1a6d3b:sm_120
flowformer-small-sintel0.00031 / 0.00148RTX 5060 Ti, torch 2.10.0+cu130flow_upsample@5eca8c1a6d3b:sm_120

Limitations

  • Only the Small architecture is included.
  • Portable runs float32 (the validated dtype): 338 ms and 1410 MB at 375x1242 against 303 ms and 2382 MB for the fp16-autocast original (EPE identical to the float32 original); switching the default to fp16 needs a full accuracy re-validation.
  • Full-frame cost memory grows quadratically with input area; upstream tiled evaluation is not included. Full-frame 1080p exceeded the RTX 5060 Ti 16GB memory in portable and original fp32/fp16 runs.

Example output

No example rendered for this checkpoint yet.