FlowFormer Small
FlowFormer Small (ECCV 2022): convolutional features and a latent cost-memory transformer for optical flow. Apache-2.0 code; the authors provide no separate checkpoint licence.
experimental Fork of drinkingcoder/FlowFormer-Official@6ba7ea8 · backends: portable, kernels
- Paper FlowFormer: A Transformer Architecture for Optical Flow ECCV 2022 · first released 30 Mar 2022
Paper 2022 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install "fastforks-core[timm]"
from fastforks import flow
model = flow.load("flowformer-small-chairs") # kernels on a supported GPU, else portable PyTorch
result = model(frame1, frame2)
result.flow
result.backend # what ran: mode, device, dtype, kernels fastforks bench flowformer-small-chairs Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp32. Portable runs fp32 on CUDA.
Input: synthetic, 375x1242
| Backend | dtype | Time per call (ms) | vs original fp16 | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32upstream FlowFormer Small, iters=32, full frame padded to multiples of 8 | fp32 | 352 | 0.86× | 3,122 MiB/3,718 MiB | 1.8 | 0.47 |
| Original fp16torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8 | fp16 | 303 | – | 2,382 MiB/2,970 MiB | 0.2 | 0.37 |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp32 | 337 | 0.90× | 1,404 MiB/1,698 MiB | 1.6 | 0.56 |
| Kernels | not measured | – | – | – | – | |
| Portable + torch.compile | 250 | 1.2× | 2,438 MiB/3,196 MiB | 99 (build) | – | |
| TensorRT fp16 | fp16 | 202 | 1.5× | 2,006 MiBenginenot comparable | 118 (build) | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-11 to 2026-11-15. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14 | Original | fp32 | out of memory | – | 1080x1920 | |
| Portable | fp16 | out of memory | – | 1080x1920 | ||
| Portable | fp32 | out of memory | – | 1080x1920 | ||
| Kernels | fp32 | not implemented | – | 1080x1920 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- Apache-2.0 AND BSD-3-Clause AND WTFPL LICENSELICENSE-RAFTLICENSE-GMA
- Weights
- Apache-2.0 AND BSD-3-Clause AND WTFPL (assumed from code)(not stated by the authors; assumed to follow the code) source
- Training data
-
- FlyingChairs (Freiburg) research-only, non-commercial
- weights licence not stated by the original authors; assumed to be the code licence
Our reading of the licences, not legal advice.
Weights
flowformer-small-chairs.pth 25.0 MB· inside a 69.0 MB archive· hosted on Google Drivefastforks downloads and checks these files on first use (fastforks download flowformer-small-chairs).
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
KITTI 2015 train
EPE / Fl-all %
| Run | dtype | EPE ↓ | Fl-all % ↓ | Δ EPE vs fp32 original | pairs | ms / pair |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 10.952 | 36.72 | reference | 200 | 352 |
| Original fp16 | fp16 | 11.102 | 36.80 | −0.149 | 200 | 303 |
| Portable fp32 | fp32 | 10.952 | 36.72 | ±0.000 | 200 | 351 |
| Portable | fp32 | 10.952 | 36.72 | ±0.000 | 200 | 351 |
| Kernels | not measured | |||||
Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-11; portable fp32, portable: RTX 5060 Ti, torch 2.14.1+cu130, fastforks a6ea0725+dir, 2026-11-11.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float32 (the default), portable and kernels: mean |diff| <= 0.0021 and p99 <= 0.013 (absolute error of the output flow).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.0001 | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.0021 | 0.00039 |
| GPU default dtypes, p99 |diff| | ≤ 0.013 | 0.00146 |
Recorded for flowformer-small-chairs on RTX 5060 Ti, torch 2.10.0+cu130, 2026-11-17, with these kernels: flow_upsample
Details
FlowFormer Small (ECCV 2022): convolutional features, a latent cost-memory transformer and recurrent flow decoding. The authors' Small checkpoints, with plain float32 inference.
- Input / output
- Input: two RGB images (consecutive frames). Output: per-pixel motion from the first to the second (dx, dy in pixels).
- Input size
- any H x W >= 64, padded to a multiple of 8; tested up to 1024x1024 (11.2 GiB allocated)
- Options
iters=32>= 1· changes the output
- Outputs
flow- CUDA kernels
flow_upsample
- Apple (MPS)
- fp32, fp16
Limitations
- Only the Small architecture is included.
- Portable runs float32 (the validated dtype): 338 ms and 1410 MB at 375x1242 against 303 ms and 2382 MB for the fp16-autocast original (EPE identical to the float32 original); switching the default to fp16 needs a full accuracy re-validation.
- Full-frame cost memory grows quadratically with input area; upstream tiled evaluation is not included. Full-frame 1080p exceeded the RTX 5060 Ti 16GB memory in portable and original fp32/fp16 runs.
Full reference, generated from the repository
FlowFormer Small (ECCV 2022): convolutional features and a latent cost-memory transformer for optical flow. Apache-2.0 code; the authors provide no separate checkpoint licence.
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
flowformer-small-chairs | synthetic (375x1242) | original (fp32) original portable kernels portable + torch.compile TensorRT fp16 | 352 303 337 – 250 202 | 3122 / 3718 2382 / 2970 1404 / 1698 – 2438 / 3196 2087 / 2138 |
flowformer-small-things | synthetic (375x1242) | original (fp32) original portable kernels portable + torch.compile TensorRT fp16 | 352 303 337 – 250 202 | 3122 / 3678 2382 / 3010 1404 / 1698 – 2438 / 3196 2087 / 2138 |
flowformer-small-sintel | synthetic (375x1242) | original (fp32) original portable kernels portable + torch.compile TensorRT fp16 | 352 303 336 – 250 202 | 3122 / 3718 2382 / 2990 1404 / 1698 – 2438 / 3196 2087 / 2138 |
Details
- Original code: https://github.com/drinkingcoder/FlowFormer-Official at
6ba7ea82b453 - Code licence: Apache-2.0 AND BSD-3-Clause AND WTFPL (permissive); files:
LICENSE,LICENSE-RAFT,LICENSE-GMA - Weights licence: Apache-2.0 AND BSD-3-Clause AND WTFPL (assumed from code)
- Licence evidence: https://github.com/drinkingcoder/FlowFormer-Official/blob/6ba7ea82b45394d3a7a25808399695427c9febd8/README.md#license
- Training data: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa) (The authors' Small checkpoints follow the chairs, things and sintel training stages; per-variant training data follows configs/small_*.py.)
flowformer-small-chairs: FlyingChairs (Freiburg)flowformer-small-things: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa)flowformer-small-sintel: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), MPI Sintel optical flow, KITTI stereo / optical flow 2012 / 2015, HCI HD1K optical flow benchmark
- Options:
iters= 32 (>= 1) - Kernel libraries:
flow_upsample - Variants:
flowformer-small-chairs,flowformer-small-things,flowformer-small-sintel - Outputs: flow
- Input size: any H x W >= 64, padded to a multiple of 8
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
flowformer-small-chairs1024x1024 (11.2 GiB peak allocated, 12.3 reserved);flowformer-small-things1024x1024 (11.2 GiB peak allocated, 12.3 reserved);flowformer-small-sintel1024x1024 (11.2 GiB peak allocated, 12.3 reserved) flowformer-small-chairs375x1242, first call: portable + torch.compile 99 s (cold Inductor cache), TensorRT fp16 118 s (ONNX + engine).flowformer-small-things375x1242, first call: portable + torch.compile 99 s (cold Inductor cache), TensorRT fp16 118 s (ONNX + engine).flowformer-small-sintel375x1242, first call: portable + torch.compile 99 s (cold Inductor cache), TensorRT fp16 117 s (ONNX + engine).flowformer-small-chairsoriginal (fp32) 375x1242: upstream FlowFormer Small, iters=32, full frame padded to multiples of 8flowformer-small-chairsoriginal 375x1242: torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8flowformer-small-thingsoriginal (fp32) 375x1242: upstream FlowFormer Small, iters=32, full frame padded to multiples of 8flowformer-small-thingsoriginal 375x1242: torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8flowformer-small-sinteloriginal (fp32) 375x1242: upstream FlowFormer Small, iters=32, full frame padded to multiples of 8flowformer-small-sinteloriginal 375x1242: torch.autocast(float16) around the call; upstream FlowFormer Small, iters=32, full frame padded to multiples of 8flowformer-small-chairsload / first call: original 0.16 s / 0.366 s (375x1242), portable 1.59 s / 0.561 s (375x1242)flowformer-small-thingsload / first call: original 0.15 s / 0.304 s (375x1242), portable 1.42 s / 0.559 s (375x1242)flowformer-small-sintelload / first call: original 0.15 s / 0.304 s (375x1242), portable 1.56 s / 0.581 s (375x1242)- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | note |
|---|---|---|---|---|---|
flowformer-small-chairs | KITTI 2015 train, EPE / Fl-all % | 10.952 / 36.72 | 11.102 / 36.80 | 10.952 / 36.72 | |
flowformer-small-things | KITTI 2015 train, EPE / Fl-all % | 6.884 / 17.92 | 7.034 / 17.91 | 6.884 / 17.92 | |
flowformer-small-sintel | KITTI 2015 train, EPE / Fl-all % | 4.657 / 6.35 | 4.206 / 6.30 | 4.657 / 6.35 | trained on this dataset's public images; not a zero-shot result |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks a6ea0725+dirty, 2026-11-11.
Parity
CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float32 (the default), portable and kernels: mean |diff| <= 0.0021 and p99 <= 0.013 (absolute error of the output flow).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
flowformer-small-chairs | 0.00039 / 0.00146 | RTX 5060 Ti, torch 2.10.0+cu130 | flow_upsample@5eca8c1a6d3b:sm_120 |
flowformer-small-things | 0.00037 / 0.00168 | RTX 5060 Ti, torch 2.10.0+cu130 | flow_upsample@5eca8c1a6d3b:sm_120 |
flowformer-small-sintel | 0.00031 / 0.00148 | RTX 5060 Ti, torch 2.10.0+cu130 | flow_upsample@5eca8c1a6d3b:sm_120 |
Limitations
- Only the Small architecture is included.
- Portable runs float32 (the validated dtype): 338 ms and 1410 MB at 375x1242 against 303 ms and 2382 MB for the fp16-autocast original (EPE identical to the float32 original); switching the default to fp16 needs a full accuracy re-validation.
- Full-frame cost memory grows quadratically with input area; upstream tiled evaluation is not included. Full-frame 1080p exceeded the RTX 5060 Ti 16GB memory in portable and original fp32/fp16 runs.
Example output
No example rendered for this checkpoint yet.