Models · Optical flow

MEMFOF

MEMFOF (MSU Graphics & Media Lab, ICCV 2025): memory-efficient multi-frame optical flow, a RAFT-style network that also looks at the previous frame and runs at full HD. BSD-3-Clause code and weights; some training sets are research-only.

experimental Fork of msu-video-group/memfof@a51de9f · backends: portable, kernels

Checkpoint

Paper 2025 Added in 0.2.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 427 ms 1080x1920, fp16
vs the faster original 1.2× original fp16: 531 ms
Peak memory 1,301 MiB original: 2,003 MiB
Accuracy vs original fp32 +0.001 kernels EPE, KITTI 2015 train (10.892 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install fastforks-core

from fastforks import flow

model = flow.load("memfof")  # kernels on a supported GPU, else portable PyTorch
result = model(frame1, frame2)
result.flow
result.backend  # what ran: mode, device, dtype, kernels
fastforks bench memfof

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32fp32not measured ––––
Original fp16torch.autocast(float16) around the call; upstream MEMFOF + scale_and_forward_flow, iters=8, scale=0, frames (image1, image1, image2)fp16 531–2,003 MiB/2,144 MiB0.90.74
Portable fp32fp32not measured ––––
Portablefp16 4331.2×1,911 MiB/2,196 MiB0.20.43
Kernelsfp16 4271.2×1,301 MiB/1,418 MiB1.80.43
Portable + torch.compile 2911.8×2,452 MiB/2,666 MiB51 (build)–
TensorRT fp16fp16 1603.3×1,638 MiBenginenot comparable110 (build)–

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-12. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-13Originalfp323625 5,212 MiB process–1080x1920
Portablefp323054 5,038 MiB processmatches1080x1920
Kernelsfp16not implemented–1080x1920

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Commercial use allowed

Code
BSD-3-Clause LICENSELICENSE-torchvision
Weights
BSD-3-Clause source
Training data

Our reading of the licences, not legal advice.

Weights

MEMFOF-Tartan-T-TSKH.safetensors 303 MB· hosted on Hugging Face

fastforks downloads and checks these files on first use (fastforks download memfof).

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

KITTI 2015 train

EPE / Fl-all %trained on this dataset’s public images: not zero-shot

RundtypeEPE ↓Fl-all % ↓Δ EPE vs fp32 originalpairsms / pair
Original fp32fp3210.89227.07 reference 200 208
Original fp16fp1610.89127.07 +0.002 200 106
Portable fp32fp3210.89227.07 ±0.000 200 165
Portablefp1610.89127.07 +0.002 200 90.1
Kernelsfp1610.89127.07 +0.001 200 90.4

Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 6f0a70e1, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 087f1817+unc, 2026-11-12.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 1e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.00099 and p99 <= 0.0068 (absolute error of the output flow).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.00001 not recorded
GPU default dtypes, mean |diff|≤ 0.00099 0.00065
GPU default dtypes, p99 |diff|≤ 0.0068 0.00304

Recorded for memfof on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12, with these kernels: raft_flow

Details

MEMFOF (Bargatin et al. 2025): memory-efficient multi-frame optical flow. A RAFT-style network at 1/16 resolution on three frames: a ResNet34 context encoder on all three, a GMA attention over the context, 1024-channel features with an all-pairs correlation pyramid toward the previous and the next frame, a ConvNeXt update block and convex 16x upsampling; it predicts the backward and the forward flow of the middle frame at once. The authors' released weights; the network is ported from msu-video-group/memfof. A pair (image1, image2) runs as upstream's first frame of a clip, the frames (image1, image1, image2), and returns its forward flow. Paper (three frames per prediction, the fine-tuned checkpoints): Spring test 1px 3.289 %, Sintel test EPE 0.963 (clean), KITTI 2015 test Fl-all 2.94 %.

Input / output
Input: two RGB images (consecutive frames). Output: per-pixel motion from the first to the second (dx, dy in pixels).
Input size
any H x W >= 256, padded to a multiple of 16; tested up to 1988x2880 (9,252 MiB allocated)
Options
  • iters=8>= 1· changes the output
Outputs
flow
CUDA kernels
  • raft_flowRAFT optical flow kernels
Apple (MPS)
fp32

Limitations

  • Pairs only: the multi-frame input (a previous frame and the feature cache of upstream's video demo) is not exposed; each pair runs as the first frame of a clip.
  • Full-frame 1080x1920 runs out of memory on the RTX 5060 Ti 16 GB in the upstream original and in portable; only kernels fit (7.2 GB).
Full reference, generated from the repository

MEMFOF (MSU Graphics & Media Lab, ICCV 2025): memory-efficient multi-frame optical flow, a RAFT-style network that also looks at the previous frame and runs at full HD. BSD-3-Clause code and weights; some training sets are research-only.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
memfofsynthetic (1080x1920)original
portable
kernels
portable + torch.compile
TensorRT fp16
531
433
427
291
160
2003 / 2144
1911 / 2196
1301 / 1418
2452 / 2666
2070 / 2132
memfof-kittisynthetic (375x1242)original
portable
kernels
portable + torch.compile
TensorRT fp16
927
386
380
258
fails: unsupported: engine build needs more than the 16 GB of this
1968 / 2276
1619 / 1856
1137 / 1218
2041 / 2226
–
memfof-sintelsynthetic (1080x1920)original
portable
kernels
portable + torch.compile
TensorRT fp16
OOM
OOM
1993
fails: OutOfMemoryError: CUDA out of memory. Tried to allocate 1.96
fails: OutOfMemoryError: CUDA out of memory. Tried to allocate 3.91
–
–
7249 / 8820
–
–
memfof-springsynthetic (1080x1920)original
portable
kernels
portable + torch.compile
TensorRT fp16
1035
433
427
291
160
2340 / 2676
1900 / 2196
1290 / 1398
2452 / 2666
2070 / 2112
Details
  • Original code: https://github.com/msu-video-group/memfof at a51de9fc59c6
  • Code licence: BSD-3-Clause (permissive); files: LICENSE, LICENSE-torchvision
  • Weights licence: BSD-3-Clause
  • Licence evidence: https://huggingface.co/egorchistov/optical-flow-MEMFOF-Tartan-T-TSKH/blob/6c6c9aa3ad64f93aee8efbc2f7a6e4535814ee96/README.md
  • Training data: TartanAir, Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), MPI Sintel optical flow, KITTI stereo / optical flow 2012 / 2015, HCI HD1K optical flow benchmark (TartanAir, FlyingThings3D (part of SceneFlow), then a mix of FlyingThings3D, Sintel, KITTI 2015 and HD1K (TSKH), as upstream's config/train/*.json)
    • memfof-kitti: TartanAir, Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), MPI Sintel optical flow, KITTI stereo / optical flow 2012 / 2015, HCI HD1K optical flow benchmark (TartanAir, FlyingThings3D, the TSKH mix, then fine-tuned on KITTI 2015)
    • memfof-sintel: TartanAir, Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), MPI Sintel optical flow, KITTI stereo / optical flow 2012 / 2015, HCI HD1K optical flow benchmark (TartanAir, FlyingThings3D, the TSKH mix, then fine-tuned on Sintel)
    • memfof-spring: TartanAir, Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), MPI Sintel optical flow, KITTI stereo / optical flow 2012 / 2015, HCI HD1K optical flow benchmark, Spring (scene flow / optical flow / stereo, University of Stuttgart) (TartanAir, FlyingThings3D, the TSKH mix, then fine-tuned on Spring)
  • Options: iters = 8 (>= 1)
  • Kernel libraries: raft_flow
  • Variants: memfof, memfof-kitti, memfof-sintel, memfof-spring
  • Outputs: flow
  • Input size:
    • memfof, memfof-spring: any H x W >= 256, padded to a multiple of 16
    • memfof-kitti, memfof-sintel: any H x W >= 128, padded to a multiple of 16
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): memfof 1988x2880 (9.0 GiB peak allocated, 10.6 reserved); memfof-kitti 1024x1024 (5.2 GiB peak allocated, 6.1 reserved); memfof-sintel 1024x1024 (5.2 GiB peak allocated, 6.1 reserved); memfof-spring 1988x2880 (9.0 GiB peak allocated, 10.6 reserved)
  • memfof 1080x1920, first call: portable + torch.compile 51 s (cold Inductor cache), TensorRT fp16 110 s (ONNX + engine).
  • memfof-kitti 375x1242, first call: portable + torch.compile 49 s (cold Inductor cache).
  • memfof-spring 1080x1920, first call: portable + torch.compile 45 s (cold Inductor cache), TensorRT fp16 96 s (ONNX + engine).
  • memfof original 1080x1920: torch.autocast(float16) around the call; upstream MEMFOF + scale_and_forward_flow, iters=8, scale=0, frames (image1, image1, image2)
  • memfof-kitti original 375x1242: upstream MEMFOF + scale_and_forward_flow, iters=8, scale=1, frames (image1, image1, image2)
  • memfof-spring original 1080x1920: upstream MEMFOF + scale_and_forward_flow, iters=8, scale=0, frames (image1, image1, image2)
  • memfof load / first call: original 2.06 s / 1.293 s, portable 0.22 s / 0.415 s (375x1242), kernels 1.84 s / 0.286 s (375x1242)
  • memfof-kitti load / first call: original 2.59 s / 0.961 s (375x1242), portable 0.21 s / 0.651 s (375x1242), kernels 1.07 s / 0.565 s (375x1242)
  • memfof-sintel load / first call: original 1.3 s / 0.897 s (375x1242), portable 0.69 s / 0.574 s (375x1242), kernels 2.49 s / 1.994 s
  • memfof-spring load / first call: original 1.12 s / 1.037 s, portable 0.71 s / 0.433 s, kernels 0.51 s / 0.632 s
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portableportable fp32kernelsnote
memfofKITTI 2015 train, EPE / Fl-all %10.892 / 27.0710.891 / 27.0710.891 / 27.0710.892 / 27.0710.891 / 27.07trained on this dataset's public images; not a zero-shot result

Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 087f1817+uncommitted, 2026-11-12; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 6f0a70e1, 2026-11-17; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 087f1817+uncommitted, 2026-11-12.

Parity

CPU, float32: max |diff| <= 1e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.00099 and p99 <= 0.0068 (absolute error of the output flow).

variantkernels error vs original, mean / p99GPU, torchkernels used
memfof0.00065 / 0.00304RTX 5060 Ti, torch 2.14.1+cu130raft_flow@a101a33607b6:sm_120
memfof-kitti0.00035 / 0.00157RTX 5060 Ti, torch 2.14.1+cu130raft_flow@a101a33607b6:sm_120
memfof-sintel0.00041 / 0.00206RTX 5060 Ti, torch 2.14.1+cu130raft_flow@a101a33607b6:sm_120
memfof-spring0.00066 / 0.00442RTX 5060 Ti, torch 2.14.1+cu130raft_flow@a101a33607b6:sm_120

Limitations

  • Pairs only: the multi-frame input (a previous frame and the feature cache of upstream's video demo) is not exposed; each pair runs as the first frame of a clip.
  • Full-frame 1080x1920 runs out of memory on the RTX 5060 Ti 16 GB in the upstream original and in portable; only kernels fit (7.2 GB).

Example output

No example rendered for this checkpoint yet.