Models · Frame interpolation

GIMM-VFI

GIMM-VFI (GIMM-VFI-R and its perceptual GIMM-VFI-R-P, Guo et al., NeurIPS 2024): makes the frame at any time between two video frames from a learned continuous motion field (RAFT flow), for slow motion and higher frame rates. Non-commercial code and weights (S-Lab License 1.0).

experimental Fork of GSeanCDAT/GIMM-VFI@dbc5644 · backends: portable, kernels

Checkpoint

Paper 2024 Added in 0.2.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 1629 ms 1080x1920, fp16
vs the faster original – original not timed
Peak memory 4,587 MiB original not measured
Accuracy vs original fp32 ±0.00 kernels psnr, DAVIS 2017 480p (28.94 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install "fastforks-core[nc]"

from fastforks import interpolation

model = interpolation.load("gimm-vfi-r")  # kernels on a supported GPU, else portable PyTorch
result = model(frame1, frame2)
result.frame
result.backend  # what ran: mode, device, dtype, kernels

The weights are non-commercial: fastforks downloads them only with FASTFORKS_NONCOMMERCIAL=1 set.

fastforks bench gimm-vfi-r

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

BackenddtypeTime per call (ms)vs originalPeak memory
allocated / reserved
Load (s)First call (s)
Original fp32fp32not measured ––––
Original fp16fp16out of memory––––
Portable fp32fp32not measured ––––
Portablefp16 2622–9,899 MiB/10.2 GiB0.32.62
Kernelsfp16 1629–4,587 MiB/5,362 MiB0.61.63
Portable + torch.compile 1498–11.8 GiB/12.6 GiB307 (build)–
TensorRTTensorRT parser: Sc nodefails––––

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-14. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14Originalfp32cannot run here–1080x1920
Portablefp16fails: worker rc=1: line 544, in main return worker_local(json.loads(a.spec)) File "/Us–1080x1920
Portablefp32out of memory–1080x1920
Kernelsfp16not implemented–1080x1920

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Non-commercial only

Code
LicenseRef-noncommercial LICENSE
Weights
LicenseRef-noncommercial source
Training data
  • code: LicenseRef-noncommercial (noncommercial)
  • weights: LicenseRef-noncommercial

Our reading of the licences, not legal advice.

Weights

gimmvfi_r_arb.pt 79.3 MB· hosted on Hugging Face

fastforks downloads and checks these files on first use (fastforks download gimm-vfi-r). Non-commercial weights need FASTFORKS_NONCOMMERCIAL=1.

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

DAVIS 2017 480p

2 frame triplets per video (180), PSNR / SSIM

Rundtypepsnr ↑ssim ↑Δ psnr vs fp32 originaltripletsms / triplet
Original fp32fp3228.940.8543 reference 180 –
Original fp16fp1628.940.8543 ±0.00 180 –
Portable fp32not measured
Portablefp1628.940.8543 ±0.00 180 –
Kernelsfp1628.940.8543 ±0.00 180 –

DAVIS 2017 trainval 480p (JPEG), frames 0 / 2 and 20 / 22 of each of the 90 videos give the frame 1 / 21 at time 0.5, compared with the real one: PSNR and SSIM (Gaussian 11 x 11, sigma 1.5) of the output rounded to uint8, on RGB in [0, 1], averaged over the triplets for the 180-triplet benchmark. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks e97998ab, 2026-11-14.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 5e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0003 and p99 <= 0.003 (absolute error of the output frame).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.00005 not recorded
GPU default dtypes, mean |diff|≤ 0.0003 0.00006
GPU default dtypes, p99 |diff|≤ 0.003 0.00045

Recorded for gimm-vfi-r on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: cudnn_haccraft_flow

Details

GIMM-VFI (Guo et al., NeurIPS 2024), the RAFT-based GIMM-VFI-R: arbitrary-time video frame interpolation. RAFT estimates the flow between both frames in both directions; a generalizable implicit motion model (a SIREN MLP on time and position, conditioned on the forward-splatted flow latents) predicts the flow at the requested time; an AMT-style decoder refines it with the bidirectional feature correlation and blends both warped frames. Any time in [0, 1]. `gimm-vfi-r` is trained for accuracy, `gimm-vfi-r-p` adds an LPIPS loss for sharper, perceptually better frames.

Input / output
Input: two RGB images (consecutive frames). Output: the frame between them at timestep (0.5 by default; 0 = the first, 1 = the second), RGB in [0, 1].
Input size
any H x W >= 128, padded to a multiple of 32; tested up to 1024x2048 (13.4 GiB allocated)
Outputs
frame
CUDA kernels
  • raft_flowRAFT optical flow kernels
  • cudnn_haccconvolutions with fp16 accumulation (cuDNN)
Apple (MPS)
fp32, fp16
Full reference, generated from the repository

GIMM-VFI (GIMM-VFI-R and its perceptual GIMM-VFI-R-P, Guo et al., NeurIPS 2024): makes the frame at any time between two video frames from a learned continuous motion field (RAFT flow), for slow motion and higher frame rates. Non-commercial code and weights (S-Lab License 1.0).

Commercial use: no (code: LicenseRef-noncommercial (noncommercial); weights: LicenseRef-noncommercial).

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
gimm-vfi-rsynthetic (1080x1920)original
portable
kernels
portable + torch.compile
TensorRT fp16
OOM
2622
1629
1498
fails: TensorRT parser: Sc node
–
9899 / 10400
4587 / 5362
12051 / 12880
–
gimm-vfi-r-psynthetic (1080x1920)original
portable
kernels
portable + torch.compile
TensorRT fp16
OOM
2622
1628
1498
fails: TensorRT parser: Sc node
–
9899 / 10400
4587 / 5282
12051 / 12880
–
Details
  • Original code: https://github.com/GSeanCDAT/GIMM-VFI at dbc56449994a
  • Code licence: LicenseRef-noncommercial (noncommercial); files: LICENSE
  • Weights licence: LicenseRef-noncommercial
  • Licence evidence: https://github.com/GSeanCDAT/GIMM-VFI/blob/dbc56449994a3c2e045d46fd46bed570239913f8/LICENSE ; https://github.com/GSeanCDAT/GIMM-VFI/blob/dbc56449994a3c2e045d46fd46bed570239913f8/README.md (weights: https://huggingface.co/GSean/GIMM-VFI)
  • Training data: Vimeo-90K (septuplets / triplets for frame interpolation) (configs/gimmvfi/gimmvfi_r_arb.yaml: Vimeo-90K septuplets with arbitrary-time targets; the RAFT flow estimator starts from the raft-things checkpoint (FlyingChairs + FlyingThings3D).)
  • Kernel libraries: raft_flow, cudnn_hacc
  • Variants: gimm-vfi-r, gimm-vfi-r-p
  • Outputs: frame
  • Input size: any H x W >= 128, padded to a multiple of 32
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): gimm-vfi-r 1024x2048 (13.4 GiB peak allocated, 14.5 reserved); gimm-vfi-r-p 1024x2048 (13.0 GiB peak allocated, 14.2 reserved)
  • gimm-vfi-r 1080x1920, first call: portable + torch.compile 307 s (cold Inductor cache).
  • gimm-vfi-r-p 1080x1920, first call: portable + torch.compile 307 s (cold Inductor cache).
  • gimm-vfi-r load / first call: original 2.05 s / 1.336 s (480x854), portable 0.27 s / 2.622 s, kernels 0.59 s / 1.631 s
  • gimm-vfi-r-p load / first call: original 2.96 s / 1.438 s (480x854), portable 0.27 s / 0.997 s (480x854), kernels 0.58 s / 0.684 s (480x854)
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

DAVIS 2017 trainval 480p (JPEG), frames 0 / 2 and 20 / 22 of each of the 90 videos give the frame 1 / 21 at time 0.5, compared with the real one: PSNR and SSIM (Gaussian 11 x 11, sigma 1.5) of the output rounded to uint8, on RGB in [0, 1], averaged over the triplets for the 180-triplet benchmark. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portablekernelsnote
gimm-vfi-rDAVIS 2017 480p, 2 frame triplets per video (180), PSNR / SSIM28.94 / 0.854328.94 / 0.854328.94 / 0.854328.94 / 0.8543
gimm-vfi-r-pDAVIS 2017 480p, 2 frame triplets per video (180), PSNR / SSIM28.81 / 0.848128.81 / 0.848128.81 / 0.848128.81 / 0.8481

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks e97998ab, 2026-11-14.

Parity

CPU, float32: max |diff| <= 5e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0003 and p99 <= 0.003 (absolute error of the output frame).

variantkernels error vs original, mean / p99GPU, torchkernels used
gimm-vfi-r6e-05 / 0.00045RTX 5060 Ti, torch 2.14.1+cu130cudnn_hacc@cudnn92400, raft_flow@a101a33607b6:sm_120
gimm-vfi-r-p5e-05 / 0.00029RTX 5060 Ti, torch 2.14.1+cu130cudnn_hacc@cudnn92400, raft_flow@a101a33607b6:sm_120

Example output

No example rendered for this checkpoint yet.