GIMM-VFI
GIMM-VFI (GIMM-VFI-R and its perceptual GIMM-VFI-R-P, Guo et al., NeurIPS 2024): makes the frame at any time between two video frames from a learned continuous motion field (RAFT flow), for slow motion and higher frame rates. Non-commercial code and weights (S-Lab License 1.0).
experimental Fork of GSeanCDAT/GIMM-VFI@dbc5644 · backends: portable, kernels
- Paper Generalizable Implicit Motion Modeling for Video Frame Interpolation NeurIPS 2024 · first released 11 Jul 2024
Paper 2024 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install "fastforks-core[nc]"
from fastforks import interpolation
model = interpolation.load("gimm-vfi-r") # kernels on a supported GPU, else portable PyTorch
result = model(frame1, frame2)
result.frame
result.backend # what ran: mode, device, dtype, kernels The weights are non-commercial: fastforks downloads them only with FASTFORKS_NONCOMMERCIAL=1 set.
fastforks bench gimm-vfi-r Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.
| Backend | dtype | Time per call (ms) | vs original | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | not measured | – | – | – | – |
| Original fp16 | fp16 | out of memory | – | – | – | – |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp16 | 2622 | – | 9,899 MiB/10.2 GiB | 0.3 | 2.62 |
| Kernels | fp16 | 1629 | – | 4,587 MiB/5,362 MiB | 0.6 | 1.63 |
| Portable + torch.compile | 1498 | – | 11.8 GiB/12.6 GiB | 307 (build) | – | |
| TensorRTTensorRT parser: Sc node | fails | – | – | – | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-14. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14 | Original | fp32 | cannot run here | – | 1080x1920 | |
| Portable | fp16 | fails: worker rc=1: line 544, in main return worker_local(json.loads(a.spec)) File "/Us | – | 1080x1920 | ||
| Portable | fp32 | out of memory | – | 1080x1920 | ||
| Kernels | fp16 | not implemented | – | 1080x1920 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Non-commercial only
- code: LicenseRef-noncommercial (noncommercial)
- weights: LicenseRef-noncommercial
Our reading of the licences, not legal advice.
Weights
gimmvfi_r_arb.pt 79.3 MB· hosted on Hugging Facefastforks downloads and checks these files on first use (fastforks download gimm-vfi-r). Non-commercial weights need FASTFORKS_NONCOMMERCIAL=1.
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
DAVIS 2017 480p
2 frame triplets per video (180), PSNR / SSIM
| Run | dtype | psnr ↑ | ssim ↑ | Δ psnr vs fp32 original | triplets | ms / triplet |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 28.94 | 0.8543 | reference | 180 | – |
| Original fp16 | fp16 | 28.94 | 0.8543 | ±0.00 | 180 | – |
| Portable fp32 | not measured | |||||
| Portable | fp16 | 28.94 | 0.8543 | ±0.00 | 180 | – |
| Kernels | fp16 | 28.94 | 0.8543 | ±0.00 | 180 | – |
DAVIS 2017 trainval 480p (JPEG), frames 0 / 2 and 20 / 22 of each of the 90 videos give the frame 1 / 21 at time 0.5, compared with the real one: PSNR and SSIM (Gaussian 11 x 11, sigma 1.5) of the output rounded to uint8, on RGB in [0, 1], averaged over the triplets for the 180-triplet benchmark. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks e97998ab, 2026-11-14.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 5e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0003 and p99 <= 0.003 (absolute error of the output frame).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.00005 | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.0003 | 0.00006 |
| GPU default dtypes, p99 |diff| | ≤ 0.003 | 0.00045 |
Recorded for gimm-vfi-r on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: cudnn_haccraft_flow
Details
GIMM-VFI (Guo et al., NeurIPS 2024), the RAFT-based GIMM-VFI-R: arbitrary-time video frame interpolation. RAFT estimates the flow between both frames in both directions; a generalizable implicit motion model (a SIREN MLP on time and position, conditioned on the forward-splatted flow latents) predicts the flow at the requested time; an AMT-style decoder refines it with the bidirectional feature correlation and blends both warped frames. Any time in [0, 1]. `gimm-vfi-r` is trained for accuracy, `gimm-vfi-r-p` adds an LPIPS loss for sharper, perceptually better frames.
- Input / output
- Input: two RGB images (consecutive frames). Output: the frame between them at
timestep(0.5 by default; 0 = the first, 1 = the second), RGB in [0, 1]. - Input size
- any H x W >= 128, padded to a multiple of 32; tested up to 1024x2048 (13.4 GiB allocated)
- Outputs
frame- CUDA kernels
raft_flowRAFT optical flow kernelscudnn_haccconvolutions with fp16 accumulation (cuDNN)
- Apple (MPS)
- fp32, fp16
Full reference, generated from the repository
GIMM-VFI (GIMM-VFI-R and its perceptual GIMM-VFI-R-P, Guo et al., NeurIPS 2024): makes the frame at any time between two video frames from a learned continuous motion field (RAFT flow), for slow motion and higher frame rates. Non-commercial code and weights (S-Lab License 1.0).
Commercial use: no (code: LicenseRef-noncommercial (noncommercial); weights: LicenseRef-noncommercial).
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
gimm-vfi-r | synthetic (1080x1920) | original portable kernels portable + torch.compile TensorRT fp16 | OOM 2622 1629 1498 fails: TensorRT parser: Sc node | – 9899 / 10400 4587 / 5362 12051 / 12880 – |
gimm-vfi-r-p | synthetic (1080x1920) | original portable kernels portable + torch.compile TensorRT fp16 | OOM 2622 1628 1498 fails: TensorRT parser: Sc node | – 9899 / 10400 4587 / 5282 12051 / 12880 – |
Details
- Original code: https://github.com/GSeanCDAT/GIMM-VFI at
dbc56449994a - Code licence: LicenseRef-noncommercial (noncommercial); files:
LICENSE - Weights licence: LicenseRef-noncommercial
- Licence evidence: https://github.com/GSeanCDAT/GIMM-VFI/blob/dbc56449994a3c2e045d46fd46bed570239913f8/LICENSE ; https://github.com/GSeanCDAT/GIMM-VFI/blob/dbc56449994a3c2e045d46fd46bed570239913f8/README.md (weights: https://huggingface.co/GSean/GIMM-VFI)
- Training data: Vimeo-90K (septuplets / triplets for frame interpolation) (configs/gimmvfi/gimmvfi_r_arb.yaml: Vimeo-90K septuplets with arbitrary-time targets; the RAFT flow estimator starts from the raft-things checkpoint (FlyingChairs + FlyingThings3D).)
- Kernel libraries:
raft_flow,cudnn_hacc - Variants:
gimm-vfi-r,gimm-vfi-r-p - Outputs: frame
- Input size: any H x W >= 128, padded to a multiple of 32
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
gimm-vfi-r1024x2048 (13.4 GiB peak allocated, 14.5 reserved);gimm-vfi-r-p1024x2048 (13.0 GiB peak allocated, 14.2 reserved) gimm-vfi-r1080x1920, first call: portable + torch.compile 307 s (cold Inductor cache).gimm-vfi-r-p1080x1920, first call: portable + torch.compile 307 s (cold Inductor cache).gimm-vfi-rload / first call: original 2.05 s / 1.336 s (480x854), portable 0.27 s / 2.622 s, kernels 0.59 s / 1.631 sgimm-vfi-r-pload / first call: original 2.96 s / 1.438 s (480x854), portable 0.27 s / 0.997 s (480x854), kernels 0.58 s / 0.684 s (480x854)- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
DAVIS 2017 trainval 480p (JPEG), frames 0 / 2 and 20 / 22 of each of the 90 videos give the frame 1 / 21 at time 0.5, compared with the real one: PSNR and SSIM (Gaussian 11 x 11, sigma 1.5) of the output rounded to uint8, on RGB in [0, 1], averaged over the triplets for the 180-triplet benchmark. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | kernels | note |
|---|---|---|---|---|---|---|
gimm-vfi-r | DAVIS 2017 480p, 2 frame triplets per video (180), PSNR / SSIM | 28.94 / 0.8543 | 28.94 / 0.8543 | 28.94 / 0.8543 | 28.94 / 0.8543 | |
gimm-vfi-r-p | DAVIS 2017 480p, 2 frame triplets per video (180), PSNR / SSIM | 28.81 / 0.8481 | 28.81 / 0.8481 | 28.81 / 0.8481 | 28.81 / 0.8481 |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks e97998ab, 2026-11-14.
Parity
CPU, float32: max |diff| <= 5e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0003 and p99 <= 0.003 (absolute error of the output frame).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
gimm-vfi-r | 6e-05 / 0.00045 | RTX 5060 Ti, torch 2.14.1+cu130 | cudnn_hacc@cudnn92400, raft_flow@a101a33607b6:sm_120 |
gimm-vfi-r-p | 5e-05 / 0.00029 | RTX 5060 Ti, torch 2.14.1+cu130 | cudnn_hacc@cudnn92400, raft_flow@a101a33607b6:sm_120 |
Example output
No example rendered for this checkpoint yet.