Models · Point tracking

TAPNext

TAPNext (Google DeepMind, 2025): tracks any point through a video, online one frame at a time, as next-token prediction with a ViT + linear-recurrence (RG-LRU) backbone; BootsTAPNext-B and the long-video TAPNext++. Apache-2.0 code and weights.

experimental Fork of google-deepmind/tapnet@ceedd60 · backends: portable, kernels

Checkpoint

Paper 2025 Added in 0.2.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 381 ms 480x854, fp16
vs the faster original 2.4× original fp16: 909 ms
Peak memory 1,205 MiB original: 1,721 MiB
Accuracy vs original fp32 +0.01 kernels aj, TAP-Vid DAVIS (query first) (71.01 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install fastforks-core

from fastforks import tracking

model = tracking.load("tapnext")  # kernels on a supported GPU, else portable PyTorch
result = model(video, queries)
result.tracks  # also: visibility
result.backend  # what ran: mode, device, dtype, kernels
fastforks bench tapnext

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

Input: synthetic, 480x854 (the task’s standard size)

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32fp32not measured ––––
Original fp16the authors' mixed-precision option (float16); upstream TAPNext, online one frame per call; frames resized to 256 x 256 (antialiased bilinear)fp16 909–1,721 MiB/1,902 MiB13.10.90
Portable fp32fp32not measured ––––
Portablefp16 5281.7×1,220 MiB/1,294 MiB2.40.67
Kernelsfp16 3812.4×1,205 MiB/1,254 MiB3.00.54
Portable + torch.compile 3982.3×1,164 MiB/1,272 MiB41 (build)–
TensorRT fp16fp16 69.413×750 MiBenginenot comparable366 (build)–

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-12 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14Originalfp326104 3,210 MiB process–480x854
Portablefp165998 3,755 MiB processmatches480x854
Portablefp325855 3,754 MiB processmatches480x854
Kernelsfp16not implemented–480x854

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Commercial use allowed

Code
Apache-2.0 LICENSE
Weights
Apache-2.0 source
Training data
Upstream: TAPNext is trained on synthetic Kubric MOVi-F sequences and fine-tuned with BootsTAP self-training on unlabelled real videos (not released); TAPNext++ is further fine-tuned on PointOdyssey and Kubric-1024 (1024-frame synthetic sequences). None has a dataset entry here yet.

Our reading of the licences, not legal advice.

Weights

bootstapnext_ckpt.npz 777 MB· hosted on Google Drive

fastforks downloads and checks these files on first use (fastforks download tapnext).

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

TAP-Vid DAVIS (query first)

every 3rd video, AJ / delta_avg / OA %

Rundtypeaj ↑delta_avg ↑oa ↑Δ aj vs fp32 originalvideosms / video
Original fp32fp3271.0182.0195.18 reference 10 –
Original fp16fp1671.0282.0295.19 +0.01 10 –
Portable fp32fp3271.0182.0195.18 ±0.00 10 –
Portablefp1671.0282.0295.19 +0.01 10 –
Kernelsfp1671.0282.0395.23 +0.01 10 –

Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16, portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17.

TAP-Vid DAVIS (query first)

AJ / delta_avg / OA %

Rundtypeaj ↑delta_avg ↑oa ↑Δ aj vs fp32 originalvideosms / video
Original fp32fp3266.5579.4892.16 reference 30 –
Original fp16fp1666.5579.4992.17 ±0.00 30 –
Portable fp32fp3266.5579.4892.16 ±0.00 30 –
Portablefp1666.5579.4992.17 ±0.00 30 –
Kernelsfp1666.5679.5092.18 +0.01 30 –

Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16, portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-18.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 4.6e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0063 and p99 <= 0.036 (absolute error of the output tracks).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.000046 not recorded
GPU default dtypes, mean |diff|≤ 0.0063 0.00369
GPU default dtypes, p99 |diff|≤ 0.036 0.02116

Recorded for tapnext on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12, with these kernels: fvm_conv

Details

TAPNext (Zholus, Doersch, Yang, Koppula et al., Google DeepMind 2025): tracking any point as next-token prediction. A TRecViT-B (12 blocks, each a gated linear recurrence over time per token and a ViT layer over the tokens of a frame) reads 8 x 8 patches of 256 x 256 frames plus one token per query point and predicts, frame by frame, the point's position (512-bin classification, windowed soft-argmax) and whether it is visible. Online: each frame only sees the past.

Input / output
Input: a video (its frames in order) and query points (t, x, y). Output: the position of every query in every frame and the probability that it is visible there.
Input size
any H x W >= 32, padded to a multiple of 32; tested up to 3000x4000 (4,429 MiB allocated)
Outputs
tracks, visibility
CUDA kernels
  • fvm_convfused convolutions for image classifiers
Apple (MPS)
fp32, fp16

Limitations

  • Online, causal: a point is tracked from its query frame on; in frames before it the model has no information (the tracks there are its guess, with a low visibility). Query on the first frame, or run the reversed video for the backward part.
  • Frames are resized to 256 x 256 (the training size), whatever their size and aspect ratio.
  • One video per call; all queries of a call share the video.
Full reference, generated from the repository

TAPNext (Google DeepMind, 2025): tracks any point through a video, online one frame at a time, as next-token prediction with a ViT + linear-recurrence (RG-LRU) backbone; BootsTAPNext-B and the long-video TAPNext++. Apache-2.0 code and weights.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
tapnextsynthetic (480x854)original
portable
kernels
portable + torch.compile
TensorRT fp16
909
528
381
398
69.4
1721 / 1902
1220 / 1294
1205 / 1254
1164 / 1272
1498 / 1660
tapnext-ppsynthetic (480x854)original
portable
kernels
portable + torch.compile
TensorRT fp16
1493
529
381
395
69.5
1527 / 1718
1220 / 1294
1205 / 1254
1164 / 1272
1498 / 1660
Details
  • Original code: https://github.com/google-deepmind/tapnet at ceedd60bb9f6
  • Code licence: Apache-2.0 (permissive); files: LICENSE
  • Weights licence: Apache-2.0
  • Licence evidence: https://github.com/google-deepmind/tapnet/blob/ceedd60bb9f6c9f7c74796e4445aff6c6ddf7c9b/README.md#license-and-disclaimer
  • Training data: Upstream: TAPNext is trained on synthetic Kubric MOVi-F sequences and fine-tuned with BootsTAP self-training on unlabelled real videos (not released); TAPNext++ is further fine-tuned on PointOdyssey and Kubric-1024 (1024-frame synthetic sequences). None has a dataset entry here yet.
  • Kernel libraries: fvm_conv
  • Variants: tapnext, tapnext-pp
  • Outputs: tracks, visibility
  • Input size: any H x W >= 32, padded to a multiple of 32
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): tapnext 3000x4000 (4.3 GiB peak allocated, 4.5 reserved); tapnext-pp 3000x4000 (4.3 GiB peak allocated, 4.5 reserved)
  • tapnext 480x854, first call: portable + torch.compile 41 s (cold Inductor cache), TensorRT fp16 366 s (ONNX + engine).
  • tapnext-pp 480x854, first call: portable + torch.compile 34 s (cold Inductor cache), TensorRT fp16 362 s (ONNX + engine).
  • tapnext original 480x854: the authors' mixed-precision option (float16); upstream TAPNext, online one frame per call; frames resized to 256 x 256 (antialiased bilinear)
  • tapnext-pp original 480x854: upstream TAPNext, online one frame per call; frames resized to 256 x 256 (antialiased bilinear)
  • tapnext load / first call: original 16.92 s / 2.165 s, portable 2.37 s / 0.666 s, kernels 3.01 s / 0.542 s
  • tapnext-pp load / first call: original 4.83 s / 1.802 s, portable 1.34 s / 0.772 s, kernels 1.28 s / 0.665 s
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portableportable fp32kernelsnote
tapnextTAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA %71.01 / 82.01 / 95.1871.02 / 82.02 / 95.1971.02 / 82.02 / 95.1971.01 / 82.01 / 95.1871.02 / 82.03 / 95.23
tapnextTAP-Vid DAVIS (query first), AJ / delta_avg / OA %66.55 / 79.48 / 92.1666.55 / 79.49 / 92.1766.55 / 79.49 / 92.1766.55 / 79.48 / 92.1666.56 / 79.50 / 92.18
tapnext-ppTAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA %71.48 / 82.43 / 94.9671.50 / 82.43 / 94.9671.50 / 82.43 / 94.9671.48 / 82.43 / 94.9671.47 / 82.41 / 94.98

Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17 to 2026-11-18; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12.

Parity

CPU, float32: max |diff| <= 4.6e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0063 and p99 <= 0.036 (absolute error of the output tracks).

variantkernels error vs original, mean / p99GPU, torchkernels used
tapnext0.00369 / 0.02116RTX 5060 Ti, torch 2.14.1+cu130fvm_conv@f1d064e539e1:sm_120
tapnext-pp0.00415 / 0.02383RTX 5060 Ti, torch 2.14.1+cu130fvm_conv@f1d064e539e1:sm_120

Limitations

  • Online, causal: a point is tracked from its query frame on; in frames before it the model has no information (the tracks there are its guess, with a low visibility). Query on the first frame, or run the reversed video for the backward part.
  • Frames are resized to 256 x 256 (the training size), whatever their size and aspect ratio.
  • One video per call; all queries of a call share the video.

Example output

No example rendered for this checkpoint yet.