Models · Point tracking

AllTracker

AllTracker (Harley et al., ICCV 2025): track every pixel of a query frame through a whole video offline, as long-range optical flow over 16-frame windows; point queries read the dense result. MIT code and weights.

experimental Fork of aharley/alltracker@e755313 · backends: portable, kernels

Checkpoint

Paper 2025 Added in 0.2.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 221 ms 480x854, fp16
vs the faster original 2.2× original fp16: 485 ms
Peak memory 1,076 MiB original: 2,100 MiB
Accuracy vs original fp32 −0.07 kernels aj, TAP-Vid DAVIS (query first) (67.45 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install fastforks-core

from fastforks import tracking

model = tracking.load("alltracker")  # kernels on a supported GPU, else portable PyTorch
result = model(video, queries)
result.tracks  # also: visibility
result.backend  # what ran: mode, device, dtype, kernels
fastforks bench alltracker

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

Input: synthetic, 480x854 (the task’s standard size)

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32fp32not measured ––––
Original fp16the authors' mixed-precision option (float16); aharley/alltracker@e7553135e7b3 Net.forward, 384 x 512, 4 itersfp16 485–2,100 MiB/2,318 MiB0.50.83
Portable fp32fp32not measured ––––
Portablefp16 3911.2×993 MiB/1,238 MiB0.92.08
Kernelsfp16 2212.2×1,076 MiB/1,288 MiB0.30.49
Portable + torch.compile 2841.7×1,273 MiB/1,358 MiB105 (build)–
TensorRTplanned for 0.2.1fails––––

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-13 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14Originalfp323628 6,081 MiB process–480x854
Portablefp163200 3,806 MiB processmatches480x854
Portablefp323508 4,571 MiB processmatches480x854
Kernelsfp16not implemented–480x854

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Commercial use allowed

Code
MIT LICENSE
Weights
MIT source
Training data

Our reading of the licences, not legal advice.

Weights

alltracker.pth 66.0 MB· hosted on Hugging Face

fastforks downloads and checks these files on first use (fastforks download alltracker).

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

TAP-Vid DAVIS (query first)

every 3rd video, AJ / delta_avg / OA %

Rundtypeaj ↑delta_avg ↑oa ↑Δ aj vs fp32 originalvideosms / video
Original fp32fp3267.4578.6690.82 reference 10 –
Original fp16fp1667.4178.6590.69 −0.04 10 –
Portable fp32fp3267.4278.6690.79 −0.03 10 –
Portablefp1667.4278.7190.70 −0.03 10 –
Kernelsfp1667.3878.6390.65 −0.07 10 –

Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13.

TAP-Vid DAVIS (query first)

AJ / delta_avg / OA %

Rundtypeaj ↑delta_avg ↑oa ↑Δ aj vs fp32 originalvideosms / video
Original fp32fp3264.1777.0089.49 reference 30 –
Original fp16not measured
Portable fp32fp3264.1577.0289.46 −0.02 30 –
Portablefp1664.1477.0489.44 −0.03 30 –
Kernelsfp1664.1577.0489.44 −0.02 30 –

Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0019 and p99 <= 0.0099 (absolute error of the output tracks).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.000023 not recorded
GPU default dtypes, mean |diff|≤ 0.0019 0.00125
GPU default dtypes, p99 |diff|≤ 0.0099 0.00626

Recorded for alltracker on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13, with these kernels: fvm_conv

Details

AllTracker (Harley et al., ICCV 2025): tracking every pixel of a query frame through a video offline, as long-range optical flow. A ConvNeXt (tiny: a residual CNN) encodes every frame at 1/8; 16-frame windows (stride 8) refine the flow and visibility of all anchor pixels to all window frames at once, with a correlation pyramid, 2D ConvNeXt blocks over space and 1D blocks over time, then an 8x convex upsampling. Point queries read the dense result.

Input / output
Input: a video (its frames in order) and query points (t, x, y). Output: the position of every query in every frame and the probability that it is visible there.
Input size
any H x W >= 32, padded to a multiple of 32; tested up to 3000x4000 (5,626 MiB allocated)
Options
  • iters=4>= 1· changes the output
Outputs
tracks, visibility
CUDA kernels
  • fvm_convfused convolutions for image classifiers
Apple (MPS)
fp32, fp16

Limitations

  • Offline: the features of every frame are kept in memory (about 2.5 MB per frame in float16 at 384 x 512); each query frame costs a forward pass (and a backward pass from frame 0 if it is not the first).
  • Frames are resized to 384 x 512 (upstream's default evaluation size), whatever their size and aspect ratio. Upstream reports higher accuracy at larger sizes (TAP-Vid DAVIS AJ 65.9 at 448 x 768, 67.2 at 768 x 1024); the size is a card parameter.
  • Tracks are sampled bilinearly at the query position; upstream's evaluation rounds the query to the nearest pixel.
  • One video per call; all queries of a call share the video.
  • Kernels hold packed conv weights next to the originals the call reads (+31 MiB) and fragment the allocator: peak reserved is ~70 MiB above portable.
  • No exportable module: the track set is data-dependent (`torch.compile` runs with graph breaks; ONNX / TensorRT export fails).
Full reference, generated from the repository

AllTracker (Harley et al., ICCV 2025): track every pixel of a query frame through a whole video offline, as long-range optical flow over 16-frame windows; point queries read the dense result. MIT code and weights.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
alltrackersynthetic (480x854)original
portable
kernels
portable + torch.compile
TensorRT fp16
485
391
221
284
planned for 0.2.1
2100 / 2318
993 / 1238
1076 / 1288
1273 / 1358
–
alltracker-tinysynthetic (480x854)original
portable
kernels
portable + torch.compile
TensorRT fp16
424
330
217
229
planned for 0.2.1
1995 / 2200
919 / 1178
978 / 1166
1198 / 1338
–
Details
  • Original code: https://github.com/aharley/alltracker at e7553135e7b3
  • Code licence: MIT (permissive); files: LICENSE
  • Weights licence: MIT
  • Licence evidence: https://huggingface.co/aharley/alltracker
  • Training data: Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), FlyingChairs (Freiburg), Spring (scene flow / optical flow / stereo, University of Stuttgart), HCI HD1K optical flow benchmark, KITTI stereo / optical flow 2012 / 2015, TartanAir, not stated by the authors (Upstream README: stage 1 on Kubric MOVi-F, stage 2 on a mix of point tracking data (Kubric, PointOdyssey, Dynamic Replica and point tracks made from FlyingThings3D, Monkaa, Driving, SPRING) and optical flow data (FlyingChairs, FlyingThings3D, Monkaa, Driving, AutoFlow, SPRING, VIPER, HD1K, KITTI, TartanAir). Kubric, PointOdyssey, Dynamic Replica, AutoFlow and VIPER have no dataset entry here yet ("unstated").)
  • Options: iters = 4 (>= 1)
  • Kernel libraries: fvm_conv
  • Variants: alltracker, alltracker-tiny
  • Not in this release (planned for 0.2.1): alltracker ONNX / TensorRT export: torch.export fails (TorchExportError, step 1/3 of the ONNX export); alltracker-tiny ONNX / TensorRT export: torch.export fails (TorchExportError, step 1/3 of the ONNX export)
  • Outputs: tracks, visibility
  • Input size: any H x W >= 32, padded to a multiple of 32
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): alltracker 3000x4000 (5.5 GiB peak allocated, 5.9 reserved); alltracker-tiny 3000x4000 (5.5 GiB peak allocated, 5.9 reserved)
  • alltracker 480x854, first call: portable + torch.compile 105 s (cold Inductor cache).
  • alltracker-tiny 480x854, first call: portable + torch.compile 103 s (cold Inductor cache).
  • alltracker original 480x854: the authors' mixed-precision option (float16); aharley/alltracker@e7553135e7b3 Net.forward, 384 x 512, 4 iters
  • alltracker-tiny original 480x854: the authors' mixed-precision option (float16); aharley/alltracker@e7553135e7b3 Net.forward, 384 x 512, 4 iters
  • alltracker load / first call: original 1.68 s / 1.309 s, portable 0.89 s / 2.078 s, kernels 0.26 s / 0.493 s
  • alltracker-tiny load / first call: original 1.22 s / 1.398 s, portable 0.06 s / 0.731 s, kernels 0.96 s / 0.54 s
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portableportable fp32kernelsnote
alltrackerTAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA %67.45 / 78.66 / 90.8267.41 / 78.65 / 90.6967.42 / 78.71 / 90.7067.42 / 78.66 / 90.7967.38 / 78.63 / 90.65
alltrackerTAP-Vid DAVIS (query first), AJ / delta_avg / OA %64.17 / 77.00 / 89.4964.14 / 77.04 / 89.4464.15 / 77.02 / 89.4664.15 / 77.04 / 89.44
alltracker-tinyTAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA %65.33 / 76.69 / 88.1465.36 / 76.66 / 88.1465.38 / 76.69 / 88.1765.34 / 76.66 / 88.10

Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13.

Parity

CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0019 and p99 <= 0.0099 (absolute error of the output tracks).

variantkernels error vs original, mean / p99GPU, torchkernels used
alltracker0.00125 / 0.00626RTX 5060 Ti, torch 2.14.1+cu130fvm_conv@f1d064e539e1:sm_120
alltracker-tiny0.00119 / 0.00662RTX 5060 Ti, torch 2.14.1+cu130fvm_conv@f1d064e539e1:sm_120

Limitations

  • Offline: the features of every frame are kept in memory (about 2.5 MB per frame in float16 at 384 x 512); each query frame costs a forward pass (and a backward pass from frame 0 if it is not the first).
  • Frames are resized to 384 x 512 (upstream's default evaluation size), whatever their size and aspect ratio. Upstream reports higher accuracy at larger sizes (TAP-Vid DAVIS AJ 65.9 at 448 x 768, 67.2 at 768 x 1024); the size is a card parameter.
  • Tracks are sampled bilinearly at the query position; upstream's evaluation rounds the query to the nearest pixel.
  • One video per call; all queries of a call share the video.
  • Kernels hold packed conv weights next to the originals the call reads (+31 MiB) and fragment the allocator: peak reserved is ~70 MiB above portable.
  • No exportable module: the track set is data-dependent (torch.compile runs with graph breaks; ONNX / TensorRT export fails).

Example output

No example rendered for this checkpoint yet.