AllTracker
AllTracker (Harley et al., ICCV 2025): track every pixel of a query frame through a whole video offline, as long-range optical flow over 16-frame windows; point queries read the dense result. MIT code and weights.
experimental Fork of aharley/alltracker@e755313 · backends: portable, kernels
- Paper AllTracker: Efficient Dense Point Tracking at High Resolution ICCV 2025 · first released 8 Jun 2025
Paper 2025 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install fastforks-core
from fastforks import tracking
model = tracking.load("alltracker") # kernels on a supported GPU, else portable PyTorch
result = model(video, queries)
result.tracks # also: visibility
result.backend # what ran: mode, device, dtype, kernels fastforks bench alltracker Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.
Input: synthetic, 480x854 (the task’s standard size)
| Backend | dtype | Time per call (ms) | vs original fp16 | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | not measured | – | – | – | – |
| Original fp16the authors' mixed-precision option (float16); aharley/alltracker@e7553135e7b3 Net.forward, 384 x 512, 4 iters | fp16 | 485 | – | 2,100 MiB/2,318 MiB | 0.5 | 0.83 |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp16 | 391 | 1.2× | 993 MiB/1,238 MiB | 0.9 | 2.08 |
| Kernels | fp16 | 221 | 2.2× | 1,076 MiB/1,288 MiB | 0.3 | 0.49 |
| Portable + torch.compile | 284 | 1.7× | 1,273 MiB/1,358 MiB | 105 (build) | – | |
| TensorRTplanned for 0.2.1 | fails | – | – | – | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-13 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14 | Original | fp32 | 3628 | 6,081 MiB process | – | 480x854 |
| Portable | fp16 | 3200 | 3,806 MiB process | matches | 480x854 | |
| Portable | fp32 | 3508 | 4,571 MiB process | matches | 480x854 | |
| Kernels | fp16 | not implemented | – | 480x854 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- MIT LICENSE
- Weights
- MIT source
- Training data
-
- Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa) research-only, non-commercial
- FlyingChairs (Freiburg) research-only, non-commercial
- Spring (scene flow / optical flow / stereo, University of Stuttgart) CC-BY-4.0
- HCI HD1K optical flow benchmark not stated
- KITTI stereo / optical flow 2012 / 2015 CC-BY-NC-SA-3.0, non-commercial
- TartanAir CC-BY-4.0
Our reading of the licences, not legal advice.
Weights
alltracker.pth 66.0 MB· hosted on Hugging Facefastforks downloads and checks these files on first use (fastforks download alltracker).
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
TAP-Vid DAVIS (query first)
every 3rd video, AJ / delta_avg / OA %
| Run | dtype | aj ↑ | delta_avg ↑ | oa ↑ | Δ aj vs fp32 original | videos | ms / video |
|---|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 67.45 | 78.66 | 90.82 | reference | 10 | – |
| Original fp16 | fp16 | 67.41 | 78.65 | 90.69 | −0.04 | 10 | – |
| Portable fp32 | fp32 | 67.42 | 78.66 | 90.79 | −0.03 | 10 | – |
| Portable | fp16 | 67.42 | 78.71 | 90.70 | −0.03 | 10 | – |
| Kernels | fp16 | 67.38 | 78.63 | 90.65 | −0.07 | 10 | – |
Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13.
TAP-Vid DAVIS (query first)
AJ / delta_avg / OA %
| Run | dtype | aj ↑ | delta_avg ↑ | oa ↑ | Δ aj vs fp32 original | videos | ms / video |
|---|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 64.17 | 77.00 | 89.49 | reference | 30 | – |
| Original fp16 | not measured | ||||||
| Portable fp32 | fp32 | 64.15 | 77.02 | 89.46 | −0.02 | 30 | – |
| Portable | fp16 | 64.14 | 77.04 | 89.44 | −0.03 | 30 | – |
| Kernels | fp16 | 64.15 | 77.04 | 89.44 | −0.02 | 30 | – |
Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0019 and p99 <= 0.0099 (absolute error of the output tracks).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.000023 | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.0019 | 0.00125 |
| GPU default dtypes, p99 |diff| | ≤ 0.0099 | 0.00626 |
Recorded for alltracker on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13, with these kernels: fvm_conv
Details
AllTracker (Harley et al., ICCV 2025): tracking every pixel of a query frame through a video offline, as long-range optical flow. A ConvNeXt (tiny: a residual CNN) encodes every frame at 1/8; 16-frame windows (stride 8) refine the flow and visibility of all anchor pixels to all window frames at once, with a correlation pyramid, 2D ConvNeXt blocks over space and 1D blocks over time, then an 8x convex upsampling. Point queries read the dense result.
- Input / output
- Input: a video (its frames in order) and query points
(t, x, y). Output: the position of every query in every frame and the probability that it is visible there. - Input size
- any H x W >= 32, padded to a multiple of 32; tested up to 3000x4000 (5,626 MiB allocated)
- Options
iters=4>= 1· changes the output
- Outputs
tracks,visibility- CUDA kernels
fvm_convfused convolutions for image classifiers
- Apple (MPS)
- fp32, fp16
Limitations
- Offline: the features of every frame are kept in memory (about 2.5 MB per frame in float16 at 384 x 512); each query frame costs a forward pass (and a backward pass from frame 0 if it is not the first).
- Frames are resized to 384 x 512 (upstream's default evaluation size), whatever their size and aspect ratio. Upstream reports higher accuracy at larger sizes (TAP-Vid DAVIS AJ 65.9 at 448 x 768, 67.2 at 768 x 1024); the size is a card parameter.
- Tracks are sampled bilinearly at the query position; upstream's evaluation rounds the query to the nearest pixel.
- One video per call; all queries of a call share the video.
- Kernels hold packed conv weights next to the originals the call reads (+31 MiB) and fragment the allocator: peak reserved is ~70 MiB above portable.
- No exportable module: the track set is data-dependent (`torch.compile` runs with graph breaks; ONNX / TensorRT export fails).
Full reference, generated from the repository
AllTracker (Harley et al., ICCV 2025): track every pixel of a query frame through a whole video offline, as long-range optical flow over 16-frame windows; point queries read the dense result. MIT code and weights.
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
alltracker | synthetic (480x854) | original portable kernels portable + torch.compile TensorRT fp16 | 485 391 221 284 planned for 0.2.1 | 2100 / 2318 993 / 1238 1076 / 1288 1273 / 1358 – |
alltracker-tiny | synthetic (480x854) | original portable kernels portable + torch.compile TensorRT fp16 | 424 330 217 229 planned for 0.2.1 | 1995 / 2200 919 / 1178 978 / 1166 1198 / 1338 – |
Details
- Original code: https://github.com/aharley/alltracker at
e7553135e7b3 - Code licence: MIT (permissive); files:
LICENSE - Weights licence: MIT
- Licence evidence: https://huggingface.co/aharley/alltracker
- Training data: Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), FlyingChairs (Freiburg), Spring (scene flow / optical flow / stereo, University of Stuttgart), HCI HD1K optical flow benchmark, KITTI stereo / optical flow 2012 / 2015, TartanAir, not stated by the authors (Upstream README: stage 1 on Kubric MOVi-F, stage 2 on a mix of point tracking data (Kubric, PointOdyssey, Dynamic Replica and point tracks made from FlyingThings3D, Monkaa, Driving, SPRING) and optical flow data (FlyingChairs, FlyingThings3D, Monkaa, Driving, AutoFlow, SPRING, VIPER, HD1K, KITTI, TartanAir). Kubric, PointOdyssey, Dynamic Replica, AutoFlow and VIPER have no dataset entry here yet ("unstated").)
- Options:
iters= 4 (>= 1) - Kernel libraries:
fvm_conv - Variants:
alltracker,alltracker-tiny - Not in this release (planned for 0.2.1):
alltrackerONNX / TensorRT export: torch.export fails (TorchExportError, step 1/3 of the ONNX export);alltracker-tinyONNX / TensorRT export: torch.export fails (TorchExportError, step 1/3 of the ONNX export) - Outputs: tracks, visibility
- Input size: any H x W >= 32, padded to a multiple of 32
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
alltracker3000x4000 (5.5 GiB peak allocated, 5.9 reserved);alltracker-tiny3000x4000 (5.5 GiB peak allocated, 5.9 reserved) alltracker480x854, first call: portable + torch.compile 105 s (cold Inductor cache).alltracker-tiny480x854, first call: portable + torch.compile 103 s (cold Inductor cache).alltrackeroriginal 480x854: the authors' mixed-precision option (float16); aharley/alltracker@e7553135e7b3 Net.forward, 384 x 512, 4 itersalltracker-tinyoriginal 480x854: the authors' mixed-precision option (float16); aharley/alltracker@e7553135e7b3 Net.forward, 384 x 512, 4 itersalltrackerload / first call: original 1.68 s / 1.309 s, portable 0.89 s / 2.078 s, kernels 0.26 s / 0.493 salltracker-tinyload / first call: original 1.22 s / 1.398 s, portable 0.06 s / 0.731 s, kernels 0.96 s / 0.54 s- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | portable fp32 | kernels | note |
|---|---|---|---|---|---|---|---|
alltracker | TAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA % | 67.45 / 78.66 / 90.82 | 67.41 / 78.65 / 90.69 | 67.42 / 78.71 / 90.70 | 67.42 / 78.66 / 90.79 | 67.38 / 78.63 / 90.65 | |
alltracker | TAP-Vid DAVIS (query first), AJ / delta_avg / OA % | 64.17 / 77.00 / 89.49 | 64.14 / 77.04 / 89.44 | 64.15 / 77.02 / 89.46 | 64.15 / 77.04 / 89.44 | ||
alltracker-tiny | TAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA % | 65.33 / 76.69 / 88.14 | 65.36 / 76.66 / 88.14 | 65.38 / 76.69 / 88.17 | 65.34 / 76.66 / 88.10 |
Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8d5bccf, 2026-11-13.
Parity
CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0019 and p99 <= 0.0099 (absolute error of the output tracks).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
alltracker | 0.00125 / 0.00626 | RTX 5060 Ti, torch 2.14.1+cu130 | fvm_conv@f1d064e539e1:sm_120 |
alltracker-tiny | 0.00119 / 0.00662 | RTX 5060 Ti, torch 2.14.1+cu130 | fvm_conv@f1d064e539e1:sm_120 |
Limitations
- Offline: the features of every frame are kept in memory (about 2.5 MB per frame in float16 at 384 x 512); each query frame costs a forward pass (and a backward pass from frame 0 if it is not the first).
- Frames are resized to 384 x 512 (upstream's default evaluation size), whatever their size and aspect ratio. Upstream reports higher accuracy at larger sizes (TAP-Vid DAVIS AJ 65.9 at 448 x 768, 67.2 at 768 x 1024); the size is a card parameter.
- Tracks are sampled bilinearly at the query position; upstream's evaluation rounds the query to the nearest pixel.
- One video per call; all queries of a call share the video.
- Kernels hold packed conv weights next to the originals the call reads (+31 MiB) and fragment the allocator: peak reserved is ~70 MiB above portable.
- No exportable module: the track set is data-dependent (
torch.compileruns with graph breaks; ONNX / TensorRT export fails).
Example output
No example rendered for this checkpoint yet.