LocoTrack
LocoTrack (Cho et al., ECCV 2024): track any point through a whole video offline: a per-frame match from a cost volume, then refinement with local 4-D correlation and a transformer over the frames of each track. Apache-2.0 code and weights.
experimental Fork of KU-CVLAB/LocoTrack@3385a23 · backends: portable, kernels
- Paper Local All-Pair Correspondence for Point Tracking ECCV 2024 · first released 22 Jul 2024
Paper 2024 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install fastforks-core
from fastforks import tracking
model = tracking.load("locotrack-b") # kernels on a supported GPU, else portable PyTorch
result = model(video, queries)
result.tracks # also: visibility
result.backend # what ran: mode, device, dtype, kernels fastforks bench locotrack-b Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.
Input: synthetic, 480x854 (the task’s standard size)
| Backend | dtype | Time per call (ms) | vs original fp16 | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | not measured | – | – | – | – |
| Original fp16the authors' mixed-precision option (float16); KU-CVLAB/LocoTrack@3385a233a270 forward, offline (whole video), frames resized to 256 x 256 | fp16 | 203 | – | 719 MiB/784 MiB | 0.1 | 0.43 |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp16 | 188 | 1.1× | 603 MiB/656 MiB | 0.3 | 0.72 |
| Kernels | fp16 | 173 | 1.2× | 614 MiB/660 MiB | 0.2 | 0.87 |
| Portable + torch.compile | 82.6 | 2.5× | 1,117 MiB/1,140 MiB | 146 (build) | – | |
| TensorRT fp168 frames with 16 queries only | fp16 | 6.12 | 33× | 150 MiBenginenot comparable | – | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-13 to 2026-11-15. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14 | Original | fp32 | 1973 | 2,491 MiB process | – | 480x854 |
| Portable | fp16 | 2231 | 2,476 MiB process | matches | 480x854 | |
| Portable | fp32 | 2104 | 2,474 MiB process | matches | 480x854 | |
| Kernels | fp16 | not implemented | – | 480x854 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- Apache-2.0 LICENSE
- Weights
- Apache-2.0 source
- Training data
- Upstream README: trained on the panning Kubric MOVi-E sequences (synthetic, released as hamacojr/LocoTrack-panning-MOVi-E). No dataset entry here yet.
Our reading of the licences, not legal advice.
Weights
locotrack_base.ckpt 46.1 MB· hosted on Hugging Facefastforks downloads and checks these files on first use (fastforks download locotrack-b).
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
TAP-Vid DAVIS (query first)
every 3rd video, AJ / delta_avg / OA %
| Run | dtype | aj ↑ | delta_avg ↑ | oa ↑ | Δ aj vs fp32 original | videos | ms / video |
|---|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 67.19 | 78.05 | 89.58 | reference | 10 | – |
| Original fp16 | fp16 | 67.31 | 78.11 | 89.85 | +0.12 | 10 | – |
| Portable fp32 | fp32 | 67.23 | 78.11 | 89.76 | +0.04 | 10 | – |
| Portable | fp16 | 67.31 | 78.07 | 89.89 | +0.12 | 10 | – |
| Kernels | fp16 | 67.29 | 78.10 | 89.83 | +0.10 | 10 | – |
Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks db9e79d5, 2026-11-13.
TAP-Vid DAVIS
8 frames and 16 queries per clip, AJ / delta_avg / OA %
| Run | dtype | aj ↑ | delta_avg ↑ | oa ↑ | clips | ms / clip |
|---|---|---|---|---|---|---|
| Original fp32 | not measured | |||||
| Original fp16 | not measured | |||||
| Portable fp32 | not measured | |||||
| Portable | fp16 | 89.53 | 94.82 | 97.35 | 28 | – |
| Kernels | not measured | |||||
The original code was not run on this dataset, so the table cannot show the distance to it.
Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. The fixed export check uses 28 annotated eight-frame clips resized to 480 x 854, with 16 queries sampled from frames 0, 2, and 4. Scores are averaged over clips after scoring the frames following each query in TAP-Vid's 256 x 256 coordinate system. Measured on RTX 5060 Ti, torch 2.14.1+cu130, fastforks b3358de3, 2026-11-17.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0036 and p99 <= 0.021 (absolute error of the output tracks).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.000023 | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.0036 | 0.00172 |
| GPU default dtypes, p99 |diff| | ≤ 0.021 | 0.00969 |
Recorded for locotrack-b on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-13, with these kernels: fvm_conv
Details
LocoTrack (Cho et al., ECCV 2024): tracking any point offline over a whole video, TAPIR-like with local 4D correlation. A ResNet encodes every 256 x 256 frame at three resolutions; two cost volumes against the query's features give the initial position, visibility and uncertainty; then four refinements correlate the 7 x 7 neighbourhood of each track point with the 7 x 7 support of its query (a 4D correlation, encoded by small convs) and a transformer with ALiBi attention over the frames of each track corrects the whole track at once.
- Input / output
- Input: a video (its frames in order) and query points
(t, x, y). Output: the position of every query in every frame and the probability that it is visible there. - Input size
- any H x W >= 32, padded to a multiple of 32; tested up to 3000x4000 (5,583 MiB allocated)
- Outputs
tracks,visibility- CUDA kernels
fvm_convfused convolutions for image classifiers
- Apple (MPS)
- fp32, fp16
Limitations
- Offline: the whole video is encoded before tracking (features of every frame in memory, about 7 MB per frame in float32); for live video use an online tracker (TAPNext, Track-On2).
- Frames are resized to 256 x 256 (the training size and that of upstream's TAP-Vid numbers), whatever their size and aspect ratio. Upstream's higher-resolution refinement is not ported.
- One video per call; all queries of a call share the video.
Full reference, generated from the repository
LocoTrack (Cho et al., ECCV 2024): track any point through a whole video offline: a per-frame match from a cost volume, then refinement with local 4-D correlation and a transformer over the frames of each track. Apache-2.0 code and weights.
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
locotrack-b | synthetic (480x854) | original portable kernels portable + torch.compile TensorRT fp16 | 203 188 173 82.6 6.12 (8 frames with 16 queries only) | 719 / 784 603 / 656 614 / 660 1117 / 1140 233 / 242 |
locotrack-s | synthetic (480x854) | original portable kernels portable + torch.compile TensorRT fp16 | 193 180 151 74.1 5.66 (8 frames with 16 queries only) | 706 / 812 759 / 842 596 / 642 1000 / 1034 251 / 256 |
Details
- Original code: https://github.com/KU-CVLAB/LocoTrack at
3385a233a270 - Code licence: Apache-2.0 (permissive); files:
LICENSE - Weights licence: Apache-2.0
- Licence evidence: https://huggingface.co/datasets/hamacojr/LocoTrack-pytorch-weights/blob/a8754d0edfd9e53713dcb31ee21734aac8b727cf/README.md (license: apache-2.0); https://github.com/KU-CVLAB/LocoTrack/blob/3385a233a270160b253a69050c6a3934527f7775/LICENSE (Apache-2.0)
- Training data: Upstream README: trained on the panning Kubric MOVi-E sequences (synthetic, released as hamacojr/LocoTrack-panning-MOVi-E). No dataset entry here yet.
- Kernel libraries:
fvm_conv - Variants:
locotrack-b,locotrack-s - Outputs: tracks, visibility
- Input size: any H x W >= 32, padded to a multiple of 32
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
locotrack-b3000x4000 (5.5 GiB peak allocated, 5.9 reserved);locotrack-s3000x4000 (5.4 GiB peak allocated, 5.8 reserved) locotrack-b480x854, first call: portable + torch.compile 146 s (cold Inductor cache).locotrack-s480x854, first call: portable + torch.compile 140 s (cold Inductor cache), TensorRT fp16 51 s (ONNX + engine).locotrack-boriginal 480x854: the authors' mixed-precision option (float16); KU-CVLAB/LocoTrack@3385a233a270 forward, offline (whole video), frames resized to 256 x 256locotrack-soriginal 480x854: the authors' mixed-precision option (float16); KU-CVLAB/LocoTrack@3385a233a270 forward, offline (whole video), frames resized to 256 x 256locotrack-bload / first call: original 0.79 s / 1.058 s, portable 0.35 s / 0.718 s, kernels 0.18 s / 0.872 slocotrack-sload / first call: original 0.29 s / 0.229 s, portable 0.06 s / 0.653 s, kernels 1.13 s / 0.655 s- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
Query-first TAP-Vid protocol: frames resized to 256 x 256 (PIL Lanczos) before the model (AllTracker: the original frames, resized by the model), one query per track at its first visible point, tracked online (TAPNext, Track-On) or over the whole clip (TAPIR, AllTracker, LocoTrack); AJ, delta_avg (< 1, 2, 4, 8, 16 px at 256 x 256) and occlusion accuracy averaged over the videos. The fixed export check uses 28 annotated eight-frame clips resized to 480 x 854, with 16 queries sampled from frames 0, 2, and 4. Scores are averaged over clips after scoring the frames following each query in TAP-Vid's 256 x 256 coordinate system. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | portable fp32 | kernels | TensorRT | note |
|---|---|---|---|---|---|---|---|---|
locotrack-b | TAP-Vid DAVIS (query first), every 3rd video, AJ / delta_avg / OA % | 67.19 / 78.05 / 89.58 | 67.31 / 78.11 / 89.85 | 67.31 / 78.07 / 89.89 | 67.23 / 78.11 / 89.76 | 67.29 / 78.10 / 89.83 | ||
locotrack-b | TAP-Vid DAVIS, 8 frames and 16 queries per clip, AJ / delta_avg / OA % | 89.53 / 94.82 / 97.35 | 89.53 / 94.81 / 97.35 | |||||
locotrack-s | TAP-Vid DAVIS, 8 frames and 16 queries per clip, AJ / delta_avg / OA % | 89.00 / 94.41 / 97.48 | 89.07 / 94.41 / 97.48 |
Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks b3358de3 / db9e79d5, 2026-11-13 to 2026-11-17; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks bb4e8b64, 2026-11-17; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks db9e79d5, 2026-11-13; tensorrt: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks b3358de3, 2026-11-17.
Parity
CPU, float32: max |diff| <= 2.3e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0036 and p99 <= 0.021 (absolute error of the output tracks).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
locotrack-b | 0.00172 / 0.00969 | RTX 5060 Ti, torch 2.14.1+cu130 | fvm_conv@f1d064e539e1:sm_120 |
locotrack-s | 0.00243 / 0.01314 | RTX 5060 Ti, torch 2.14.1+cu130 | fvm_conv@f1d064e539e1:sm_120 |
Limitations
- Offline: the whole video is encoded before tracking (features of every frame in memory, about 7 MB per frame in float32); for live video use an online tracker (TAPNext, Track-On2).
- Frames are resized to 256 x 256 (the training size and that of upstream's TAP-Vid numbers), whatever their size and aspect ratio. Upstream's higher-resolution refinement is not ported.
- One video per call; all queries of a call share the video.
Example output
No example rendered for this checkpoint yet.