TIPSv2
Google DeepMind's TIPSv2 image encoders (B, L): image features aligned with text, with patch features meant for dense tasks. Apache-2.0.
experimental Fork of google-deepmind/tips@bf10a73 · backends: portable, kernels
- Paper TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment CVPR 2026 · first released 13 Apr 2026
- Paper TIPS: Text-Image Pretraining with Spatial awareness ICLR 2025 · first released 21 Oct 2024
Paper 2026 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install fastforks-core
from fastforks import features
model = features.load("tipsv2-b14") # kernels on a supported GPU, else portable PyTorch
result = model(image)
result.last_hidden_state # also: pooler_output, hidden_states
result.backend # what ran: mode, device, dtype, kernels fastforks bench tipsv2-b14 Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.
Input: synthetic, 224x224 (the task’s standard size)
| Backend | dtype | Time per call (ms) | vs original fp32 | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32tips image_encoder first cls token | fp32 | 7.03 | – | 373 MiB/386 MiB | 1.0 | 0.27 |
| Original fp16tips image_encoder first cls token | fp16 | 7.21 | 0.98× | 529 MiB/548 MiB | 0.9 | 0.07 |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp16 | 2.93 | 2.4× | 204 MiB/224 MiB | 0.2 | 0.18 |
| Kernels | fp16 | 1.49 | 4.7× | 206 MiB/226 MiB | 1.3 | 0.11 |
| Portable + torch.compile | 2.18 | 3.2× | 239 MiB/266 MiB | 9 (build) | – | |
| TensorRT fp16 | fp16 | 1.33 | 5.3× | 198 MiBenginenot comparable | 42 (build) | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-10 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-13 | Original | fp32 | 31.7 | 930 MiB process | – | 224x224 |
| Portable | fp16 | 23.1 | 775 MiB process | matches | 224x224 | |
| Portable | fp32 | 24.6 | 895 MiB process | matches | 224x224 | |
| Kernels | fp16 | not implemented | – | 224x224 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- Apache-2.0 LICENSE
- Weights
- Apache-2.0 source
- Training data
- image-text pairs; the dataset is not named in the repository README or on the model cards at the pinned revisions
Our reading of the licences, not legal advice.
Weights
tipsv2-b14/model.safetensors 784 MB· hosted on Hugging Facefastforks downloads and checks these files on first use (fastforks download tipsv2-b14).
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
ImageNet-1k val
kNN top-1 % (k = 20, 10000 labelled / 5000 test images)
| Run | dtype | top-1 % ↑ | Δ top-1 % vs fp32 original | images | ms / image |
|---|---|---|---|---|---|
| Original fp32 | fp32 | 59.34 | reference | 5000 | 5.20 |
| Original fp16 | fp16 | 59.30 | −0.04 | 5000 | 2.12 |
| Portable fp32 | not measured | ||||
| Portable | fp16 | 59.36 | +0.02 | 5000 | 1.36 |
| Kernels | fp16 | 59.16 | −0.18 | 5000 | 0.968 |
Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-10; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8b2f2df5, 2026-11-10.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 6.9e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 1.9 % and p99 <= 6.9 % of the reference's median |value| (absolute error of the output last_hidden_state).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.000069 (relative) | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.019 (relative) | 0.01252 (relative) |
| GPU default dtypes, p99 |diff| | ≤ 0.069 (relative) | 0.046 (relative) |
Recorded for tipsv2-b14 on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12, with these kernels: dpt_fusedfastbb
Details
TIPSv2 (Google DeepMind, CVPR 2026) image encoders: ViT-B / L at patch 14 from text-image pretraining with patch-text alignment; dense patch tokens and two global tokens at any size (native 448 px).
- Input / output
- Input: one RGB image. Output: dense patch features and a pooled image feature.
- Input size
- any H x W >= 1, resized or padded inside the model; tested up to 3000x4000 (1,467 MiB allocated)
- Options
fp16_accumulate=false
- Outputs
last_hidden_state,pooler_output,hidden_states- Checkpoint
arch="vitb"- CUDA kernels
fastbbfast backbone kernels: flash attention, window attention, instance norm and rotary embeddings for ViT and similar backbonesdpt_fusedfused fp16 matrix multiplies and convolutions for DINOv2 backbones and DPT depth heads
- Apple (MPS)
- fp32, fp16
Limitations
- Image encoder only: the text encoder in the checkpoint is not loaded (the features task has no text input), although the file is downloaded whole.
- last_hidden_state holds 2 prefix tokens, then the patches: upstream's first cls token (pooler_output, the global image embedding aligned with text) and its second cls token (index 1, the spatially aware global embedding).
- kernels replays a CUDA graph per small input shape without hidden states (up to about 520k activations: ViT-B ~360 px, ViT-L ~310 px), captured on the shape's second call, at most 4 per model; `model.release_graphs()` frees them.
Full reference, generated from the repository
Google DeepMind's TIPSv2 image encoders (B, L): image features aligned with text, with patch features meant for dense tasks. Apache-2.0.
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
tipsv2-b14 | synthetic (224x224) | original (fp32) original portable kernels portable + torch.compile TensorRT fp16 | 7.03 7.21 2.93 1.49 2.18 1.33 | 373 / 386 529 / 548 204 / 224 206 / 226 239 / 266 398 / 422 |
tipsv2-l14 | synthetic (224x224) | original (fp32) original portable kernels portable + torch.compile TensorRT fp16 | 21.9 14.2 6.15 4.58 6.04 4.28 | 1207 / 1226 1776 / 1786 621 / 624 624 / 648 659 / 666 1236 / 1264 |
Details
- Original code: https://github.com/google-deepmind/tips at
bf10a73 - Code licence: Apache-2.0 (permissive); files:
LICENSE - Weights licence: Apache-2.0
- Licence evidence: https://huggingface.co/google/tipsv2-b14/blob/ed1e4dc6b74bf3935ae099e9d5eb30fa96528454/README.md (license: apache-2.0); code: https://github.com/google-deepmind/tips/blob/bf10a73765b048edfbfcdf1f7e6ca8292542baae/LICENSE
- Training data: image-text pairs; the dataset is not named in the repository README or on the model cards at the pinned revisions
- Options:
fp16_accumulate= False (precision only) - Kernel libraries:
fastbb,dpt_fused - Variants:
tipsv2-b14,tipsv2-l14 - Outputs: last_hidden_state, pooler_output, hidden_states
- Input size: any H x W >= 1, resized or padded inside the model
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
tipsv2-b143000x4000 (1.4 GiB peak allocated, 2.2 reserved);tipsv2-l143000x4000 (2.2 GiB peak allocated, 2.2 reserved) tipsv2-b14224x224, first call: portable + torch.compile 9 s (cold Inductor cache), TensorRT fp16 42 s (ONNX + engine).tipsv2-l14224x224, first call: portable + torch.compile 12 s (cold Inductor cache), TensorRT fp16 48 s (ONNX + engine).tipsv2-b14original (fp32) 224x224, original 224x224: tips image_encoder first cls tokentipsv2-l14original (fp32) 224x224, original 224x224: tips image_encoder first cls tokentipsv2-b14load / first call: original 0.9 s / 0.066 s, portable 0.2 s / 0.182 s, kernels 1.42 s / 0.085 stipsv2-l14load / first call: original 3.1 s / 0.019 s, portable 2.91 s / 0.083 s, kernels 1.44 s / 0.108 s- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | kernels | note |
|---|---|---|---|---|---|---|
tipsv2-b14 | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 59.34 | 59.30 | 59.36 | 59.16 | |
tipsv2-l14 | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 63.48 | 63.44 | 63.42 | 63.42 |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8b2f2df5, 2026-11-10.
Parity
CPU, float32: max |diff| <= 6.9e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 1.9 % and p99 <= 6.9 % of the reference's median |value| (absolute error of the output last_hidden_state).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
tipsv2-b14 | 0.01252 / 0.046 | RTX 5060 Ti, torch 2.14.1+cu130 | dpt_fused@48e0065bb7cf:sm_120, fastbb@7d058322634d:sm_120 |
tipsv2-l14 | 0.00988 / 0.03882 | RTX 5060 Ti, torch 2.14.1+cu130 | dpt_fused@48e0065bb7cf:sm_120, fastbb@7d058322634d:sm_120 |
Limitations
- Image encoder only: the text encoder in the checkpoint is not loaded (the features task has no text input), although the file is downloaded whole.
- last_hidden_state holds 2 prefix tokens, then the patches: upstream's first cls token (pooler_output, the global image embedding aligned with text) and its second cls token (index 1, the spatially aware global embedding).
- kernels replays a CUDA graph per small input shape without hidden states (up to about 520k activations: ViT-B ~360 px, ViT-L ~310 px), captured on the shape's second call, at most 4 per model;
model.release_graphs()frees them.
Example output
One fixed input through this checkpoint, the same for every model of the task (kernels · float16 · NVIDIA GeForce RTX 5060 Ti).
Input: Chelsea the cat, Stéfan van der Walt (CC0).