Models · Object detection

OWLv2 (open-vocabulary, CLIP B/16 and L/14)

Google's OWLv2 (CLIP ViT-B/16 and L/14): open-vocabulary detector, finds boxes for any class names given as text at call time (default: the 80 COCO classes). Apache-2.0 code and weights.

experimental Fork of huggingface/transformers@c4c4605 · backends: portable, kernels

Checkpoint

Paper 2023 Added in 0.2.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 23.7 ms 800x1344, fp16
vs the faster original 3.9× original fp16: 92.3 ms
Peak memory 461 MiB original: 921 MiB
Accuracy vs original fp32 ±0.00 kernels box mAP, COCO val2017 (46.14 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install "fastforks-core[transformers]"

from fastforks import detection

model = detection.load("owlv2-b16")  # kernels on a supported GPU, else portable PyTorch
result = model(image)
result.boxes  # also: scores, labels
result.backend  # what ran: mode, device, dtype, kernels
fastforks bench owlv2-b16

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

Input: synthetic, 800x1344 (the task’s standard size)

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32fp32not measured ––––
Original fp16torch.autocast(float16) around the call; transformers Owlv2ForObjectDetection + processor (CPU), coco-80 queries, text tower per call, 300 bestfp16 92.3–921 MiB/960 MiB3.00.21
Portable fp32fp32not measured ––––
Portablefp16 31.82.9×460 MiB/510 MiB3.40.26
Kernelsfp16 23.73.9×461 MiB/612 MiB3.30.24
Portable + torch.compile 28.53.2×487 MiB/530 MiB11 (build)–
TensorRT fp16fp16 18.45.0×256 MiBenginenot comparable26 (build)–

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-12 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-13Originalfp32468 2,079 MiB process–800x1344
Portablefp32394 1,515 MiB processmatches800x1344
Kernelsfp16not implemented–800x1344

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Commercial use allowed

Code
Apache-2.0
Weights
Apache-2.0 source
Training data

Our reading of the licences, not legal advice.

Weights

owlv2-b16.config.json 414 B· hosted on Hugging Face
owlv2-b16.safetensors 620 MB· hosted on Hugging Face
clip-bpe.vocab.json 1.1 MB· hosted on Hugging Face
clip-bpe.merges.txt 525 KB· hosted on Hugging Face

fastforks downloads and checks these files on first use (fastforks download owlv2-b16).

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

COCO val2017

every 10th image, box mAP

Rundtypebox mAP ↑Δ box mAP vs fp32 originalimagesms / image
Original fp32fp3246.14 reference 500 –
Original fp16fp1646.04 −0.10 500 –
Portable fp32fp3246.14 ±0.00 500 135
Portablefp1646.11 −0.03 500 –
Kernelsfp1646.14 ±0.00 500 –

pycocotools COCOeval of the top 100 detections per image. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12; portable fp32: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 44640884, 2026-11-17; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks a981ff86, 2026-11-12.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 and p99 <= 0.39 (absolute error of the output boxes).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.002 not recorded
GPU default dtypes, mean |diff|≤ 0.15 0.09926
GPU default dtypes, p99 |diff|≤ 0.39 0.22711

Recorded for owlv2-b16 on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-12, with these kernels: fvm_conv

Details

OWLv2 (Minderer et al. 2023, Google): open-vocabulary detection by self-training OWL-ViT on web image-text data; a CLIP ViT image tower predicts one box per patch and scores it against CLIP text embeddings of free-text class names. Pass `text=[...]` at call time (default: the 80 COCO class names); the image is padded to a square and resized to 960 (B/16) or 1008 (L/14) px. The portable `module()` is the image tower and heads with the COCO names as constant queries.

Input / output
Input: one RGB image. Output: boxes, class labels, scores and instance masks.
Input size
any H x W >= 32, padded to a multiple of 32; tested up to 3000x4000 (1,083 MiB allocated)
Options
  • score_threshold=0.10.0 to 1.0· changes the output
  • max_detections=100>= 1· changes the output
  • cuda_graphs=true
Outputs
boxes, scores, labels
CUDA kernels
  • fvm_convfused convolutions for image classifiers
Apple (MPS)
fp32

Limitations

  • Requires the optional transformers library (the network is its `Owlv2ForObjectDetection`).
  • The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (google/owlv2-*-ensemble); the official scenic (JAX) code was not run.
  • Text queries are used as given (no prompt templates or prompt ensembling as in the paper's COCO / LVIS numbers); each is cut to 16 tokens as the transformers processor does.
  • No objectness scores and no image-guided (one-shot) detection.
  • One box per patch token (3600 for B/16, 5184 for L/14) without NMS, as the original post-processing; overlapping boxes of the same object are common at low thresholds.
  • Kernels keep CUDA graphs: their private pool and static outputs stay reserved, so peak reserved is ~120 MiB above portable (peak allocated is not); `cuda_graphs=false` or `release_graphs()` frees it.
Full reference, generated from the repository

Google's OWLv2 (CLIP ViT-B/16 and L/14): open-vocabulary detector, finds boxes for any class names given as text at call time (default: the 80 COCO classes). Apache-2.0 code and weights.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
owlv2-b16synthetic (800x1344)original
portable
kernels
portable + torch.compile
TensorRT fp16
92.3
31.8
23.7
28.5
18.4
921 / 960
460 / 510
461 / 612
487 / 530
655 / 726
owlv2-l14synthetic (800x1344)original
portable
kernels
portable + torch.compile
TensorRT fp16
212
154
110
141
95.9
2550 / 2606
1097 / 1144
1027 / 1184
1190 / 1242
1786 / 1820
Details
  • Original code: https://github.com/huggingface/transformers at c4c4605f9618
  • Code licence: Apache-2.0 (permissive); files:
  • Weights licence: Apache-2.0
  • Licence evidence: https://huggingface.co/google/owlv2-base-patch16-ensemble/blob/cfd3195ba4ea9592eec887ded089f4c08eff231d/README.md
  • Training data: WebLI (Google web image-text data, not released), Objects365 (train, box annotations) (Self-training on WebLI image-text pairs with pseudo-boxes from an OWL-ViT annotator (itself trained on Objects365 and Visual Genome), then the weight ensemble of that model and its fine-tune on Objects365 + Visual Genome; Visual Genome has no dataset id here (OWLv2 paper, arXiv:2306.09683; the ensemble checkpoints).)
  • Options: score_threshold = 0.1 (0.0 to 1.0), max_detections = 100 (>= 1), cuda_graphs = True (speed only)
  • Kernel libraries: fvm_conv
  • Variants: owlv2-b16, owlv2-l14
  • Outputs: boxes, scores, labels; label_set: coco-80
  • Input size: any H x W >= 32, padded to a multiple of 32
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): owlv2-b16 3000x4000 (1.1 GiB peak allocated, 1.6 reserved); owlv2-l14 3000x4000 (1.7 GiB peak allocated, 2.3 reserved)
  • owlv2-b16 800x1344, first call: portable + torch.compile 11 s (cold Inductor cache), TensorRT fp16 26 s (ONNX + engine).
  • owlv2-l14 800x1344, first call: portable + torch.compile 12 s (cold Inductor cache), TensorRT fp16 34 s (ONNX + engine).
  • owlv2-b16 original 800x1344: torch.autocast(float16) around the call; transformers Owlv2ForObjectDetection + processor (CPU), coco-80 queries, text tower per call, 300 best
  • owlv2-l14 original 800x1344: torch.autocast(float16) around the call; transformers Owlv2ForObjectDetection + processor (CPU), coco-80 queries, text tower per call, 300 best
  • owlv2-b16 load / first call: original 8.22 s / 0.744 s, portable 3.41 s / 0.256 s, kernels 3.32 s / 0.244 s
  • owlv2-l14 load / first call: original 9.55 s / 1.182 s, portable 4.44 s / 0.388 s, kernels 4.1 s / 0.35 s
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

pycocotools COCOeval of the top 100 detections per image. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portableportable fp32kernelsnote
owlv2-b16COCO val2017, every 10th image, box mAP46.1446.0446.1146.1446.14
owlv2-l14COCO val2017, every 10th image, box mAP47.7547.8047.8147.7547.88

Measured on portable: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks a981ff86, 2026-11-12; portable32: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 44640884, 2026-11-17; kernels: NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks a981ff86, 2026-11-12.

Parity

CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 and p99 <= 0.39 (absolute error of the output boxes).

variantkernels error vs original, mean / p99GPU, torchkernels used
owlv2-b160.09926 / 0.22711RTX 5060 Ti, torch 2.14.1+cu130fvm_conv@f1d064e539e1:sm_120
owlv2-l140.09665 / 0.25782RTX 5060 Ti, torch 2.14.1+cu130fvm_conv@f1d064e539e1:sm_120

Limitations

  • Requires the optional transformers library (the network is its Owlv2ForObjectDetection).
  • The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (google/owlv2-*-ensemble); the official scenic (JAX) code was not run.
  • Text queries are used as given (no prompt templates or prompt ensembling as in the paper's COCO / LVIS numbers); each is cut to 16 tokens as the transformers processor does.
  • No objectness scores and no image-guided (one-shot) detection.
  • One box per patch token (3600 for B/16, 5184 for L/14) without NMS, as the original post-processing; overlapping boxes of the same object are common at low thresholds.
  • Kernels keep CUDA graphs: their private pool and static outputs stay reserved, so peak reserved is ~120 MiB above portable (peak allocated is not); cuda_graphs=false or release_graphs() frees it.

Example output

No example rendered for this checkpoint yet.