Models · Promptable segmentation

CLIPSeg (text-prompted segmentation, CLIP B/16)

CLIPSeg (Lüddecke and Ecker, CVPR 2022): one coarse mask for whatever a free-text prompt names, from a frozen CLIP ViT-B/16 and a small decoder. Apache-2.0 code and weights; the PhraseCut training data has no stated terms.

experimental Fork of huggingface/transformers@c4c4605 · backends: portable, kernels

  • Paper Image Segmentation Using Text and Image Prompts CVPR 2022 · first released 18 Dec 2021
  • Authors’ code timojl/clipseg (PyTorch), not the code forked here: fastforks follows the transformers port, which loads the authors' converted weights (CIDAS/clipseg-rd64-refined) without the repo's CLIP dependency.

Checkpoint clipseg

Paper 2021 Added in 0.2.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 2.76 ms 1024x1024, fp16
vs the faster original 5.6× original fp16: 15.3 ms
Peak memory 335 MiB original: 346 MiB
Accuracy vs original fp32 – original not run on the test set

Use it

pip install fastforks

Lean install with only what this model needs: pip install "fastforks-core[transformers]"

from fastforks import promptable_segmentation

model = promptable_segmentation.load("clipseg")  # kernels on a supported GPU, else portable PyTorch
result = model(image, input_points=...)
result.boxes  # also: scores, masks
result.backend  # what ran: mode, device, dtype, kernels
fastforks bench clipseg

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32fp32not measured ––––
Original fp16.half() on 1 modules, fp16 inputs; transformers CLIPSegProcessor (CPU) + CLIPSegForImageSegmentation, text tower per call, masks at logits > 0fp16 15.3–346 MiB/404 MiB0.30.02
Portable fp32fp32not measured ––––
Portablefp16 4.463.4×328 MiB/356 MiB4.90.29
Kernelsfp16 2.765.6×335 MiB/362 MiB4.50.24
Portable + torch.compile 3.194.8×348 MiB/374 MiB10 (build)–
TensorRT fp16segmentation network on a normalized 352x352 image and precomputed text embedding onlyfp16 1.758.8×170 MiBenginenot comparable40 (build)–

Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-14 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14Originalfp3267.6 1,606 MiB process–1024x1024
Portablefp1637.7 1,561 MiB processmatches1024x1024
Portablefp3244.7 1,532 MiB processmatches1024x1024
Kernelsfp16not implemented–1024x1024

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Commercial use allowed

Code
Apache-2.0
Weights
Apache-2.0 source
Training data

Our reading of the licences, not legal advice.

Weights

clipseg-rd64-refined.config.json 4.7 KB· hosted on Hugging Face
clipseg-rd64-refined.safetensors 603 MB· hosted on Hugging Face
clip-bpe.vocab.json 1.1 MB· hosted on Hugging Face
clip-bpe.merges.txt 525 KB· hosted on Hugging Face

fastforks downloads and checks these files on first use (fastforks download clipseg).

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

COCO val2017

every 10th image, class masks, class / prompt mIoU %

Rundtypemiou_class ↑miou_prompt ↑imagesms / image
Original fp32not measured
Original fp16not measured
Portable fp32not measured
Portablefp1655.0038.37 500 –
Kernelsnot measured

The original code was not run on this dataset, so the table cannot show the distance to it.

One text prompt per present COCO class. The label is the union of the class's non-crowd masks. Scores average pixel IoU aggregated per class, and IoU per image and prompt. Measured on RTX 5060 Ti, torch 2.14.1+cu130, fastforks bc6ebb31, 2026-11-17.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.0005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.1 and p99 <= 0.5 (absolute error of the output boxes).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.0005 not recorded
GPU default dtypes, mean |diff|≤ 0.1 0.01273
GPU default dtypes, p99 |diff|≤ 0.5 0.081

Recorded for clipseg on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: fvm_conv

Details

CLIPSeg (Lüddecke and Ecker 2022): a frozen CLIP ViT-B/16 with a small transformer decoder that segments whatever a free-text prompt names; the text embedding conditions the decoder by FiLM. Semantic: one mask per prompt (all pixels of the concept), at 352 x 352, resized to the image. Pass `text=` at call time. The portable `module()` is the network on the normalised 352 x 352 image plus a projected text embedding.

Input / output
Input: one image plus a point, box or text prompt. Output: masks with quality scores.
Input size
any H x W >= 1, resized inside to the fixed encoder input 352x352 (the default export size); tested up to 3000x4000 (748 MiB allocated)
Options
  • cuda_graphs=true
Outputs
boxes, scores, masks
CUDA kernels
  • fvm_convfused convolutions for image classifiers
Apple (MPS)
fp32, fp16

Limitations

  • No accuracy measurement: CLIPSeg is scored on PhraseCut and Pascal-5i one-shot splits, which are not part of the evaluation data here; tiers are compared by output parity and speed only.
  • Requires the optional transformers library (the network is its `CLIPSegForImageSegmentation`).
  • The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (CIDAS/clipseg-rd64-refined); the authors' timojl/clipseg code was not run.
  • Text prompts only (no box exemplars or visual prompts); one instance per prompt: its mask is every pixel above probability 0.5, its score the highest pixel probability, its box the mask's bounding box.
  • The image is resized to 352 x 352 without keeping the aspect ratio, as the original processor; masks are coarse (22 x 22 patch tokens).
Full reference, generated from the repository

CLIPSeg (Lüddecke and Ecker, CVPR 2022): one coarse mask for whatever a free-text prompt names, from a frozen CLIP ViT-B/16 and a small decoder. Apache-2.0 code and weights; the PhraseCut training data has no stated terms.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
clipsegsynthetic (1024x1024)original
portable
kernels
portable + torch.compile
TensorRT fp16
15.3
4.46
2.76
3.19
1.75 (segmentation network on a normalized 352x352 image and precomputed text embedding only)
346 / 404
328 / 356
335 / 362
348 / 374
464 / 480
Details
  • Original code: https://github.com/huggingface/transformers at c4c4605f9618
  • Code licence: Apache-2.0 (permissive); files:
  • Weights licence: Apache-2.0
  • Licence evidence: https://huggingface.co/CIDAS/clipseg-rd64-refined/blob/999e0328d9e10b484360c477313983f9afdd7050/README.md
  • Training data: PhraseCut (referring-phrase masks on Visual Genome images) (The decoder is trained on PhraseCut (extended with visual prompts in the paper; the refined rd64 checkpoint, reduce_dim 64); the CLIP towers are OpenAI's frozen ViT-B/16, trained on web image-text pairs that were not released (CLIPSeg paper, arXiv:2112.10003).)
  • Options: cuda_graphs = True (speed only)
  • Kernel libraries: fvm_conv
  • Variants: clipseg
  • Outputs: boxes, scores, masks
  • Input size: any H x W >= 1, resized inside to the fixed encoder input 352x352 (the default export size)
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): clipseg 3000x4000 (0.7 GiB peak allocated, 0.9 reserved)
  • clipseg 1024x1024, first call: portable + torch.compile 10 s (cold Inductor cache), TensorRT fp16 40 s (ONNX + engine).
  • clipseg original 1024x1024: .half() on 1 modules, fp16 inputs; transformers CLIPSegProcessor (CPU) + CLIPSegForImageSegmentation, text tower per call, masks at logits > 0
  • clipseg load / first call: original 3.05 s / 0.021 s, portable 4.93 s / 0.287 s, kernels 4.54 s / 0.236 s
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

One text prompt per present COCO class. The label is the union of the class's non-crowd masks. Scores average pixel IoU aggregated per class, and IoU per image and prompt. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetportablenote
clipsegCOCO val2017, every 10th image, class masks, class / prompt mIoU %55.00 / 38.37

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks bc6ebb31, 2026-11-17.

Parity

CPU, float32: max |diff| <= 0.0005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.1 and p99 <= 0.5 (absolute error of the output boxes).

variantkernels error vs original, mean / p99GPU, torchkernels used
clipseg0.01273 / 0.081RTX 5060 Ti, torch 2.14.1+cu130fvm_conv@f1d064e539e1:sm_120

Limitations

  • No accuracy measurement: CLIPSeg is scored on PhraseCut and Pascal-5i one-shot splits, which are not part of the evaluation data here; tiers are compared by output parity and speed only.
  • Requires the optional transformers library (the network is its CLIPSegForImageSegmentation).
  • The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (CIDAS/clipseg-rd64-refined); the authors' timojl/clipseg code was not run.
  • Text prompts only (no box exemplars or visual prompts); one instance per prompt: its mask is every pixel above probability 0.5, its score the highest pixel probability, its box the mask's bounding box.
  • The image is resized to 352 x 352 without keeping the aspect ratio, as the original processor; masks are coarse (22 x 22 patch tokens).

Example output

No example rendered for this checkpoint yet.