Models · Promptable segmentation
CLIPSeg (text-prompted segmentation, CLIP B/16)
CLIPSeg (Lüddecke and Ecker, CVPR 2022): one coarse mask for whatever a free-text prompt names, from a frozen CLIP ViT-B/16 and a small decoder. Apache-2.0 code and weights; the PhraseCut training data has no stated terms.
experimental Fork of huggingface/transformers@c4c4605 · backends: portable, kernels
- Paper Image Segmentation Using Text and Image Prompts CVPR 2022 · first released 18 Dec 2021
- Authors’ code timojl/clipseg (PyTorch), not the code forked here: fastforks follows the transformers port, which loads the authors' converted weights (CIDAS/clipseg-rd64-refined) without the repo's CLIP dependency.
Checkpoint clipseg
Paper 2021 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install "fastforks-core[transformers]"
from fastforks import promptable_segmentation
model = promptable_segmentation.load("clipseg") # kernels on a supported GPU, else portable PyTorch
result = model(image, input_points=...)
result.boxes # also: scores, masks
result.backend # what ran: mode, device, dtype, kernels fastforks bench clipseg Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.
| Backend | dtype | Time per call (ms) | vs original fp16 | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | not measured | – | – | – | – |
| Original fp16.half() on 1 modules, fp16 inputs; transformers CLIPSegProcessor (CPU) + CLIPSegForImageSegmentation, text tower per call, masks at logits > 0 | fp16 | 15.3 | – | 346 MiB/404 MiB | 0.3 | 0.02 |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp16 | 4.46 | 3.4× | 328 MiB/356 MiB | 4.9 | 0.29 |
| Kernels | fp16 | 2.76 | 5.6× | 335 MiB/362 MiB | 4.5 | 0.24 |
| Portable + torch.compile | 3.19 | 4.8× | 348 MiB/374 MiB | 10 (build) | – | |
| TensorRT fp16segmentation network on a normalized 352x352 image and precomputed text embedding only | fp16 | 1.75 | 8.8× | 170 MiBenginenot comparable | 40 (build) | – |
Measured on RTX 5060 Ti 16 GB, torch 2.14.1+cu130, 2026-11-14 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-14 | Original | fp32 | 67.6 | 1,606 MiB process | – | 1024x1024 |
| Portable | fp16 | 37.7 | 1,561 MiB process | matches | 1024x1024 | |
| Portable | fp32 | 44.7 | 1,532 MiB process | matches | 1024x1024 | |
| Kernels | fp16 | not implemented | – | 1024x1024 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- Apache-2.0
- Weights
- Apache-2.0 source
- Training data
Our reading of the licences, not legal advice.
Weights
clipseg-rd64-refined.config.json 4.7 KB· hosted on Hugging Faceclipseg-rd64-refined.safetensors 603 MB· hosted on Hugging Faceclip-bpe.vocab.json 1.1 MB· hosted on Hugging Faceclip-bpe.merges.txt 525 KB· hosted on Hugging Facefastforks downloads and checks these files on first use (fastforks download clipseg).
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
COCO val2017
every 10th image, class masks, class / prompt mIoU %
| Run | dtype | miou_class ↑ | miou_prompt ↑ | images | ms / image |
|---|---|---|---|---|---|
| Original fp32 | not measured | ||||
| Original fp16 | not measured | ||||
| Portable fp32 | not measured | ||||
| Portable | fp16 | 55.00 | 38.37 | 500 | – |
| Kernels | not measured | ||||
The original code was not run on this dataset, so the table cannot show the distance to it.
One text prompt per present COCO class. The label is the union of the class's non-crowd masks. Scores average pixel IoU aggregated per class, and IoU per image and prompt. Measured on RTX 5060 Ti, torch 2.14.1+cu130, fastforks bc6ebb31, 2026-11-17.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 0.0005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.1 and p99 <= 0.5 (absolute error of the output boxes).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.0005 | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.1 | 0.01273 |
| GPU default dtypes, p99 |diff| | ≤ 0.5 | 0.081 |
Recorded for clipseg on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: fvm_conv
Details
CLIPSeg (Lüddecke and Ecker 2022): a frozen CLIP ViT-B/16 with a small transformer decoder that segments whatever a free-text prompt names; the text embedding conditions the decoder by FiLM. Semantic: one mask per prompt (all pixels of the concept), at 352 x 352, resized to the image. Pass `text=` at call time. The portable `module()` is the network on the normalised 352 x 352 image plus a projected text embedding.
- Input / output
- Input: one image plus a point, box or text prompt. Output: masks with quality scores.
- Input size
- any H x W >= 1, resized inside to the fixed encoder input 352x352 (the default export size); tested up to 3000x4000 (748 MiB allocated)
- Options
cuda_graphs=true
- Outputs
boxes,scores,masks- CUDA kernels
fvm_convfused convolutions for image classifiers
- Apple (MPS)
- fp32, fp16
Limitations
- No accuracy measurement: CLIPSeg is scored on PhraseCut and Pascal-5i one-shot splits, which are not part of the evaluation data here; tiers are compared by output parity and speed only.
- Requires the optional transformers library (the network is its `CLIPSegForImageSegmentation`).
- The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (CIDAS/clipseg-rd64-refined); the authors' timojl/clipseg code was not run.
- Text prompts only (no box exemplars or visual prompts); one instance per prompt: its mask is every pixel above probability 0.5, its score the highest pixel probability, its box the mask's bounding box.
- The image is resized to 352 x 352 without keeping the aspect ratio, as the original processor; masks are coarse (22 x 22 patch tokens).
Full reference, generated from the repository
CLIPSeg (Lüddecke and Ecker, CVPR 2022): one coarse mask for whatever a free-text prompt names, from a frozen CLIP ViT-B/16 and a small decoder. Apache-2.0 code and weights; the PhraseCut training data has no stated terms.
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
clipseg | synthetic (1024x1024) | original portable kernels portable + torch.compile TensorRT fp16 | 15.3 4.46 2.76 3.19 1.75 (segmentation network on a normalized 352x352 image and precomputed text embedding only) | 346 / 404 328 / 356 335 / 362 348 / 374 464 / 480 |
Details
- Original code: https://github.com/huggingface/transformers at
c4c4605f9618 - Code licence: Apache-2.0 (permissive); files:
- Weights licence: Apache-2.0
- Licence evidence: https://huggingface.co/CIDAS/clipseg-rd64-refined/blob/999e0328d9e10b484360c477313983f9afdd7050/README.md
- Training data: PhraseCut (referring-phrase masks on Visual Genome images) (The decoder is trained on PhraseCut (extended with visual prompts in the paper; the refined rd64 checkpoint, reduce_dim 64); the CLIP towers are OpenAI's frozen ViT-B/16, trained on web image-text pairs that were not released (CLIPSeg paper, arXiv:2112.10003).)
- Options:
cuda_graphs= True (speed only) - Kernel libraries:
fvm_conv - Variants:
clipseg - Outputs: boxes, scores, masks
- Input size: any H x W >= 1, resized inside to the fixed encoder input 352x352 (the default export size)
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
clipseg3000x4000 (0.7 GiB peak allocated, 0.9 reserved) clipseg1024x1024, first call: portable + torch.compile 10 s (cold Inductor cache), TensorRT fp16 40 s (ONNX + engine).clipsegoriginal 1024x1024: .half() on 1 modules, fp16 inputs; transformers CLIPSegProcessor (CPU) + CLIPSegForImageSegmentation, text tower per call, masks at logits > 0clipsegload / first call: original 3.05 s / 0.021 s, portable 4.93 s / 0.287 s, kernels 4.54 s / 0.236 s- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
One text prompt per present COCO class. The label is the union of the class's non-crowd masks. Scores average pixel IoU aggregated per class, and IoU per image and prompt. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | portable | note |
|---|---|---|---|
clipseg | COCO val2017, every 10th image, class masks, class / prompt mIoU % | 55.00 / 38.37 |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks bc6ebb31, 2026-11-17.
Parity
CPU, float32: max |diff| <= 0.0005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.1 and p99 <= 0.5 (absolute error of the output boxes).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
clipseg | 0.01273 / 0.081 | RTX 5060 Ti, torch 2.14.1+cu130 | fvm_conv@f1d064e539e1:sm_120 |
Limitations
- No accuracy measurement: CLIPSeg is scored on PhraseCut and Pascal-5i one-shot splits, which are not part of the evaluation data here; tiers are compared by output parity and speed only.
- Requires the optional transformers library (the network is its
CLIPSegForImageSegmentation). - The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (CIDAS/clipseg-rd64-refined); the authors' timojl/clipseg code was not run.
- Text prompts only (no box exemplars or visual prompts); one instance per prompt: its mask is every pixel above probability 0.5, its score the highest pixel probability, its box the mask's bounding box.
- The image is resized to 352 x 352 without keeping the aspect ratio, as the original processor; masks are coarse (22 x 22 patch tokens).
Example output
No example rendered for this checkpoint yet.