Models · 3D volume segmentation
nnFoundation
nnFoundation (DKFZ): self-supervised 3D encoders (a convolutional one and a vision transformer) pre-trained on 2.1 million CT, MRI and PET volumes; they turn a volume into feature maps for your own medical models. Share-alike weights (CC-BY-SA-4.0): fine-tuned models keep the licence.
experimental Fork of MIC-DKFZ/nnssl@3d774e9 · backends: portable, kernels
- Paper nnFoundation: 3D Foundation Models for Radiology arXiv 2026 · first released 22 Sept 2026
Paper 2026 Added in 0.2.0 Evaluated with 0.2.0
Use it
pip install fastforks Lean install with only what this model needs: pip install fastforks-core
from fastforks import volume
model = volume.load("nnfoundation-cnn") # kernels on a supported GPU, else portable PyTorch
result = model(volume, spacing)
result.features
result.backend # what ran: mode, device, dtype, kernels fastforks bench nnfoundation-cnn Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.
Speed and memory
Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.
Input: synthetic, 200x200 (the task’s standard size)
| Backend | dtype | Time per call (ms) | vs original fp16 | Peak memory allocated / reserved | Load (s) | First call (s) |
|---|---|---|---|---|---|---|
| Original fp32nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forward | fp32 | 501 | 0.55× | 7,247 MiB/8,734 MiB | 1.3 | 0.92 |
| Original fp16nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forward | fp16 | 274 | – | 3,818 MiB/4,628 MiB | 1.2 | 0.69 |
| Portable fp32 | fp32 | not measured | – | – | – | – |
| Portable | fp16 | 226 | 1.2× | 3,011 MiB/3,130 MiB | 0.6 | 0.27 |
| Kernels | fp16 | 211 | 1.3× | 2,985 MiB/3,710 MiB | 0.5 | 0.25 |
| Portable + torch.compile | 170 | 1.6× | 5,067 MiB/5,110 MiB | 20 (build) | – | |
| TensorRT fp16encoder on a normalized, padded volume only; memory estimated only | fp16 | 255 | 1.1× | 2,418 MiBenginenot comparable | 49 (build) | – |
Measured on RTX 5060 Ti 16 GB, torch 2.10.0+cu130, 2.14.1+cu130, 2026-11-14 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two
times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.
Other devices
| Device | Backend | dtype | ms / call | Peak memory | Output vs reference | Input |
|---|---|---|---|---|---|---|
| Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-15 | Original | fp32 | cannot run here | – | 200x200 | |
| Portable | fp16 | 2935 | 7,824 MiB process | matches | 200x200 | |
| Portable | fp32 | 3645 | 9,435 MiB process | matches | 200x200 | |
| Kernels | fp16 | not implemented | – | 200x200 | ||
CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.
Licence and weights
Commercial use allowed
- Code
- Apache-2.0 LICENSE
- Weights
- CC-BY-SA-4.0 source
- Training data
- 2.1 million CT, MRI and PET volumes from 125 institutional and public datasets (Human Radiome Project); the dataset list is not published with the weights. Share-alike: models fine-tuned from these weights are adapted material under CC-BY-SA-4.0.
- share-alike: weights CC-BY-SA-4.0; fine-tuned or adapted weights keep this licence
Our reading of the licences, not legal advice.
Weights
nnFoundationCNN/checkpoint_final.pth 410 MB· hosted on Hugging Facefastforks downloads and checks these files on first use (fastforks download nnfoundation-cnn).
Accuracy
The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.
TotalSegmentator test subset (8 CT cases)
encoder features vs the original fp32: cosine (worst case) / relative L2
| Run | dtype | cos_min ↑ | rel_l2 ↑ | Δ cos_min vs fp32 original | cases | ms / case |
|---|---|---|---|---|---|---|
| Original fp32 | fp32 | 1.00000 | 0.0000 | reference | 8 | – |
| Original fp16 | fp16 | 0.99999 | 0.0014 | −0.00001 | 8 | – |
| Portable fp32 | not measured | |||||
| Portable | fp16 | 0.99999 | 0.0017 | −0.00001 | 8 | – |
| Kernels | fp16 | 0.99999 | 0.0015 | −0.00001 | 8 | – |
A small subset of real CT cases of the TotalSegmentator dataset (not the full test split), each run at its card's spacing (1.5 or 3 mm) in the same pipeline; mean over the cases of the Dice (0 to 1) averaged over the classes present, against the ground-truth segmentations (vertebrae_L6 of the subset has no counterpart and is skipped). Original = TotalSegmentator 2.18.0, default fp16 autocast. Encoder models have no task head, so they are judged by their output: the encoder feature maps against the original fp32 maps (cosine similarity, worst case over the cases and the mean over the maps; relative L2 error, mean). Original (fp32) is the reference; the original fp16 row shows the error the original itself has in half precision. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 715d7cb5, 2026-11-14.
Closeness to the original
Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 1.8e-06 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 % and p99 <= 1.1 % of the reference's median |value| (absolute error of the output features).
| Check | Tolerance | Measured |
|---|---|---|
| CPU fp32, max |diff| | ≤ 0.0000018 (relative) | not recorded |
| GPU default dtypes, mean |diff| | ≤ 0.0015 (relative) | 0.00067 (relative) |
| GPU default dtypes, p99 |diff| | ≤ 0.011 (relative) | 0.00463 (relative) |
Recorded for nnfoundation-cnn on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: vol_norm
Details
nnFoundation (Harsy et al. 2026): DKFZ's self-supervised 3D radiology foundation encoders, pre-trained with nnssl on 2.1 M CT, MRI and PET volumes. `nnfoundation-cnn` is the ResEnc-L U-Net encoder (6 stages, 32-320 channels, strides 1-32), `nnfoundation-vit` the Primus ViT (40 layers, width 1056, 8³ patch tokens, fixed 192³ input). Gives the encoder feature maps of a z-score normalised volume for fine-tuning or frozen-feature use; the MAE decoder is not included.
- Input / output
- Input: a CT volume
[D, H, W]in Hounsfield units and its voxel spacing in mm (volume.read_niftireads both from a NIfTI file). Output: a class index per voxel[D, H, W]on the input grid (0 = background).What the numbers mean, and which models need the camera focal length
Encoder models (nnFoundation) take a CT or MRI volume and give its encoder feature maps. For a trained nnU-Net v2 folder or checkpoint, see the [nnU-Net v2 loader](../api/volume.md#nnu-net-v2).
- Input size
- any H x W >= 32, padded to a multiple of 32; tested up to 384x384 (14.2 GiB allocated)
- Outputs
features- Checkpoint
arch="cnn"- CUDA kernels
vol_norminstance norm and LeakyReLU on 3D volumes
- Apple (MPS)
- fp32, fp16
Limitations
- Kernels peak allocated is equal to portable, but the cuDNN workspace next to the large conv maps (1.4 GiB and 0.3 GiB workspaces) fragments the allocator: peak reserved is ~580 MiB above portable.
Full reference, generated from the repository
nnFoundation (DKFZ): self-supervised 3D encoders (a convolutional one and a vision transformer) pre-trained on 2.1 million CT, MRI and PET volumes; they turn a volume into feature maps for your own medical models. Share-alike weights (CC-BY-SA-4.0): fine-tuned models keep the licence.
Commercial use: yes.
| variant | input | backend | time (ms) | peak memory (MiB, allocated / reserved) |
|---|---|---|---|---|
nnfoundation-cnn | synthetic (200x200) | original (fp32) original portable kernels portable + torch.compile TensorRT fp16 | 501 274 226 211 170 255 (encoder on a normalized, padded volume only; memory estimated) | 7247 / 8734 3818 / 4628 3011 / 3130 2985 / 3710 5067 / 5110 3374 / 3398 |
nnfoundation-vit | synthetic (192x192) | original (fp32) original portable kernels portable + torch.compile TensorRT fp16 | OOM 1930 1848 1732 1440 planned for 0.2.1 | – 3930 / 4298 1669 / 1884 1669 / 1884 1406 / 1484 – |
Details
- Original code: https://github.com/MIC-DKFZ/nnssl at
3d774e942256 - Code licence: Apache-2.0 (permissive); files:
LICENSE - Weights licence: CC-BY-SA-4.0
- Licence evidence: https://huggingface.co/MIC-DKFZ/nnFoundationCNN/blob/f443c09319988e833b3d9589bf95909e4443d7b2/README.md ; https://huggingface.co/MIC-DKFZ/nnFoundationViT/blob/c691f2115140458ec64017712eb8ebbc8dde6ae4/README.md ; https://github.com/MIC-DKFZ/dynamic-network-architectures/blob/main/LICENSE
- Training data: 2.1 million CT, MRI and PET volumes from 125 institutional and public datasets (Human Radiome Project); the dataset list is not published with the weights. Share-alike: models fine-tuned from these weights are adapted material under CC-BY-SA-4.0.
- Kernel libraries:
vol_norm - Variants:
nnfoundation-cnn,nnfoundation-vit - Not in this release (planned for 0.2.1):
nnfoundation-vitONNX / TensorRT export: the fixed-volume graph exports but TensorRT encoder features diverge from portable on real CT crops - Outputs: features
- Input size: any D x H x W volume, no resampling, one encoder pass (a fixed-grid ViT: each side up to its grid)
- Tested up to (portable, fp16, RTX 5060 Ti 16 GB):
nnfoundation-cnn384x384x384 voxels (14.2 GiB peak allocated, 14.8 reserved);nnfoundation-vit192x192x192 voxels (1.6 GiB peak allocated, 1.8 reserved) nnfoundation-cnn200x200, first call: portable + torch.compile 20 s (cold Inductor cache), TensorRT fp16 49 s (ONNX + engine).nnfoundation-vit192x192, first call: portable + torch.compile 69 s (cold Inductor cache).nnfoundation-cnnoriginal (fp32) 200x200, original 200x200: nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forwardnnfoundation-vitoriginal 192x192: nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forwardnnfoundation-cnnload / first call: original 1.18 s / 0.69 s, portable 0.56 s / 0.269 s, kernels 0.39 s / 0.291 snnfoundation-vitload / first call: original 9.54 s / 2.37 s (192x192), portable 1.24 s / 1.867 s (192x192), kernels 1.27 s / 1.756 s (192x192)- Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5
Measured
A small subset of real CT cases of the TotalSegmentator dataset (not the full test split), each run at its card's spacing (1.5 or 3 mm) in the same pipeline; mean over the cases of the Dice (0 to 1) averaged over the classes present, against the ground-truth segmentations (vertebrae_L6 of the subset has no counterpart and is skipped). Original = TotalSegmentator 2.18.0, default fp16 autocast. Encoder models have no task head, so they are judged by their output: the encoder feature maps against the original fp32 maps (cosine similarity, worst case over the cases and the mean over the maps; relative L2 error, mean). Original (fp32) is the reference; the original fp16 row shows the error the original itself has in half precision. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | kernels | note |
|---|---|---|---|---|---|---|
nnfoundation-cnn | TotalSegmentator test subset (8 CT cases), encoder features vs the original fp32: cosine (worst case) / relative L2 | 1.00000 / 0.0000 | 0.99999 / 0.0014 | 0.99999 / 0.0017 | 0.99999 / 0.0015 | |
nnfoundation-vit | TotalSegmentator test subset (8 CT cases), encoder features vs the original fp32: cosine (worst case) / relative L2 | 1.00000 / 0.0000 | 0.99947 / 0.0167 | 0.99950 / 0.0165 | 0.99940 / 0.0213 |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 715d7cb5, 2026-11-14 to 2026-11-15.
Parity
CPU, float32: max |diff| <= 1.8e-06 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 % and p99 <= 1.1 % of the reference's median |value| (absolute error of the output features).
| variant | kernels error vs original, mean / p99 | GPU, torch | kernels used |
|---|---|---|---|
nnfoundation-cnn | 0.00067 / 0.00463 | RTX 5060 Ti, torch 2.14.1+cu130 | vol_norm@44036d4da567:sm_120 |
nnfoundation-vit | 0.0067 / 0.03039 | RTX 5060 Ti, torch 2.14.1+cu130 | vol_norm@44036d4da567:sm_120 |
Limitations
- Kernels peak allocated is equal to portable, but the cuDNN workspace next to the large conv maps (1.4 GiB and 0.3 GiB workspaces) fragments the allocator: peak reserved is ~580 MiB above portable.
Example output
No example rendered for this checkpoint yet.