Models · 3D volume segmentation

nnFoundation

nnFoundation (DKFZ): self-supervised 3D encoders (a convolutional one and a vision transformer) pre-trained on 2.1 million CT, MRI and PET volumes; they turn a volume into feature maps for your own medical models. Share-alike weights (CC-BY-SA-4.0): fine-tuned models keep the licence.

experimental Fork of MIC-DKFZ/nnssl@3d774e9 · backends: portable, kernels

Checkpoint

Paper 2026 Added in 0.2.0 Evaluated with 0.2.0

Kernels, RTX 5060 Ti 16 GB 211 ms 200x200, fp16
vs the faster original 1.3× original fp16: 274 ms
Peak memory 2,985 MiB original: 3,818 MiB
Accuracy vs original fp32 −0.00001 kernels cos_min, TotalSegmentator test subset (8 CT cases) (1.00000 original)

Use it

pip install fastforks

Lean install with only what this model needs: pip install fastforks-core

from fastforks import volume

model = volume.load("nnfoundation-cnn")  # kernels on a supported GPU, else portable PyTorch
result = model(volume, spacing)
result.features
result.backend  # what ran: mode, device, dtype, kernels
fastforks bench nnfoundation-cnn

Times the model on this machine. load(…, mode="portable") or mode="kernels" picks a backend; usage guide.

Speed and memory

Every backend on one GPU (RTX 5060 Ti 16 GB) and the same input, the authors’ code included. Kernels runs fp16. Portable runs fp16 on CUDA.

Input: synthetic, 200x200 (the task’s standard size)

BackenddtypeTime per call (ms)vs original fp16Peak memory
allocated / reserved
Load (s)First call (s)
Original fp32nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forwardfp32 5010.55×7,247 MiB/8,734 MiB1.30.92
Original fp16nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forwardfp16 274–3,818 MiB/4,628 MiB1.20.69
Portable fp32fp32not measured ––––
Portablefp16 2261.2×3,011 MiB/3,130 MiB0.60.27
Kernelsfp16 2111.3×2,985 MiB/3,710 MiB0.50.25
Portable + torch.compile 1701.6×5,067 MiB/5,110 MiB20 (build)–
TensorRT fp16encoder on a normalized, padded volume only; memory estimated onlyfp16 2551.1×2,418 MiBenginenot comparable49 (build)–

Measured on RTX 5060 Ti 16 GB, torch 2.10.0+cu130, 2.14.1+cu130, 2026-11-14 to 2026-11-16. First call, 3 warm-up, peak reset, median of 3 rounds x median of 5. Ratios divide two times of this table (same GPU, same input). Original fp16 is the authors’ own precision option, else torch.autocast. torch.compile and TensorRT: build time instead of load time. TensorRT memory (engine, not comparable) is the GPU memory the engine took, read from the driver because TensorRT allocates outside PyTorch, so it can't be compared with the PyTorch peaks of the other rows.

Other devices

DeviceBackenddtypems / callPeak memoryOutput vs referenceInput
Apple M1 Pro 16 GB MPS, torch 2.14.1, 2026-11-15Originalfp32cannot run here–200x200
Portablefp162935 7,824 MiB processmatches200x200
Portablefp323645 9,435 MiB processmatches200x200
Kernelsfp16not implemented–200x200

CUDA memory is torch’s peak allocated / reserved; Apple has no such counter, so the Mac rows show the process footprint (weights, Python and torch included). “Output vs reference” compares the device’s output with the stored golden output under the card’s tolerance; – = not checked on that device. Kernels “not implemented” = no kernels for this model on that backend yet: it runs portable there.

Licence and weights

Commercial use allowed

Code
Apache-2.0 LICENSE
Weights
CC-BY-SA-4.0 source
Training data
2.1 million CT, MRI and PET volumes from 125 institutional and public datasets (Human Radiome Project); the dataset list is not published with the weights. Share-alike: models fine-tuned from these weights are adapted material under CC-BY-SA-4.0.
  • share-alike: weights CC-BY-SA-4.0; fine-tuned or adapted weights keep this licence

Our reading of the licences, not legal advice.

Weights

nnFoundationCNN/checkpoint_final.pth 410 MB· hosted on Hugging Face

fastforks downloads and checks these files on first use (fastforks download nnfoundation-cnn).

Accuracy

The test set through the authors’ code and through ours: same checkpoint, same protocol. The distance between the original’s fp32 and fp16 runs is what half precision alone costs; our runs should stay within it.

TotalSegmentator test subset (8 CT cases)

encoder features vs the original fp32: cosine (worst case) / relative L2

Rundtypecos_min ↑rel_l2 ↑Δ cos_min vs fp32 originalcasesms / case
Original fp32fp321.000000.0000 reference 8 –
Original fp16fp160.999990.0014 −0.00001 8 –
Portable fp32not measured
Portablefp160.999990.0017 −0.00001 8 –
Kernelsfp160.999990.0015 −0.00001 8 –

A small subset of real CT cases of the TotalSegmentator dataset (not the full test split), each run at its card's spacing (1.5 or 3 mm) in the same pipeline; mean over the cases of the Dice (0 to 1) averaged over the classes present, against the ground-truth segmentations (vertebrae_L6 of the subset has no counterpart and is skipped). Original = TotalSegmentator 2.18.0, default fp16 autocast. Encoder models have no task head, so they are judged by their output: the encoder feature maps against the original fp32 maps (cosine similarity, worst case over the cases and the mean over the maps; relative L2 error, mean). Original (fp32) is the reference; the original fp16 row shows the error the original itself has in half precision. Original = the authors’ code, same checkpoint and protocol. Measured on original fp32, original fp16: RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14; portable, kernels: RTX 5060 Ti, torch 2.14.1+cu130, fastforks 715d7cb5, 2026-11-14.

Closeness to the original

Beyond the test-set scores: the raw output of ours against the original’s fp32 output on the same input, pixel by pixel. CPU, float32: max |diff| <= 1.8e-06 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 % and p99 <= 1.1 % of the reference's median |value| (absolute error of the output features).

CheckToleranceMeasured
CPU fp32, max |diff|≤ 0.0000018 (relative)not recorded
GPU default dtypes, mean |diff|≤ 0.0015 (relative)0.00067 (relative)
GPU default dtypes, p99 |diff|≤ 0.011 (relative)0.00463 (relative)

Recorded for nnfoundation-cnn on RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-14, with these kernels: vol_norm

Details

nnFoundation (Harsy et al. 2026): DKFZ's self-supervised 3D radiology foundation encoders, pre-trained with nnssl on 2.1 M CT, MRI and PET volumes. `nnfoundation-cnn` is the ResEnc-L U-Net encoder (6 stages, 32-320 channels, strides 1-32), `nnfoundation-vit` the Primus ViT (40 layers, width 1056, 8³ patch tokens, fixed 192³ input). Gives the encoder feature maps of a z-score normalised volume for fine-tuning or frozen-feature use; the MAE decoder is not included.

Input / output
Input: a CT volume [D, H, W] in Hounsfield units and its voxel spacing in mm (volume.read_nifti reads both from a NIfTI file). Output: a class index per voxel [D, H, W] on the input grid (0 = background).
What the numbers mean, and which models need the camera focal length

Encoder models (nnFoundation) take a CT or MRI volume and give its encoder feature maps. For a trained nnU-Net v2 folder or checkpoint, see the [nnU-Net v2 loader](../api/volume.md#nnu-net-v2).

Input size
any H x W >= 32, padded to a multiple of 32; tested up to 384x384 (14.2 GiB allocated)
Outputs
features
Checkpoint
arch="cnn"
CUDA kernels
  • vol_norminstance norm and LeakyReLU on 3D volumes
Apple (MPS)
fp32, fp16

Limitations

  • Kernels peak allocated is equal to portable, but the cuDNN workspace next to the large conv maps (1.4 GiB and 0.3 GiB workspaces) fragments the allocator: peak reserved is ~580 MiB above portable.
Full reference, generated from the repository

nnFoundation (DKFZ): self-supervised 3D encoders (a convolutional one and a vision transformer) pre-trained on 2.1 million CT, MRI and PET volumes; they turn a volume into feature maps for your own medical models. Share-alike weights (CC-BY-SA-4.0): fine-tuned models keep the licence.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
nnfoundation-cnnsynthetic (200x200)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
501
274
226
211
170
255 (encoder on a normalized, padded volume only; memory estimated)
7247 / 8734
3818 / 4628
3011 / 3130
2985 / 3710
5067 / 5110
3374 / 3398
nnfoundation-vitsynthetic (192x192)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
OOM
1930
1848
1732
1440
planned for 0.2.1
–
3930 / 4298
1669 / 1884
1669 / 1884
1406 / 1484
–
Details
  • Original code: https://github.com/MIC-DKFZ/nnssl at 3d774e942256
  • Code licence: Apache-2.0 (permissive); files: LICENSE
  • Weights licence: CC-BY-SA-4.0
  • Licence evidence: https://huggingface.co/MIC-DKFZ/nnFoundationCNN/blob/f443c09319988e833b3d9589bf95909e4443d7b2/README.md ; https://huggingface.co/MIC-DKFZ/nnFoundationViT/blob/c691f2115140458ec64017712eb8ebbc8dde6ae4/README.md ; https://github.com/MIC-DKFZ/dynamic-network-architectures/blob/main/LICENSE
  • Training data: 2.1 million CT, MRI and PET volumes from 125 institutional and public datasets (Human Radiome Project); the dataset list is not published with the weights. Share-alike: models fine-tuned from these weights are adapted material under CC-BY-SA-4.0.
  • Kernel libraries: vol_norm
  • Variants: nnfoundation-cnn, nnfoundation-vit
  • Not in this release (planned for 0.2.1): nnfoundation-vit ONNX / TensorRT export: the fixed-volume graph exports but TensorRT encoder features diverge from portable on real CT crops
  • Outputs: features
  • Input size: any D x H x W volume, no resampling, one encoder pass (a fixed-grid ViT: each side up to its grid)
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): nnfoundation-cnn 384x384x384 voxels (14.2 GiB peak allocated, 14.8 reserved); nnfoundation-vit 192x192x192 voxels (1.6 GiB peak allocated, 1.8 reserved)
  • nnfoundation-cnn 200x200, first call: portable + torch.compile 20 s (cold Inductor cache), TensorRT fp16 49 s (ONNX + engine).
  • nnfoundation-vit 192x192, first call: portable + torch.compile 69 s (cold Inductor cache).
  • nnfoundation-cnn original (fp32) 200x200, original 200x200: nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forward
  • nnfoundation-vit original 192x192: nnssl network (dna 0.4.4) encoder; nnU-Net ZScoreNormalization on the CPU + pad + forward
  • nnfoundation-cnn load / first call: original 1.18 s / 0.69 s, portable 0.56 s / 0.269 s, kernels 0.39 s / 0.291 s
  • nnfoundation-vit load / first call: original 9.54 s / 2.37 s (192x192), portable 1.24 s / 1.867 s (192x192), kernels 1.27 s / 1.756 s (192x192)
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5

Measured

A small subset of real CT cases of the TotalSegmentator dataset (not the full test split), each run at its card's spacing (1.5 or 3 mm) in the same pipeline; mean over the cases of the Dice (0 to 1) averaged over the classes present, against the ground-truth segmentations (vertebrae_L6 of the subset has no counterpart and is skipped). Original = TotalSegmentator 2.18.0, default fp16 autocast. Encoder models have no task head, so they are judged by their output: the encoder feature maps against the original fp32 maps (cosine similarity, worst case over the cases and the mean over the maps; relative L2 error, mean). Original (fp32) is the reference; the original fp16 row shows the error the original itself has in half precision. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portablekernelsnote
nnfoundation-cnnTotalSegmentator test subset (8 CT cases), encoder features vs the original fp32: cosine (worst case) / relative L21.00000 / 0.00000.99999 / 0.00140.99999 / 0.00170.99999 / 0.0015
nnfoundation-vitTotalSegmentator test subset (8 CT cases), encoder features vs the original fp32: cosine (worst case) / relative L21.00000 / 0.00000.99947 / 0.01670.99950 / 0.01650.99940 / 0.0213

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 715d7cb5, 2026-11-14 to 2026-11-15.

Parity

CPU, float32: max |diff| <= 1.8e-06 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 % and p99 <= 1.1 % of the reference's median |value| (absolute error of the output features).

variantkernels error vs original, mean / p99GPU, torchkernels used
nnfoundation-cnn0.00067 / 0.00463RTX 5060 Ti, torch 2.14.1+cu130vol_norm@44036d4da567:sm_120
nnfoundation-vit0.0067 / 0.03039RTX 5060 Ti, torch 2.14.1+cu130vol_norm@44036d4da567:sm_120

Limitations

  • Kernels peak allocated is equal to portable, but the cuDNN workspace next to the large conv maps (1.4 GiB and 0.3 GiB workspaces) fragments the allocator: peak reserved is ~580 MiB above portable.

Example output

No example rendered for this checkpoint yet.