PuzzleTuning
Only 1% of data to build pathology foundation model.
Building a pathology foundation model with only 1% of the data: explicitly learning pathology-specific structural relationships, preserving general visual knowledge through prompt tuning, and progressively adapting to different feature scales via curriculum learning.
Teaching Foundational Model to Restore a Batch of Puzzles
A batch of pathology images is cut into patches. Under a fix-position ratio, some are assigned as position patches that stay behind as hints, while the relation patches are shuffled in place across images into the Puzzle state, then restored by the model. The batch below is 4 images, m = 9 patches each, at a 1/3 fix-position ratio.
Patching & identity assignment
Every image in the batch is cut into m patches, and those m patches form that image’s bag. Under the fix-position ratio, r of them are picked as position patches (blue); the remaining m − r are relation patches (yellow).
In-place shuffle
Relation patches are swapped only across these bags, always staying in the same (X, Y) cell — that is what “in-place” means. Position patches never move at all, which yields the Puzzle state.
Puzzle restoring
At decoding time the original position tokens replace the encoded ones as a hint of the puzzle starting point; the decoder then reconstructs the relation patches, supervised by the original images themselves.
Scale alone does not tell a model what matters in pathology.
Conventional SSL learns from unlabeled pathology images, but the training objective may remain generic. PuzzleTuning starts from two explicit questions: what relationships are essential in pathology, and how can natural-image knowledge be preserved rather than overwritten?
Explicitly model pathology-specific structure
Pathology combines local homogeneity, global heterogeneity, spatial continuity and biological entities at different scales. The pre-training task should target these relationships rather than discover them only implicitly.
Keep general vision knowledge while adding domain knowledge
Instead of fully rewriting an ImageNet-pretrained backbone, prompt tokens carry pathology-specific bridging knowledge while the backbone is frozen during pre-training.
Teach the model the relationships pathologists rely on.
PuzzleTuning defines three task focuses — appearance consistency, spatial consistency and restoration understanding — mapped to grouping, junction and semantic-alignment relationships.
Appearance consistency
Patches from the same image tend to share coherent appearance. Mixing organs or conditions exposes consistent versus conflicting features.
Spatial consistency
Adjacent patches carry continuous textures and structures. Restoring their junctions forces the model to understand local tissue composition.
Restoration understanding
Correcting shuffled or missing regions requires semantic alignment across cells, clusters, glands and tissue-level structures.
Make puzzles. Understand relationships. Restore pathology.
A batch of pathology images is split into patches. Relation patches are shuffled across images while position patches remain as hints. The encoder learns the relational structure and a low-level decoder reconstructs the original batch under self-supervision.
Puzzle making
Split images into patch bags; randomly assign fixed-position and relation identities; shuffle relation patches in-place across the batch.
Puzzle understanding
The ViT encoder uses prompt tokens to learn grouping, junction and restoration relationships while preserving the general backbone.
Puzzle restoring
Position patches provide hints; a decoder reconstructs relation patches and is supervised against the original images.
Preserve general vision. Inject pathology knowledge efficiently.
Two complementary mechanisms control the bridge: compact prompt tokens encode the additional domain knowledge, while curriculum learning progressively increases puzzle difficulty and traverses feature scales.
Prompt tuning: knowledge without overwriting
Start from natural-image pretrained ViT weights, freeze backbone parameters during PuzzleTuning, and update only additional prompt tokens that explicitly carry domain-bridging knowledge.
Curriculum: easy-to-hard, scale by scale
The fix-position ratio decreases to increase shuffling difficulty, while patch size cycles across scales so the model learns relationships from larger structures down to finer morphology.
One pre-training recipe, multiple organs, scales and tasks.
PuzzleTuning is evaluated from subcellular and cellular ROIs to tissue-level regions, nuclei segmentation, and whole-slide classification — a deliberate test of whether the learned representation transfers across pathology scales.
Competitive with billion-tile foundation models — using 3M tiles.
Under the reported identical fine-tuning protocol, PuzzleTuning achieves the highest Accuracy and F1 on five of seven evaluated datasets. It is not the top model on pRCC or WBC, which are shown explicitly below.
F1 score: PuzzleTuning vs best competing foundation model
Vision-only and vision-language pathology foundation models include GigaPath, UNI/UNI2, Virchow2, CONCH and MUSK.
Bars are normalized within each card for visual comparison; the numeric F1 values are the reported scores.
ROI classification
Across CAM16, pRCC, ROSE and WBC, PuzzleTuning reports the strongest overall performance against same-scale SSL baselines.
Nuclei segmentation
PuzzleTuning transfers to low-level vision with CellViT/CellVPT, showing that the representation is not limited to classification.
Whole-slide classification
At WSI scale, PuzzleTuning improves both feature extraction and downstream feature-modeling stages in the reported experiments.
ROI classification vs. SSL SOTA
Table II compares PuzzleTuning with 12 representative SSL pre-training baselines. Each card below contrasts PuzzleTuning with the strongest non-Puzzle baseline on that ROI dataset.
Metric shown: F1 (%). Strongest competing baselines are MaskFeat, MAE, TransPath (BYOL) and SimCLR respectively.
Nuclei segmentation transfer
Table III evaluates CellViT / CellVPT transfer under four downstream tuning modes. PuzzleTuning is best or near-best across the board, while remaining clearly competitive even in the single setting where MaskFeat is slightly higher.
Metric shown: DICE (%). Settings are ViT+ft, ViT+pt, VPT+ft and VPT+pt.
Whole-slide classification transfer
Table IV reports WSI transfer in both the feature-extraction stage (CLAM) and the feature-modeling stage (GTP). The strongest gains appear when prompt tokens are preserved during WSI transfer.
Metric shown: Accuracy (%). Each block compares PuzzleTuning with the strongest non-Puzzle baseline for that transfer setting.
Restoration quality reveals more coherent local morphology.
The reconstruction studies compare the proposed shuffled auto-encoder (PuzzleTuning) with MAE. The reported visualizations show smoother, more coherent reconstructed patches and better continuity with visible hint regions.
The gains come from design choices, not only more data.
The manuscript ablates the restoration objective, prompt design, curriculum schedule, shuffling group size, prompt-token count and the fraction of CPIA-mini used for pre-training.
Puzzles beat masking
Replacing MAE masking with multiple puzzle restoration improves the best accuracy by +1.48, +4.09, +3.29 and +1.11 pts across the four ROI tasks.
Compact prompts work best
A 20-token prompt budget gives the strongest overall result; 100–200 tokens can dilute attention and overfit cohort-specific texture.
Moderate puzzle complexity
Shuffling 8 images balances difficulty and learnability across heterogeneous tissue tasks; very small groups make the task too easy.
Data-efficient activation
With only 1% of CPIA-mini, the model already reaches 94.17 CAM16 Acc and 92.07 ROSE Acc, while additional data benefits more diverse morphologies.
| Method | CAM16 F1 | pRCC F1 | ROSE F1 | WBC F1 | MHIST F1 | CRC-MSI F1 | Osteosarcoma F1 |
|---|---|---|---|---|---|---|---|
| GigaPath | 94.91 | 95.11 | 90.08 | 98.81 | 84.93 | 89.64 | 96.61 |
| UNI2 | 95.21 | 95.37 | 89.58 | 98.71 | 83.96 | 90.02 | 97.68 |
| Virchow2 | 95.61 | 94.24 | 87.83 | 98.21 | 86.35 | 89.25 | 96.61 |
| CONCH | 95.15 | 89.27 | 87.66 | 97.20 | 84.49 | 87.48 | 95.18 |
| MUSK | 95.09 | 88.51 | 86.56 | 98.16 | 82.94 | 85.90 | 95.18 |
| PuzzleTuning | 95.83 | 93.54 | 92.52 | 97.36 | 88.06 | 90.16 | 97.85 |
Reported average top-5 downstream fine-tuning results; see the PuzzleTuning manuscript for full protocols, p-values and complete model set.
A foundation-model direction beyond brute-force scaling.
PuzzleTuning explores a complementary route: explicitly encode pathology structure and domain bridging so strong representations can emerge without relying only on billion-tile corpora and extreme compute.