Authors: Deborah Boyenval

Affiliation: University of Helsinki

Email: deborah.boyenval@helsinki.fi

Description: We use TabPFN v2.5, a published tabular foundation model developed by PriorLabs (https://doi.org/10.1038/s41586-024-08328-6). TabPFN was originally designed for supervised tasks such as classification and regression, although data generation is supported through the official tabpfn-extensions package. It provides a model named `TabPFNUnsupervisedModel` that estimates an approximate joint distribution of features. Pretrained weights and implementation details are available at https://huggingface.co/Prior-Labs/tabpfn_2_5, running version 2.5 requires exporting an HF_TOKEN because the repository is authenticated.

TabPFN follows the foundation model paradigm: it is pretrained on over one hundred million synthetic datasets and can be used directly for downstream tasks without fine-tuning. Architecturally, TabPFN is a transformer trained to approximate the posterior predictive distribution for tabular inputs. During inference, it performs in-context learning: the "fit" stage corresponds to a single forward pass rather than parameter updates. During the pretraining stage, the model learns to handle a broad range of relationships among features. This setup behaves like amortized Bayesian inference, where the computational cost is paid during pretraining, enabling immediate, out-of-the-box use.

According to a recent report from Prior Labs, TabPFN-2.5 is designed to handle datasets of up to approximately 50K rows and 2K features, representing a major scalability improvement over TabPFN-v2, which was limited to around 10K samples and 500 features. While TabPFN is extremely efficient at inference, the `TabPFNUnsupervisedModel` has a practical limit of roughly 200 features. Its autoregressive factorization of the joint distribution causes the conditioning context to grow rapidly, leading to instability and occasional segmentation faults. To remain within stable bounds, we restricted each TCGA dataset to the 200 most hypervariable genes. Synthetic samples are generated by sampling from the conditional distributions produced at each autoregressive step. This sampling procedure has its own hyperparameter, such as for example a temperature parameter controlling sample diversity, that do not modify the model weights. Temperature is in principle eligible for optimisation, but we used default values and performed no hyperparameter optimisation.

Authors: Oleksandr Husak, ...

Affiliation: NCT/DKFZ

Description: ## Model Architecture and Approach
Our implementation leverages a class-conditional diffusion model with differential privacy.

### Core Components
- Denoising Neural Network: A multi-layer architecture with residual connections
- Adaptive Group Normalization (AdaGN): Custom normalization that dynamically adjusts based on both temporal and class conditioning signals
- Embedding Systems
- Sinusoidal time embeddings for diffusion timestep encoding
- Learned class embeddings for condition-guided generation
- Combined embedding projections to integrate multiple conditioning signals

## Diffusion Process Design
- Noise Schedule: Implements the improved cosine beta schedule from (DDPM++), providing several advantages:
- Smoother noise progression throughout the diffusion process
- Better sampling stability during the generative process
- Forward Process: Gradually adds noise according to the schedule, transforming real data into pure noise
- Reverse Process: Iteratively denoises from random noise to synthetic samples guided by class conditioning

## Privacy Preservation Mechanisms
- Differential Privacy Integration: Implemented via Opacus
- Privacy-Utility Tradeoff: Optimized through multi-objective hyperparameter tuning (MOASHA algorithm)

## Training Methodology
- Class-Weighted Loss Function: Addresses class imbalance using inverse square-root weighting to prevent over-representation of majority classes while stabilizing training
- Learning Rate Scheduling: Tested various schedulers (OneCycleLR, ReduceLROnPlateau, etc.) and selected ReduceLROnPlateau.

## Evaluation Framework
- Validation during training:
- Real-to-synthetic validation (train on real, test on synthetic)
- Synthetic-to-real validation (train on synthetic, test on real)

## Implementation Details
- PyTorch Lightning: Structured training and evaluation loops
- Hyperparameter Optimization: Multi-objective tuning balancing privacy budget (epsilon) and model utility (val_loss)

Authors: Jules Kreuer, Sofiane Ouaari

Affiliation: Methods in Medical Informatics - University of Tübingen

Description: The model implements a diffusion-based generative approach to the provided gene expression data with privacy considerations. At its core, the architecture uses a diffusion model with residual linear blocks with group normalisation.
The diffusion process follows a path where Gaussian noise is progressively added to the data according to different noise scheduling schemes (linear, cosine or power-based). During training, the model learns to predict and remove this noise. For reverse sampling, the model iteratively denoises random Gaussian samples guided by class conditioning to generate synthetic cancer data.
Key technical features include sinusoidal position embeddings for time-step encoding, attention blocks for capturing complex relationships within the data, and a privacy approach that combines differential privacy (by adding calibrated noise to gradients) and strong regularisation.

Additional improvements come from the implementation of early stopping, learning rate scheduling via OneCycleLR.
We explored attention and post-processing techniques such as outlier clipping, but discarded them as too computationally expensive and not useful enough.
The model training process includes gradient clipping to improve numerical stability.