Regression Should Predict Full Distributions

Key Takeaways

• TabPFN and TabICL generate full predictive distributions internally
• Quantile outputs require only a parameter change, no extra compute
• Distributional forecasts enhance uncertainty and tail‑risk insight
• Calibration of TFM distributions is still an open challenge
• Current benchmarks focus on point estimates, not distribution quality

Pulse Analysis

Regression has long been dominated by point predictions, leaving uncertainty hidden behind a single mean estimate. In contrast, classification models routinely provide class probabilities, enabling calibrated risk assessments and nuanced decision‑making. Emerging tabular foundation models—most notably TabPFN and TabICL—bridge this gap by pre‑training on synthetic data to produce a discretized predictive distribution for each input. Although their default API returns only the conditional mean, the underlying distribution is already computed, allowing users to extract any set of quantiles with a simple flag change. This design eliminates the need for multiple separate models, reducing engineering overhead while delivering richer statistical insight.

The practical upside is significant. By accessing the full distribution, analysts can retrieve medians for robustness, tail quantiles for extreme‑event planning, and variance or full density plots for visual storytelling. Because the distribution is generated once per inference, requesting quantiles incurs no additional computational cost compared with a mean‑only call. This efficiency makes distributional forecasting viable for large‑scale tabular workloads, from financial risk models to supply‑chain demand forecasts, where understanding uncertainty is as critical as the point forecast itself.

Nevertheless, the promise of full predictive distributions hinges on calibration. Tabular foundation models lack explicit distributional assumptions, which can lead to mis‑aligned quantiles if the learned distribution does not reflect reality. Current benchmark suites, such as TabArena, prioritize point‑estimate metrics, leaving a gap in systematic calibration evaluation. Practitioners must therefore employ proper scoring rules, conformal prediction, or bespoke validation pipelines to ensure reliability. As the community pushes for benchmarks that reward calibrated distributional outputs, the adoption of these models is likely to accelerate, reshaping how businesses approach predictive analytics.