Publications

2025

1. 

   Conditional Coverage Diagnostics for Conformal Prediction

   Sacha Braun, Holzmüller David, Michael I Jordan, and Francis Bach

   arXiv preprint arXiv:2512.11779, 2025

   Abs PDF Code

   Evaluating conditional coverage remains one of the most persistent challenges in assessing the reliabilityofpredictivesystems. Althoughconformalmethodscangiveguaranteesonmarginalcover- age, no method canguarantee to produce sets with correct conditional coverage, leaving practitioners without a clear way to interpret local deviations. To overcome sample-inefficiency and overfitting issues of existing metrics, we cast conditional coverage estimation as a classification problem. Condi- tional coverage is violated if and only if any classifier can achieve lower risk than the target coverage. Through the choice of a (proper) loss function, the resulting risk difference gives a conservative estimate of natural miscoverage measures such as L1 and L2 distance, and can even separate the effects of over- and under-coverage, as well as handle non-constant target coverages. We call the resulting family of metrics excess risk of the target coverage (ERT). We show experimentally that the use of modern classifiers provides much higher statistical power than simple classifiers underlying established metrics like CovGap. Additionally, we use our metric to benchmark different confor- mal prediction methods. Finally, we release an open-source package for ERT as well as previous conditional coverage metrics. Together, these contributions provide a new lens for understanding, diagnosing, and improving the conditional reliability of predictive systems.

2. 

   Multivariate Standardized Residuals for Conformal Prediction

   Sacha Braun, Eugène Berta, Michael I Jordan, and Francis Bach

   arXiv preprint arXiv:2507.20941, 2025

   Abs PDF Code

   While split conformal prediction guarantees marginal coverage, approaching the stronger property of conditional coverage is essential for reliable uncertainty quantification. Naive conformal scores, however, suffer from poor conditional coverage in heteroskedastic settings. In univariate regression, this is commonly addressed by normalizing nonconformity scores using estimated local score variance. In this work, we propose a natural extension of this normalization to the multivariate setting, effectively whitening the residuals to decouple output correlations and standardize local variance. We demonstrate that using the Mahalanobis distance induced by a learned local covariance as a nonconformity score provides a closed-form, computationally efficient mechanism for capturing inter-output correlations and heteroskedasticity, avoiding the expensive sampling required by previous methods based on cumulative distribution functions. This structure unlocks several practical extensions, including the handling of missing output values, the refinement of conformal sets when partial information is revealed, and the construction of valid conformal sets for transformations of the output. Finally, we provide extensive empirical evidence on both synthetic and real-world datasets showing that our approach yields conformal sets that significantly improve upon the conditional coverage of existing multivariate baselines.

3. 

   Minimum volume conformal sets for multivariate regression

   Sacha Braun, Liviu Aolaritei, Michael I Jordan, and Francis Bach

   arXiv preprint arXiv:2503.19068, 2025

   Abs PDF Code Poster

   Conformal prediction provides a principled framework for constructing predictive sets with finite- sample validity. While much of the focus has been on univariate response variables, existing mul- tivariate methods either impose rigid geometric assumptions or rely on flexible but computation- ally expensive approaches that do not explicitly optimize prediction set volume. We propose an optimization-driven framework based on a novel loss function that directly learns minimum-volume covering sets while ensuring valid coverage. This formulation naturally induces a new nonconfor- mity score for conformal prediction, which adapts to the residual distribution and covariates. Our approach optimizes over prediction sets defined by arbitrary norm balls—including single and multi- norm formulations. Additionally, by jointly optimizing both the predictive model and predictive uncertainty, we obtain prediction sets that are tight, informative, and computationally efficient, as demonstrated in our experiments on real-world datasets.