Adaptive Conformal Inference (ACI) provides finite-sample coverage guarantees, enhancing the prediction reliability under non-exchangeability. This study demonstrates that these desirable properties of ACI do not require the use of Conformal Predictors (CP). We show that the guarantees hold for the broader class of confidence predictors, defined by the requirement of producing nested prediction sets, a property we argue is essential for meaningful confidence statements. We empirically investigate the performance of Non-Conformal Confidence Predictors (NCCP) against CP when used with ACI on non-exchangeable data. In online settings, the NCCP offers significant computational advantages while maintaining a comparable predictive efficiency. In batch settings, inductive NCCP (INCCP) can outperform inductive CP (ICP) by utilising the full training dataset without requiring a separate calibration set, leading to improved efficiency, particularly when the data are limited. Although these initial results highlight NCCP as a theoretically sound and practically effective alternative to CP for uncertainty quantification with ACI in non-exchangeable scenarios, further empirical studies are warranted across diverse datasets and predictors.

Time-series forecasting is crucial for data-driven decisions across finance, healthcare, and environmental monitoring. Despite technological advances, identifying significant temporal segments impacting predictions remains challenging. We introduce ConformaSegment, a model-agnostic explainability framework that enhances time-series interpretability by identifying critical segments while quantifying prediction uncertainty. The framework integrates conformal prediction to generate reliable prediction intervals with guaranteed coverage rates, enabling users to understand which temporal segments most significantly influence forecasting outcomes. Our approach was validated across diverse real-world datasets using LSTM, RNN, and GRU models, demonstrating substantial performance improvements over existing techniques such as Saliency Maps and Integrated Gradients. ConformaSegment achieved mean R2 improvements of 42% and 18% respectively over these methods, while enhancing prediction interval coverage by 25.73% and 40.15%. These results demonstrate that ConformaSegment effectively identifies critical time segments in forecasting tasks, improving both interpretability and uncertainty quantification, thus enhancing model trustworthiness for applications in healthcare, industrial maintenance, and other time-sensitive domains. 

This paper introduces Fast Calibrated Explanations, an extension of an existing explanation method, Calibrated Explanations, designed for generating rapid, uncertainty-aware explanations for machine learning models. By incorporating perturbation techniques from ConformaSight, a global explanation method, into the core elements of Calibrated Explanations, we achieved significant speedups. These core elements include local feature importance with calibrated predictions, both of which retain uncertainty quantification. While the extension sacrifices some degree of detail, it excels in computational efficiency, making it ideal for high-stakes, real-time applications. Fast Calibrated Explanations applies to probabilistic explanations in classification and thresholded regression tasks, providing the probability of a target being above or below a user-defined threshold. This approach maintains the versatility of Calibrated Explanations for both classification and thresholded regression, making it suitable for a range of predictive tasks where uncertainty quantification is crucial.

Artificial Intelligence (AI) methods are an integral part of modern decision support systems. The best-performing predictive models used in AI-based decision support systems lack transparency. Explainable Artificial Intelligence (XAI) aims to create AI systems that can explain their rationale to human users. Local explanations in XAI can provide information about the causes of individual predictions in terms of feature importance. However, a critical drawback of existing local explanation methods is their inability to quantify the uncertainty associated with a feature's importance. This paper introduces an extension of a feature importance explanation method, Calibrated Explanations, previously only supporting classification, with support for standard regression and probabilistic regression, i.e., the probability that the target is below an arbitrary threshold. The extension for regression keeps all the benefits of Calibrated Explanations, such as calibration of the prediction from the underlying model with confidence intervals, uncertainty quantification of feature importance, and allows both factual and counterfactual explanations. Calibrated Explanations for regression provides fast, reliable, stable, and robust explanations. Calibrated Explanations for probabilistic regression provides an entirely new way of creating probabilistic explanations from any ordinary regression model, allowing dynamic selection of thresholds. The method is model agnostic with easily understood conditional rules. An implementation in Python is freely available on GitHub and for installation using both pip and conda, making the results in this paper easily replicable.

Machine learning (ML) models always make a prediction, even when they are likely to be wrong. This causes problems in practical applications, as we do not know if we should trust a prediction. ML with reject option addresses this issue by abstaining from making a prediction if it is likely to be incorrect. In this work, we formalise the approach to ML with reject option in binary classification, deriving theoretical guarantees on the resulting error rate. This is achieved through conformal prediction (CP), which produce prediction sets with distribution-free validity guarantees. In binary classification, CP can output prediction sets containing exactly one, two or no labels. By accepting only the singleton predictions, we turn CP into a binary classifier with reject option . Here, CP is formally put in the framework of predicting with reject option. We state and prove the resulting error rate, and give finite sample estimates. Numerical examples provide illustrations of derived error rate through several different conformal prediction settings, ranging from full conformal prediction to offline batch inductive conformal prediction. The former has a direct link to sharp validity guarantees, whereas the latter is more fuzzy in terms of validity guarantees but can be used in practice. Error-reject curves illustrate the trade-off between error rate and reject rate, and can serve to aid a user to set an acceptable error rate or reject rate in practice.

In data-driven decision support, having access to reliable confidence measures for individual predictions is crucial. Machine learning algorithms can provide probabilistic predictions, but these are often poorly calibrated, resulting in misleading decision support. This study empirically evaluates a set of readily available state-of-the-art calibration techniques, including both scaling and binning approaches. Using four different underlying models, and in total 40 publicly available datasets, the results analyzed using rigorous statistical testing show that calibration is generally successful. Specifically, applying a post-hoc calibration will reduce both log losses and calibration errors, without significantly lowering the predictive accuracy. However, the choice of calibration technique should depend on both the underlying model and the size of the dataset, resulting in the following guidelines for calibration: (i) We recommend Venn-Abers for decision trees and naïve Bayes (ii) Beta calibration for Extreme Gradient Boosting (XGB), (iii) Platt scaling for small datasets and Venn-Abers for larger ones when using random forests.

Conformal prediction (CP) has gained increasing attention in machine learning owing to its ability to provide reliable prediction sets with well-calibrated uncertainty estimates. While most existing CP implementations focus on inductive conformal prediction (ICP), full conformal prediction—also known as online or transductive CP—offers the strongest validity guarantees but has been largely absent from open-source software due to its computational complexity. In this paper, we introduce online-cp, a Python package designed for online conformal prediction, conformal predictive systems (CPS), and conformal test martingales. The package implements several online CP algorithms, enabling efficient and principled uncertainty quantification in streaming data scenarios. Additionally, it includes tools for testing the exchangeability assumption by using conformal test martingales. We demonstrate the functionality of online-cp through classification and regression examples as well as applications to predictive systems and exchangeability testing. By making online CP methods accessible, online-cp provides a foundation for the broader adoption and further development of conformal prediction in real-time machine learning applications. 

Calibrated Explanations is a recently proposed feature importance explanation method providing uncertainty quantification. It utilises Venn-Abers to generate well-calibrated factual and counterfactual explanations for binary classification. In this paper, we extend the method to support multi-class classification. The paper includes an evaluation illustrating the calibration quality of the selected multi-class calibration approach, as well as a demonstration of how the explanations can help determine which explanations to trust.