Conformal Prediction Explained with Python, Time-Series Forecasting, and Four Experiments

What happens when a model promises 90% coverage, the data changes, and the intervals stop behaving as expected?

Imagine you've built a machine learning model that predicts tomorrow's electricity consumption.

Your model says:

Tomorrow's predicted average consumption: 1.5 kW.

That sounds useful. But someone asks:

"How sure are you about that prediction?"

You could show the model's accuracy metrics, but those don't directly tell you how far tomorrow's actual consumption might be from 1.5 kW.

Instead, imagine the model produces:

Prediction: 1.5 kW
90% prediction interval: [1.1, 1.9] kW

Now we have more than a single number. But what does that 90% mean? Does it mean 90 out of every 100 consecutive predictions must be covered? What happens when the data changes?

These questions led me to explore Conformal Prediction, a framework for constructing prediction intervals from calibration errors.

I compared three approaches:

• Fixed Conformal Prediction: Calculate an interval adjustment once and keep it fixed.
• Rolling Conformal Prediction: Recalculate the adjustment using recent errors.
• Adaptive Conformal Inference (ACI): Adjust the miscoverage parameter using feedback from previous predictions.

I tested them in four experiments, including synthetic distribution shifts and real household electricity data.

One result stood out: ACI achieved 90.49% coverage over the full real-world test period, but only 78.33% over the final 60 observations. Both numbers are correct. Understanding how that can happen is the heart of this article.

1. The Only ML Basics We Need

In regression, a machine learning model predicts a numerical value. Here is a small illustrative example:

Understanding prediction errors

Every machine learning model makes mistakes. Suppose the model predicts 1.5 kW, but actual consumption is 1.8 kW. The model underestimated consumption by 0.3 kW. That difference is the prediction error:

Prediction error = Actual value − Predicted value

Here, the actual value is the actual value, the predicted value is the prediction, and the observation number identifies the observation.

When the model underestimates:

Prediction error = 1.8 − 1.5 = +0.3 kW

When it overestimates:

Prediction error = 1.5 − 1.8 = −0.3 kW

For conformal prediction, we often care about the size of the mistake, regardless of direction. That's the absolute error:

Absolute error = |Actual value − Predicted value|

For example, |1.5 − 1.8| = 0.3 kW.

import numpy as np

actual = np.array([1.5, 1.8, 1.2, 2.0, 1.6])
predicted = np.array([1.4, 1.6, 1.5, 1.7, 1.7])

absolute_errors = np.abs(actual - predicted)
print("Absolute errors:", absolute_errors)
print("MAE:", absolute_errors.mean())

Output:

Absolute errors: [0.1 0.2 0.3 0.3 0.1]
MAE: 0.2

MAE (Mean Absolute Error) measures the average size of the model's mistakes. An MAE of 0.2 kW does not mean every future error will be 0.2 kW or smaller.

What makes time-series forecasting different?

A time series is a sequence of observations ordered by time. To predict tomorrow's electricity usage, we use information available before tomorrow. Training on future observations and pretending they were available in the past would create data leakage.

That's why our experiments use chronological training, calibration, and testing periods.

2. What Does a 90% Prediction Interval Actually Mean?

Suppose we produce 100 prediction intervals and then observe the actual outcomes. If 90 intervals contain their corresponding actual values, the empirical coverage is:

Coverage = (90 ÷ 100) × 100 = 90%

A 90% target means we want our procedure to cover future outcomes at approximately that rate under the relevant conditions. But target coverage, observed coverage, and a theoretical guarantee are different things.

Even if overall coverage is 90%, every block of 100 consecutive predictions need not achieve exactly 90%. A marginal coverage guarantee also doesn't imply 90% conditional coverage for every individual case.

Coverage isn't everything

Extremely wide intervals can cover almost everything while providing little useful information. An interval of [0, 100] kW might contain tomorrow's value, but it isn't a helpful forecast for most households.

We care about both:

• Coverage: How often intervals contain actual outcomes.
• Sharpness: How narrow and informative the intervals are.

3. Conformal Prediction: The Intuition

Suppose we already have a trained regression model. We can examine its mistakes on a separate calibration dataset, then use those mistakes to choose an interval width for future predictions.

Step 1: Split the data chronologically

PAST                                                   FUTURE
|-------------------|-------------------|------------------|
       TRAINING            CALIBRATION           TESTING
   Fit the model        Measure errors       Evaluate intervals

The calibration observations were not used to fit the forecasting model. Using training errors could make the model appear unrealistically accurate.

Step 2: Calculate nonconformity scores

For ordinary symmetric regression intervals, a nonconformity score can simply be absolute error:

Score for observation i = |Actual − Predicted|

Suppose five calibration errors are:

[0.1, 0.2, 0.3, 0.4, 0.5]

Step 3: Choose target coverage

For this small teaching example, choose 80% coverage. The desired miscoverage rate is:

alpha = 1 − 0.8 = 0.2

Step 4: Find the conformal quantile

The standard split conformal rank is:

k = round up [(n + 1) × (1 − alpha)]

Here n is the number of calibration scores, alpha is target miscoverage, and rounding upward means round up.

With five scores:

k = round up (6 × 0.8) = 5

So we choose the fifth-smallest error: q = 0.5.

Step 5: Construct the interval

If the model predicts 1.5 kW:

Prediction interval = [prediction − q, prediction + q] = [1.0, 2.0] kW

We didn't need to assume normally distributed errors, but conformal prediction still depends on statistical assumptions for its classical guarantee.

4. Implementing Conformal Prediction in Python

import numpy as np

def conformal_quantile(scores, alpha):
    scores = np.sort(np.asarray(scores, dtype=float))
    if len(scores) == 0:
        raise ValueError("Calibration scores cannot be empty")
    if not 0 < alpha < 1:
        raise ValueError("alpha must be between 0 and 1")

    n = len(scores)
    k = int(np.ceil((n + 1) * (1 - alpha)))
    if k > n:
        return np.inf
    return scores[k - 1]

calibration_errors = [0.1, 0.2, 0.3, 0.4, 0.5]
q = conformal_quantile(calibration_errors, alpha=0.2)
prediction = 1.5

print(f"Prediction: {prediction:.2f} kW")
print(f"Interval: [{prediction-q:.2f}, {prediction+q:.2f}] kW")

Output:

Prediction: 1.50 kW
Interval: [1.00, 2.00] kW

Why can the quantile become infinite?

With only five calibration scores and a 90% target:

k = round up (6 × 0.9) = 6

But only five scores exist. The conservative finite-sample construction uses an infinite quantile rather than pretending a sixth observed score exists.

print(conformal_quantile(calibration_errors, alpha=0.1))
# inf

This gives an unbounded interval. A method can be conservative in theory while producing intervals that are unusable in practice.

5. Does Conformal Prediction Guarantee 90% Coverage?

Classical split conformal prediction has a finite-sample marginal coverage guarantee when its assumptions hold, including exchangeability of calibration and future observations and separate fitting of the forecasting model:

Probability that a new outcome is inside its interval ≥ 1 − alpha

Exchangeability means the joint distribution doesn't change when the relevant observations are reordered. Independent, identically distributed observations are a familiar example.

Time series may violate exchangeability because observations depend on earlier values and the distribution can change over time. Therefore, classical split conformal guarantees do not automatically carry over to dependent, nonstationary time series.

That's why empirical testing matters.

6. Three Ways to Construct Conformal Intervals

Method 1: Fixed Conformal Prediction

Calculate q on calibration data and leave it unchanged during testing. If q = 0.4, a forecast of 1.5 gives [1.1, 1.9], and a forecast of 2.0 gives [1.6, 2.4]. The center moves; the width stays fixed.

def fixed_interval(prediction, calibration_scores, alpha):
    q = conformal_quantile(calibration_scores, alpha)
    return prediction - q, prediction + q

Strength: Simple and stable. Weakness: Doesn't respond to changing error distributions.

Method 2: Rolling Conformal Prediction

Keep only a recent window of observed errors, such as the latest 100. Recalculate q as new outcomes arrive.

from collections import deque

class RollingConformal:
    def __init__(self, scores, alpha=0.1, window=100):
        self.scores = deque(scores[-window:], maxlen=window)
        self.alpha = alpha

    def predict_interval(self, prediction):
        q = conformal_quantile(list(self.scores), self.alpha)
        return prediction - q, prediction + q

    def update(self, actual, prediction):
        self.scores.append(abs(actual - prediction))

Crucial: Construct the interval before using the actual outcome. Update only after the outcome becomes available.

Strength: Can respond to recent changes. Weakness: Window size affects responsiveness and stability.

Method 3: Adaptive Conformal Inference (ACI)

ACI adjusts an internal miscoverage parameter based on whether previous intervals covered their outcomes:

next alpha = current alpha + gamma × (target alpha − miss indicator)

Where:

• alpha is target miscoverage (0.1 for 90% target coverage).
• adaptive alpha is the adaptive miscoverage parameter.
• gamma controls how quickly it changes.
• miss indicator = 1 for an interval miss and 0 for a hit.

A miss decreases adaptive alpha, generally leading to a more conservative next interval. A hit increases adaptive alpha, generally allowing a narrower interval.

class AdaptiveConformal:
    def __init__(self, target_alpha=0.1, gamma=0.005):
        self.target_alpha = target_alpha
        self.alpha_t = target_alpha
        self.gamma = gamma

    def update(self, missed):
        self.alpha_t += self.gamma * (
            self.target_alpha - int(missed)
        )

This snippet illustrates only the update rule, not a full production ACI implementation. A complete method also uses the adaptive alpha to choose quantiles and must handle out-of-range values and potentially infinite intervals.

Does adaptation actually help? Let's test it.

Experiment 1: What Happens When the Data Suddenly Changes?

We generated a synthetic autoregressive series:

Current value = 0.65 × previous value + volatility × random noise

The current observation partly depends on the previous one plus random variation. The variation scale changed from volatility = 1 to volatility = 3 at time step 2,200. This deliberately makes the series more volatile.

We kept a Ridge Regression forecasting model frozen and compared the three interval methods, targeting 90% coverage.

Figure 1. Fixed calibration fails to widen after the variance shift; rolling and adaptive methods react with wider intervals.

Fixed coverage collapsed to 39.50% because its width stayed unchanged despite larger errors. Rolling and ACI widened their intervals, with ACI reaching 89.88% post-shift coverage at the cost of greater width.

Lesson: A calibration interval that worked yesterday may be too narrow after a distribution shift.

Experiment 2: Does the Result Hold Across Multiple Random Seeds?

One random run could be misleading, so we repeated the synthetic experiment with 12 seeds (40–51), evaluating 800 pre-shift and 800 post-shift observations per seed.

Average post-shift results

The pattern remained consistent across seeds. But the choice of rolling-window size mattered:

A shorter window reacted more quickly to the sudden change in this experiment, though shorter windows can also be more sensitive to noise.

We also varied ACI's learning rate gamma. More aggressive settings could lead to infinite mean interval widths when the adaptive quantile rank exceeded the available calibration sample.

Figure 2. The results persist across seeds, but better coverage can come with wider or even unbounded intervals.

Lesson: Coverage alone doesn't tell us whether an interval method is practically useful.

Experiment 3: Conformal Prediction on Real Electricity Consumption

We next used the UCI Individual Household Electric Power Consumption dataset:

https://archive.ics.uci.edu/dataset/235/individual+household+electric+power+consumption

We aggregated Global_active_power into daily average active power (kW) and predicted the next day's average using Ridge Regression.

Our features included previous consumption at lags of 1, 2, 7, 14, and 28 days, plus cyclical day-of-week and month encodings.

import numpy as np

day_of_week = 6  # Sunday, when Monday is 0
dow_sin = np.sin(2 * np.pi * day_of_week / 7)
dow_cos = np.cos(2 * np.pi * day_of_week / 7)

Sine and cosine represent the weekly cycle without treating Sunday and Monday as far apart.

Chronological data split

The test period ran from January 4, 2010 to November 26, 2010. We used a rolling window of 100 and ACI gamma = 0.005.

Results

There was no universal winner. Fixed intervals were narrowest. ACI had the lowest mean interval score, but differences were small, and both ACI and Rolling had identical overall coverage.

What is interval score?

Interval score rewards narrow intervals and penalizes observations outside the bounds:

Interval score = interval width + penalty for falling below lower bound + penalty for exceeding upper bound

In plain language: start with interval width, then add a penalty for each miss. Lower is better.

def interval_score(actual, lower, upper, alpha=0.1):
    width = upper - lower
    below = max(lower - actual, 0)
    above = max(actual - upper, 0)
    return width + (2 / alpha) * (below + above)

Figure 3. The three methods approach 90% overall coverage but differ in sharpness and interval score.

Figure 4. The real-world ACI intervals compared with observed daily average power.

At first glance, ACI looked promising. But rolling coverage began deteriorating toward the end of the test period.

That led to Experiment 4.

Experiment 4: The Hidden Problem Behind 90% Overall Coverage

ACI achieved 90.49% overall coverage, yet the end of the test period looked less reliable.

We performed a post-hoc diagnostic analysis of the existing predictions. We did not refit or retune the models. We examined rolling coverage, widths, adaptive alpha, and errors.

Overall vs. local coverage

For ACI, 13 of the last 60 observations fell outside their intervals. That's 78.33% coverage for that window despite 90.49% coverage overall.

Figure 5. Rolling coverage exposes undercoverage that an aggregate metric can hide.

What changed for ACI?

Three things stood out:

1. Average point prediction error improved: MAE fell from 0.2399 to 0.1979 kW.
2. Intervals narrowed sharply: Mean width fell from 0.9745 to 0.5771 kW, roughly 41%.
3. Adaptive alpha increased: Its mean rose from 0.0981 to 0.1236, consistent with less conservative quantiles.

These observations are consistent with intervals becoming too narrow for some remaining errors. They do not, by themselves, prove a causal mechanism. Lower average error also doesn't rule out occasional large errors.

Figure 6. Adaptive parameters and interval widths evolve, but some observations still escape the intervals.

This matters operationally. A dashboard displaying only 90.49% overall coverage could conceal recent undercoverage.

One caution: 60-observation coverage estimates are noisy, and overlapping rolling windows are not independent. Local coverage plots are diagnostics, not a guarantee of conditional coverage at every point.

7. What Should We Monitor in Production?

Our experiments suggest monitoring:

• Point forecast error (MAE/RMSE): Is the forecasting model itself deteriorating?
• Overall interval coverage: How often do intervals contain actual observations?
• Rolling coverage: Are recent periods undercovered?
• Interval width: Are intervals becoming too narrow or too broad?
• Interval score: Are we balancing sharpness with misses?
• Infinite interval rate: Are any intervals unbounded or unusable?
• Adaptive alpha: How is ACI changing its calibration level?
• Feedback delay: Are updates applied only after actual outcomes arrive?

A one-day-ahead forecast and a 90-day-ahead forecast have very different feedback schedules. A production implementation must respect those delays.

8. Five Things I Learned

1. Accuracy isn't the same as uncertainty reliability. Low MAE doesn't automatically mean well-calibrated intervals.

2. Distribution shift can break fixed calibration. Synthetic post-shift fixed coverage fell to 39.50%.

3. Adaptation has costs. Rolling and adaptive intervals widened, and some aggressive ACI settings produced infinite intervals.

4. Complexity doesn't guarantee superiority. On real electricity data, fixed conformal intervals were narrower while still achieving coverage near the target.

5. Aggregate coverage can hide local failures. ACI's 90.49% overall coverage coexisted with 78.33% coverage in the last 60 observations.

9. Glossary: Conformal Prediction Without the Jargon

10. Reproducing the Experiments

The real-data experiments used the UCI household electricity dataset:

https://archive.ics.uci.edu/dataset/235/individual+household+electric+power+consumption

From the experiment project directory:

python src/experiment03.py --download
python src/experiment04.py
python -m pytest -q

The first command runs the real-world electricity experiment, downloading data if needed. The second diagnoses its existing predictions without refitting. The third runs the automated tests. In the completed run, 12 tests passed.

The results are specific to these datasets, settings, model, and evaluation windows. They are not universal rankings of conformal prediction methods.

Final Thoughts: A 90% Promise Is Only as Useful as Its Assumptions

Conformal prediction gives us a principled way to build prediction intervals using calibration errors. But producing an interval is only the beginning.

We also need to ask whether the assumptions hold, whether intervals respond to changes, whether they remain practically useful, and whether coverage is stable over recent windows.

The synthetic experiments showed how badly fixed intervals can fail under a sudden variance shift. The real-world experiment showed that simpler methods can remain competitive. And our final diagnostic analysis exposed the key lesson:

90% overall coverage doesn't mean 90% coverage everywhere, all the time.

The important question isn't only:

"What will happen tomorrow?"

It's also:

"How much should we trust what our model says about tomorrow — and will that trust still be justified when the world changes?"

If you're working with forecasting models, try evaluating rolling coverage alongside overall coverage. You may discover patterns that a single aggregate metric never reveals.