Your uncertainty metric might be ranking models backwards

When a model tells you how sure it is, you grade that confidence with a metric — log-likelihood, calibration error, something off a dashboard. This paper proves those metrics can rank your models backwards relative to what actually matters: the decisions they lead to.

A forecast is only as good as the decision it leads to

When a model makes a prediction, we increasingly want it to also say how sure it is — a probability, a confidence interval, an error bar. That is uncertainty quantification. And because a confident-but-wrong model is dangerous, we evaluate that uncertainty with various metrics.

The two most common are the negative log-likelihood — a proper scoring rule that rewards assigning high probability to what actually happens — and the expected calibration error, which checks whether, when a model says "seventy percent," things happen about seventy percent of the time. These metrics are everywhere. They decide which methods get published and deployed.

Here is the catch. The "true" uncertainty can never be observed directly — we only ever see outcomes — so these metrics are surrogates. They measure properties of the predicted distribution that seem like they should correlate with usefulness. The authors ask the sharp question: do they actually rank models the way real-world usefulness would? Their answer is a striking phrase. Current uncertainty benchmarks, they argue, silently optimise for the wrong downstream world.

Decision utility, and the Bayes act

In the real world, an uncertainty estimate feeds a decision. Should I approve this loan? Should I bid into the electricity market this hour? Should I flag this case for a human?

Each decision has costs and payoffs — and crucially, those costs are often asymmetric. A false alarm and a missed detection rarely cost the same. The right action, given a prediction, is the one that maximises expected utility — what decision theorists call the Bayes act. So the real test of an uncertainty estimate is not "is it pretty," it is "does acting on it lead to good decisions across the range of cost trade-offs I might face."

The paper's central contribution is to formalise this into a criterion they call decision-alignment. A metric is decision-aligned if it ranks models in exactly the same order that their expected decision utility would.

And then they prove something both elegant and damning: every standard metric, insofar as it is decision-aligned at all, secretly encodes an implicit prior — an assumption about which decision costs matter — and those implicit assumptions are, in their word, pathological.

Every metric carries a hidden bet about what matters; the honest move is to make that bet explicit, plausible, and aligned with the choice you actually have to make.

The hidden assumptions inside familiar metrics

This is the most illuminating part — what each beloved metric is quietly assuming when you let it rank your models.

• Accuracy. Ranking models by accuracy is equivalent to assuming the cost of a false positive exactly equals the cost of a false negative — a single, knife-edge assumption of perfectly symmetric costs. For most real decisions, that is just wrong.
• Negative log-likelihood. Its implicit prior places unbounded weight on the extreme regions — the situations where one type of error is essentially free and a trivial do-nothing policy already works — which is precisely where good uncertainty matters least. So NLL lavishes attention on the cases you do not care about.
• Brier score. Its implicit prior is uninformative — it weights every possible cost trade-off equally, which sounds fair but means it is not tuned to your actual decision at all.

The popular calibration metrics fare worst. The paper shows that expected and maximum calibration error are not decision-aligned in the first place, for a deep structural reason: their score for one prediction depends on all the other predictions in the batch, whereas a real per-case decision is judged on its own. A metric whose grade depends on the whole batch can never rank models the way a per-instance payoff does.

The honest alternative: make the bet explicit

The constructive answer is a family the authors call prior-weighted utility metrics. The idea is simple and honest. Instead of letting a metric smuggle in a hidden, pathological assumption about your decision costs, you make the assumption explicit and plausible.

You pick the decision you actually care about — a yes/no choice with known costs, a predict-or-abstain choice, a top-k selection — and a sensible prior over its parameters, and you build a metric that directly measures expected utility under that. By construction, it is decision-aligned. And reassuringly, they prove these metrics are still proper scoring rules, so they do not reward dishonest uncertainty.

The evidence is wonderfully concrete: bidding into a day-ahead electricity market, where you only bid when your forecast is confident enough — a real "gate the signal on uncertainty" problem. They rank a set of models by each metric, then check, using a rank correlation, whether that ranking matches the ranking by actual bidding profit.

The conventional metrics fail, and some fail spectacularly. The negative log-likelihood, the retention metrics, the error-detection metric — all hovered around zero correlation with realised bidding utility, meaning they carry essentially no information about which model will actually make you money. Worse, the calibration metrics — expected and maximum calibration error — had a negative correlation, around minus zero-point-two-four. Read that again: the calibration metrics ranked the models backwards relative to what actually paid off. The best-calibrated-looking model was, if anything, a worse bet.

The prior-weighted utility metrics, by contrast, had the strongest positive alignment with real profit. The authors also show this holds in credit-approval and peer-to-peer lending case studies with real economic payoffs, and that the approach is robust even when the chosen prior is fairly misspecified.

The honest caveats

The authors are careful about limits. No single one of their metrics covers all possible decisions — they explicitly recommend using a variety of them, matched to the decisions you face. The work focuses on what they call first-order uncertainty — a single predictive distribution — and leaves richer settings for future work.

The experiments are on modest, tabular datasets with a handful of models per task, so generality to large-scale deep learning is not demonstrated here, and a fair amount of the detailed numbers live in appendices. Building one of these metrics also requires you to actually think about your decision and commit to a plausible prior over its costs — more work than reaching for a generic number, but that is rather the point. This is a preprint.

Why it matters

Uncertainty estimates are increasingly the thing high-stakes systems lean on. In risk and trading, you gate signals on confidence — and this paper shows the metric you used to pick your uncertainty model might rank it backwards for your actual P&L. In insurance, the whole game is asymmetric costs, which accuracy and calibration metrics quietly assume away. In model governance and monitoring, teams build dashboards of calibration error and log-likelihood to decide which models to trust — and those dashboards may be systematically misleading when the goal is to support decisions.

We have a deep habit, in machine learning, of optimising the metric in front of us and assuming usefulness will follow. This paper is a rigorous demonstration that for uncertainty, that assumption can fail outright — that a model can look beautifully calibrated and lead you to worse decisions, and that the metric told you nothing, or told you the opposite of the truth. The remedy is to tie your evaluation to the decision, make your cost assumptions explicit, and grade the model on the utility it actually delivers. A calibrated-looking number is not the goal. A good decision is.