How Much Should You Trust an AI's Confidence Score?

You have seen the number before. A tool returns an answer with something like 92% printed beside it, and it is hard not to read that as close to a guarantee. It looks precise. It carries a percent sign, the universal badge of a fact you can build on. But the figure is not a measurement of truth. It is a measurement of how strongly the model preferred its own answer to the other answers it weighed, which is a related but importantly different thing.

The gap between the model is confident and the model is right causes more misunderstanding than almost any other part of using these tools. It is worth taking apart properly, because the fix is not to dismiss every percentage on sight. It is to know what the percentage is actually reporting.

Most systems that print a percentage are reporting some version of the same quantity: the score the model assigned to its leading candidate once all the candidates have been squashed into a set of numbers that add up to one. If a model chooses between cat and dog and returns 90% for cat, it means its scoring favoured cat strongly over the alternatives it evaluated. It does not mean that nine times out of ten, in photographs resembling this one, the animal turns out to be a cat.

The distinction matters because that internal score is shaped entirely by patterns in the training data. Where the data was thin, skewed, or simply unrepresentative of the picture in front of it, the model can be emphatic and wrong at the same time. Nothing in the pipeline verifies the answer; the number describes the model's own preference, and the model has no second source to check itself against. The mechanics of how those preferences are formed are covered in how an image model reads a photograph.

There is a second, quieter problem. The score says nothing about how much evidence was available. A model can arrive at the same figure from one overwhelming clue or from a dozen faint ones that happened to agree. Read as an amount of evidence, the number will mislead you; read as a margin of victory between candidates, it behaves sensibly.

A well-calibrated model that says 80% is right roughly four times in five, across a long run. A poorly calibrated one might say 80% and land half the time, or say 60% and land nine times in ten. Calibration is a property of a system in aggregate, not a property of the single answer you are looking at, and this is the part that trips people up: there is no way to audit it from one interaction. A 92% from a beautifully calibrated model and a 92% from a wildly overconfident one look identical.

The field has a scoring rule for exactly this. The Brier score, proposed in 1950 for weather forecasting, rewards a forecaster for being both accurate and honest about uncertainty. It is the reason a forecast of rain tomorrow is a serious claim rather than a mood: forecasters are kept honest by decades of scored predictions. Image models generally are not scored that way in public, which is the single best reason to keep a percentage at arm's length.

Work on modern neural networks has repeatedly found them tilted toward overconfidence: the probability attached to the leading answer tends to run ahead of the observed hit rate, and the effect grows with model size. That does not make the number useless. It makes it a relative signal, best read against the same tool's other answers rather than as an absolute claim about reality.

• —Reading one score as certainty. A single 95% can still be wrong. It only says that answers of this kind are ones the model rarely doubts, which is a claim about the model's habits, not about your photograph.
• —Reading it as an amount of evidence. One decisive clue and ten weak but agreeing clues can produce the same figure. The score does not distinguish thin evidence from thick evidence, only how cleanly it pointed one way.
• —Comparing across tools. Two products showing 70% are not saying the same thing. Each number is calibrated, or miscalibrated, against its own training and its own scoring method.
• —Skipping the reasoning. The explanation behind a guess is almost always more diagnostic than the percentage. If the stated clues are things you can check in the photograph yourself, you can judge the answer directly.
• —Forgetting that the photograph sets the ceiling. Some pictures simply cannot support a precise answer, however good the model is, which is the subject of what makes a photo hard to geolocate.

Raven is a small, honest case study, because it puts its confidence on screen as a ring next to the guess. Upload a street scene in which a legible shopfront, a national road-sign shape and a distinctive kerb all agree, and the figure runs high: several independent clues converged on one region, and the model's leading candidate beat the field comfortably. Upload a bare beach at midday and it drops, because the evidence genuinely does fit a dozen coastlines at once.

Read that way, a low number is not a malfunction. It is the most useful thing the tool can tell you about a photograph: that the picture, not the model, is the limiting factor. The single most valuable line in a Raven result is usually not the percentage but the sentence beside it naming what the model noticed, which is why signage carries so much weight in the reasoning. A partially visible word in an unfamiliar alphabet can do more than the entire building behind it, as the way script on signs narrows a location sets out in detail.

It also helps to know what is not in the pipeline. Raven reads pixels; it does not read the file's location metadata, does not search a database of your pictures, and does not keep the image once the answer is returned. The full route from upload to answer is documented in how Raven works, and the reason a single model can weigh signage, plants and light together at all is the subject of multimodal AI explained simply.

1. 1.Start with the explanation. If the model names clues you can verify in the frame, you can grade the answer yourself instead of grading the number.
2. 2.Treat the figure as a ranking. It tells you the top candidate beat the others by some margin. Useful, but not the same claim as a nine-in-ten chance of being correct in the world.
3. 3.Ask what the photograph could support. An interior with no window cannot yield a city-level answer no matter what percentage appears beside it.
4. 4.Corroborate when it matters. If anything at all depends on the answer, find independent confirmation rather than trusting a self-reported score.
5. 5.Watch the same tool over many guesses. Calibration only becomes visible in aggregate, and after twenty or thirty tries you will have a genuine feel for what its 80% is worth.

The habit transfers cleanly. A spam filter that is 99% sure, a bank alert that is 80% sure and a transcription tool that flags a word as low confidence are all making the same kind of statement: this is where my scoring landed, given what I have seen before. None of them has checked the world. What separates the trustworthy ones is not the size of the number but whether anyone measures, over time, how often those numbers turn out to be right.

That is the whole of the skill, really. Confidence is a self-report. Take it as a useful signal about how tangled the evidence was, pair it with the reasoning, and keep your own judgement in the loop. Do that and a percentage becomes genuinely informative rather than quietly persuasive.

The most interesting experiment is a deliberate one: run the same scene twice, once wide and once cropped tight, and watch what happens to the ring. The subject has not changed. The evidence has, and an honest score should follow it down.