Ethical issues aside, should you be honest when asked how certain you are about some belief? Of course,\u00a0it depends<\/em>. In this blog post, you\u2019ll learn on what.<\/strong><\/p>\n David Spiegelhalter\u2019s new (as of 2025) fantastic book, \u201cThe Art of Uncertainty<\/em>\u201d \u2013 a must-read for everyone who deals with probabilities and their communication \u2013 features a short section on scoring rules. Spiegelhalter walks the reader through the quadratic scoring rule, and briefly mentions that a linear scoring rule will lead to dishonest behavior. I elaborate on that interesting point in this blog post.<\/p>\n Let\u2019s set the stage: Just like in so many other scenarios and paradoxes, you find yourself in a TV show (yes, what an old-fashioned way to start). You have the opportunity to answer questions on common knowledge and win some cash. You are asked yes\/no-questions that are expressed in a binary fashion, such as:\u00a0Is the area of France larger than the area of Spain? Was Marie Curie born earlier than Albert Einstein? Is Montreal\u2019s population larger than Kyoto\u2019s?<\/em><\/p>\n Depending on your background, these questions might be obvious for you, or they might be difficult. In any case, you will have a subjective \u201cbest guess<\/em>\u201d in mind, and some degree of certainty. For example, I feel comfortable answering the first, slightly less for the second, and I already forgot the answer to the third, even though I looked it up to build the example. You might experience a similar level of confidence, or a very different one. Degrees of certainty are, of course, subjective.<\/p>\n The twist of the quiz: You are not supposed to give a binary yes\/no-answer as in a multiple-choice test, but to honestly communicate your degree of conviction, that is, to produce the probability that you personally assign to the true answer being \u201cyes\u201d. The number 0 then means \u201cdefinitely not\u201d, 1 expresses \u201cdefinitely yes\u201d, and 0.5 reflects the degree of uncertainty corresponding to the toss of a fair coin \u2014 you then have absolutely no idea. Let\u2019s call\u00a0P(A)<\/em><\/strong>\u00a0your true subjective conviction that statement\u00a0A<\/em><\/strong>\u00a0is true. That probability can take any value between 0 and 1, whereas\u00a0A<\/em><\/strong>\u00a0is bound to be\u00a0either<\/em><\/strong>\u00a00 or 1. You can then communicate that number, but you don\u2019t have to, so we\u2019ll call\u00a0Q(A)<\/strong>\u00a0the probability that you eventually express in that quiz.<\/p>\n In general, not every probabilistic expression\u00a0Q<\/em><\/strong>\u00a0is met with the same excitement, because humans generally dislike uncertainty. We are much happier with the expert that gives us \u201c99.99%\u201d or \u201c0.01%\u201d probabilities for something to be or not to be the case, and we favor them considerably over the experts producing \u201c25%\u201d and \u201c75%\u201d\u00a0maybe-ish<\/em>\u00a0assessments. From a rational perspective, more informative probabilities (\u201csharp predictions\u201d, close to 0 or close to 1) are favorable over uninformative ones (\u201cunsharp predictions\u201d, close to 0.5). However, a modest but truthful prediction is still worth more than a bold but unreliable one that would make you go all-in. We should therefore ensure that people do not lie about their degree of conviction, so that really 99% of the \u201c99%-sure\u201d predictions are actually true, 12% or the \u201c12%-sure\u201d, and so on. How can the quiz master ensure that?<\/p>\n The most straightforward way that one might come up with to judge probabilistic statements is to use a linear scoring rule: In the best case, you are very confident and right, which means\u00a0Q(A)=P(A)<\/em>=1<\/strong>\u00a0and\u00a0A<\/strong>\u00a0is true, or\u00a0Q(A)=P(A)<\/em>=0<\/strong>\u00a0and\u00a0A<\/strong>\u00a0is false. We then add the score\u00a0+1=r(Q=1, A=1)=r(Q=0, A=0)<\/strong>\u00a0to the balance. In the worst case, you were very sure of yourself, but wrong; that is,\u00a0Q(A)=P(A)<\/em>=1<\/strong>\u00a0while\u00a0A<\/em><\/strong>\u00a0is false, or\u00a0Q(A)=P(A)<\/em>=0<\/strong>\u00a0while\u00a0A<\/em><\/strong>\u00a0is true. In that unfortunate case, we subtract \u20131=r(Q=1, A=0)=r(Q=0, A=1)<\/strong>\u00a0from the score. Between these extreme cases, we draw a straight line. When you express maximal uncertainty via\u00a0Q(A)<\/em>=0.5<\/strong>, we have\u00a00=r(Q=0.5, A=1)=r(Q=0.5, A=0)<\/strong>, and neither add nor subtract anything.<\/p>\n The functional form of this linear reward function is not particularly spectacular, but its visualization will come handy in the following:<\/p>\n No surprise here: If\u00a0A<\/em><\/strong>\u00a0is true, the best thing you could have done is to communicate \u201cQ=1<\/strong>\u201d, if\u00a0A<\/em><\/strong>\u00a0is false, the best strategy would have been to produce \u201cQ=0<\/strong>\u201d. That\u2019s what is visualized by the black dots: They point to the largest value that the reward function can attain for the particular value of the truth. That\u2019s a good start.<\/p>\n But you typically do\u00a0not<\/em><\/strong>\u00a0know with absolute certainty whether the answer is \u201cyes, A is true<\/em>\u201d or \u201cno, A is false<\/em>\u201d, you only have a subjective gut feeling. So what should you do? Should you just be honest and communicate your true belief, e.g.\u00a0P<\/em>=0.7<\/strong>\u00a0or\u00a0P<\/em>=0.1<\/strong>?<\/p>\n Let\u2019s set ethics aside, and consider the reward that we want to maximize. It then turns out that you should not be honest.\u00a0When evaluated via the linear scoring rule, you should lie, and communicate Q(A)<\/em>=0 when P(A)<\/em><0.5 and Q(A)<\/em>=1 when P(A)<\/em>>0.5.<\/strong><\/p>\n To see this surprising result, let\u2019s compute the\u00a0expectation value<\/em><\/strong>\u00a0of the reward function, assuming that your belief is, on average, correct (cognitive psychology teaches us that this is an unrealistically optimistic assumption in the first place, we\u2019ll come back to that below). That is, we assume that in about 70% of the cases when you say\u00a0P<\/em>=0.7<\/strong>, the true answer is \u201cyes, A is true<\/em>\u201d, in about 75% of the cases when you say\u00a0P<\/em>=0.25<\/strong>, the true answer is \u201cno, A is false<\/em>\u201d. The expected reward\u00a0R(P, Q)<\/em><\/strong>\u00a0is then a function of both the\u00a0honest subjective\u00a0<\/em><\/strong>probability\u00a0P<\/em><\/strong>\u00a0and of the\u00a0communicated<\/em><\/strong>\u00a0probability\u00a0Q,\u00a0<\/em><\/strong>namely the weighted sum of the reward\u00a0r(Q, A=1)\u00a0<\/em><\/strong>and\u00a0r(Q, A=0)<\/em><\/strong>:<\/p>\n R(P, Q) = P * r(Q, A=1) + (1-P) * r(Q, A=0)<\/em><\/strong><\/p>\n Here come the resulting\u00a0R(P,Q)<\/em><\/strong>\u00a0for four different values of the honest subjective probability\u00a0P<\/em><\/strong>:<\/p>\n The maximally attainable reward on the long term is not always 1 anymore, but it\u2019s bounded by\u00a02|P-<\/em>0.5| \u2014\u00a0<\/em><\/strong>ignorance comes at a cost. Clearly, the best strategy is to confidently communicate\u00a0Q=<\/em>1<\/strong>\u00a0as long as\u00a0P>0.5<\/strong>, and to communicate an equally confident\u00a0Q<\/em>=0<\/strong>\u00a0when\u00a0P<\/em><0.5 \u2014\u00a0<\/strong>see where the black dots lie in the figure.<\/p>\n Under a linear scoring rule, when it is more likely than not that the event occurs \u2014 pretend you are absolutely certain that it will occur. When it\u2019s marginally more likely that it does not occur \u2014 be bold and proclaim \u201cthat can never happen\u201d. You will be wrong sometimes, but, on average, it\u2019s more profitable to be bold than to be honest.<\/p>\n<\/blockquote>\n Even worse: What happens when you have absolutely no clue, no idea about the outcome, and your subjective belief is\u00a0P<\/em><\/strong>=0.5? Then you can play safe and communicate that, or you can take the chance and communicate\u00a0Q=1<\/strong>\u00a0or\u00a0Q=0<\/strong>\u00a0\u2014 the expectation value is the same.<\/p>\n If find this a disturbing result: A linear reward function makes people go all-in! There is no way as forecast consumer to distinguish a slight tendency of 51% from a \u201cquite likely\u201d conviction of 95% or from an almost-certain 99.9999999%. In that quiz, the smart players will always go all-in.<\/p>\n Worse, many situations in life reward unsupported confidence more than thoughtful and careful assessments. Cautiously said, not many people are being heavily sanctioned for making clearly exaggerated claims\u2026<\/p>\n A quiz show is one thing, but, obviously, it\u2019s quite a problem when people (or machines\u2026) are pushed to not communicate their true degree of conviction when it comes to estimating the risk of serious and dramatic events such as earthquakes, war and catastrophes.<\/p>\n How can we make them to be honest (in the case of people) or\u00a0calibrated<\/a>\u00a0(in the case of machines)?<\/p>\n If the probability for something to happen is estimated to be\u00a0P<\/strong>=55% by some expert, I want that expert to communicate\u00a0Q<\/strong>=55%, and not\u00a0Q<\/strong>=100%. For probabilities to have any value for our decisions, they should reflect the true level of conviction, and not an opportunistically optimized value.<\/p>\n This reasonable ask has been formalized by statisticians by\u00a0proper<\/em><\/strong>\u00a0scoring rules: A proper scoring rule is one that incentivizes the forecaster to communicate their true degree of conviction, it is maximized when the communicated probabilities are calibrated, i.e. when predicted events are realized with the predicted frequency. At first, the question might arise whether such a scoring rule can exist at all. Thankfully, it can!<\/p>\n One proper scoring rule is the\u00a0quadratic scoring rule<\/em><\/strong>, also known as the\u00a0Brier score<\/em><\/strong>. For extreme communicated probabilities (Q<\/strong>=1,\u00a0Q<\/strong>=0), the values are the very same as for the linear scoring rule, but we don\u2019t draw straight line between these, but a parabola. By doing that, we reward honest ignorance: +0.5 is awarded for a communicated probability of\u00a0Q<\/strong>=0.5.<\/p>\n This reward function is asymmetric: When you increase your confidence from\u00a0Q<\/em><\/strong>=0.95 to\u00a0Q<\/em><\/strong>=0.98 (and\u00a0A<\/em><\/strong>\u00a0is true), the reward function only increases marginally. On the other hand, when\u00a0A<\/strong>\u00a0is false, that same increase of confidence leaning towards the wrong outcome is pushing down the reward considerably. Clearly, the quadratic reward thereby nudges one to be more cautious than the linear reward. But will it suffice to make people honest?<\/p>\n To see that, let\u2019s compute the expectation value of the quadratic reward as a function of both the true honest probability\u00a0P<\/em><\/strong>\u00a0and the communicated one\u00a0Q,\u00a0<\/em><\/strong>just like we did in the linear case:<\/p>\n R(P, Q) = P * r(Q, A=1) + (1-P) * r(Q, A=0)<\/em><\/strong><\/p>\n The resulting expected reward, for different values of the honest probability P, is shown in the next figure:<\/p>\n Now, the maxima of the curves lie exactly at the point for which\u00a0Q=P<\/em><\/strong>, which makes the correct strategy communicating honestly one\u2019s own probability\u00a0P<\/em><\/strong>. Both exaggerated confidence and excessive caution are penalized. Of course, by knowing more in the first place, you\u2019ll be able to make sharper and more confident statements (more predictions\u00a0Q=P<\/strong>\u00a0that are either close to 1 or close to 0). But honest ignorance is now rewarded with +0.5. Better be safe than sorry.<\/p>\n What do we learn from that? The reward that is maximized by honestly communicated probabilities sanctions \u201csurprises\u201d (Q<\/strong><0.5 and the event is actually true, or\u00a0Q<\/strong>>0.5 and the event is actually false) quite strongly. You lose more when you are wrong with your tendency (Q<\/strong>>0.5 or\u00a0Q<\/strong><0.5) than you would win when you are correct. At the same time, not knowing and being honest about it is rewarded a non-negligible value.<\/p>\n The quadratic reward function is not the only one that rewards honesty (there are infinitely many proper scoring rules): The logarithmic reward penalizes being confidently wrong (P<\/strong>=0, but truth is \u201cyes<\/em>,\u00a0A<\/em><\/strong>\u00a0is true<\/em>\u201d;\u00a0P<\/strong>=1, yet truth is \u201cno,\u00a0<\/em>A<\/em><\/strong>\u00a0is false<\/em>\u201d) with an unassailable\u00a0-infinity<\/em><\/strong>: The score is simply the logarithm of the probability that had been predicted for the event that eventually occurred \u2014 the plot is cut off on the y-axis for that reason:<\/p>\n The logarithmic reward breaks the symmetry between \u201chaving communicated a slightly too-high<\/em>\u201d and \u201chaving expressed a slightly too-low<\/em>\u201d probability: Towards uninformative\u00a0Q<\/strong>=0.5, the penalty is weaker than towards informative\u00a0Q<\/strong>=0 or\u00a0Q<\/strong>=1, which we see in the expectation values:<\/p>\n The logarithmic scoring rule heavily penalizes the assignment of a probability of 0 to something that then very surprisingly happened: Somebody who has to admit \u201cI really though it was absolutely impossible<\/em>\u201d after the fact that they assigned\u00a0Q<\/em><\/strong>=0 won\u2019t be invited to provide predictions ever again\u2026<\/p>\n Scoring rules can push forecasters to be over-confident (see the linear scoring rule), they can be proper (see the quadratic and logarithmic scoring rules), but they can also punish \u201cbeing boldly wrong<\/em>\u201d so thoroughly that forecasters would rather pretend they don\u2019t know really even if they do. A\u00a0cubic scoring rule<\/em><\/strong>\u00a0would lead to such excessive caution:<\/p>\n The expectation values of the reward now make people rather communicate values that are less informative (closer to 0.5) than their true convictions: Instead of an honest\u00a0Q=P<\/strong>=0.2, the optimum is at\u00a0Q<\/strong>=0.333, instead of honest\u00a0Q=P<\/strong>=0.4, the optimum is\u00a0Q<\/strong>=0.4495.<\/p>\n In other words, to be provided honest judgements, don\u2019t exaggerate the punishment of strong but eventually wrong convictions either \u2014 otherwise you\u2019ll be surrounded by indecisive and hesitant cowards\u2026<\/p>\n The following plot recapitulates the argument by showing the optimal communicated probability\u00a0Q<\/strong>\u00a0as a function of the true belief\u00a0P<\/strong>. For a linear reward (Exponent 1), you will either communicate\u00a0Q<\/strong>=0 or\u00a0Q<\/strong>=1, and not disclose any information about your true degree of conviction. The quadratic reward (Exponent 2) makes you be honest (Q<\/strong>=P<\/strong>), while the cubic reward (Exponent 3) lets you set overly cautious\u00a0Q<\/strong>\u00a0values.<\/p>\n In reality, our choices are often binary, and, depending on the \u201cfalse positive\u201d and \u201cfalse negative\u201d cost and the \u201ctrue positive\u201d and \u201ctrue negative\u201d reward, we will set the threshold on our subjective probability to take or not take a certain action to different values. It is not at all irrational to plan thoroughly for a probability\u00a0P<\/strong>=0.01=1% catastrophe.<\/p>\n Scoring rules have two main applications: On a technical level, when training a probabilistic statistical or machine learning model on data, optimizing a proper scoring rule will yield calibrated and as-sharp-as-possible probabilistic forecasts. In a more informal setting, when several experts estimate the probability for something (typically dramatic) to happen, one wants to make sure that the experts are honest and don\u2019t try to overplay or downplay their subjective uncertainty (beware of group dynamics!). Super-forecasters indeed use quadratic scoring rules to help reflect on their degree of confidence and to train themselves to become more calibrated.<\/p>\n Back to our initial quiz game. Before answering, you should definitely ask how you are evaluated. The evaluation procedure does matter, even if you are told it does not. Similarly, when you are given a multiple-choice-test, be sure to understand whether it might be worthwhile to check a box even if you are only very marginally certain about its correctness.<\/p>\n But how can a quiz involving subjective probabilities be evaluated at all in an objective fashion? According to Bruno De Finetti, \u201cprobability does not exist\u201d<\/em>, so how can we then judge the probabilities that people express? We don\u2019t judge people\u2019s taste either! David Spiegelhalter emphasizes in \u201cThe Art of Uncertainty\u201d that uncertainty is not \u201ca property of the world, but of our relationship with the world<\/em>\u201d.<\/p>\n However,\u00a0subjective<\/em>\u00a0does not mean\u00a0unfalsifiable<\/em>.<\/p>\n I might be 99% sure that France is larger than Spain, 75% sure that Marie Curie was born before Albert Einstein, and 55% sure that Montreal is larger than Kyoto. The numbers that\u00a0you<\/em><\/strong>\u00a0assign to these statements will\u00a0probably<\/em>\u00a0(pun intended) be different. Your relationship to the world is a different one than mine. That\u2019s OK.<\/p>\n We can be both right in the sense that we express calibrated probabilities, even if we assign\u00a0different<\/strong>\u00a0probabilities to the\u00a0same<\/strong>\u00a0events.<\/p>\n<\/blockquote>\n A more commonplace setting: When I enter a supermarket, I can assign quite informative (quite high or quite low) probabilities to me buying certain products \u2014 I typically know well what I intend to shop. The data scientist working at the supermarket does not know my personal shopping list, even after having collected considerable personal data. The probability that they assign to me buying a bottle of orange juice will be quite different from the one that I assign to me doing that \u2014 both probabilities can be \u201ccorrect\u201d in the sense that they are calibrated on the long term.<\/p>\n Subjectivity does not mean arbitrariness: We can aggregate predictions and outcomes, and evaluate to which extent the predictions are calibrated. Scoring rules help us precisely with that task, because they simultaneously grade honesty and information: Each forecaster can be evaluated separately upon their predicted probabilities. The one that is most informed (producing close-to-1 and close-to-0 probabilities) while being honest at the same time will win the quiz. Different scoring rules can then rank strong-but-slightly-uncalibrated against weaker-but-calibrated predictions differently.<\/p>\n As mentioned above, honesty and\u00a0calibration<\/a>\u00a0are not equivalent in practice. We might truly believe 100 times that certain events should occur in 20% of each case \u2014 but the true number of occurrences might significantly differ from 20. We might be honest about our belief and express\u00a0P=Q<\/strong>, but that belief itself is typically uncalibrated! Kahneman and Tversky have studied the cognitive biases that typically make more confident than we should be. In a way, we often behave as if a linear scoring rule judged our predictions, making us lean towards the bold side.<\/a><\/p>\n The post Honestly Uncertain<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
A probabilistic quiz game<\/strong><\/h2>\n
The Linear Scoring Rule<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_KGSnNcCuRQh3Hyj7tzZU2w-1024x661.png?resize=1024%2C661&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_lGuw-ifVzawwDNnD2hQqVA-1024x607.png?resize=1024%2C607&ssl=1\")
\n
Punishing confident wrongness: The Quadratic Scoring Rule<\/strong><\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_v4pWRg7c_nHFZ1gAmtACHw-1024x638.png?resize=1024%2C638&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_yo7Fh6c2i2_BYNdc9IY0mw-1024x639.png?resize=1024%2C639&ssl=1\")
Logarithmic reward<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_lYjULjid_mp2PM_0bWaljQ-1024x635.png?resize=1024%2C635&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_mcwmCqDJSPljmLQtgs9jww-1024x653.png?resize=1024%2C653&ssl=1\")
Incentivizing sandbagging: The Cubic Scoring Rule<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_0AQnXHKFzBBlYX5qBZzulw-1024x645.png?resize=1024%2C645&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_yGbRreaKLdbTzLoR7RYrUw-1024x658.png?resize=1024%2C658&ssl=1\")
Honest and communicated probabilities<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/1_F2jEj8ARLZWXFScPRDOXFg-1024x641.png?resize=1024%2C641&ssl=1\")
If probabilities are subjective, how can they be \u201cwrong\u201d?<\/h2>\n
\n