Is it practical and useful to distinguish aleatoric and epistemic uncertainty?


Philosophers have long emphasized a distinction between aleatoric and epistemic uncertainty. Epistemic refers to lack of knowledge -- something we could in principle know for sure -- in contrast to aleatoric "intrinsic randomness" involved in which of possible futures will actually occur. As a basic iconic example of the distinction between these two categories, consider whether the top card of a deck will be an Ace after I shuffle and look at it. The uncertainty here is aleatoric before I shuffle, but becomes epistemic after shuffling but before looking.

In Hacking's An Introduction to Probability and Inductive Logic, the brief chapter on philosophical interpretations of probability concludes with the following comments.

Our prototypical examples [of probability] are artificial randomizers. But as we start to think hard about more real-life examples, we get further and further away from the core examples. Then our examples tend to cluster into belief-type examples, and frequency-type examples, and in the end we develop ideas of two different kinds of probability.
Is this a useful way of thinking about probability? Consider our running theme of geopolitical forecasting, to assign probability (in June 2018) to events like This context -- mainly future decisions by individuals or groups -- doesn't seem to fit either kind. And I think that if you seriously examined any extensive collection of real-world examples such as our list of contexts where we perceive chance you will realize that a majority do not fit readily into either category. So, when is it possible and useful to consider a particular instance of uncertainty and somehow try to dissect it into epistemic and aleatoric components? Let me first juxtapose two quotes and give two examples, and then give my own discussion.

Two quotes

The first is from Craig Fox at UCLA Business School:
Successful decisions under uncertainty depend on our minimizing our ignorance, accepting inherent randomness and knowing the difference between the two.
The second is from Taleb's Black Swan:
In theory randomness is an intrinsic property, in practice, randomness is incomplete information … . The mere fact that a person is talking about the difference implies that he has never made a meaningful decision under uncertainty -- which is why he does not realize that they are indistinguishable in practice. Randomness, in the end, is just unknowledge.

Two prototype examples

The first is from this 2011 Fox-Ulkumen article.
On April 29, 2011 Barack Obama made one of the most difficult decisions of his presidency: launch an attack on a compound in Pakistan that intelligence agents suspected was the home of Osama bin Laden. In an interview Obama described the raid as the longest 40 minutes of his life. He attributed that tension to two sources. First, he was not certain that bin Laden was actually residing in the compound. "As outstanding a job as our intelligence teams did," said the President, "At the end of the day, this was still a 55/45 situation. I mean, we could not say definitively that bin Laden was there." Second, regardless of whether bin Laden was residing in the compound, it was not certain that the operation would succeed. Asked about this concern the President cited operations by previous presidents that had failed due to chance factors, saying, "You're making your best call, your best shot, and something goes wrong -- because these are tough, complicated operations."

Note that these two sources of uncertainty are qualitatively distinct. The first reflects the President's lack of confidence in his knowledge of a fact (i.e., whether or not bin Laden was residing at the compound). The second reflects variability in possible realizations of an event that is largely stochastic in nature -- if the mission were to be run several times it would succeed on some occasions and fail on others due to unpredictable causes (e.g., performance of mechanical equipment, effectiveness of U.S. troops and bin Laden's defenders on a particular night).

The second case involves a model for sports or games -- for simplicity a 2-team (or 2-person) game ending in win/lose. In the model each team is assumed to have a numerical ability (skill) level, and the probability that A will beat B is assumed to be a given function of the difference in skill levels. As a specific example, the International football teams of Germany and France currently (June 2018) have Elo ratimgs of 2030 and 1999, which corresponds to an estimated probability 54% of Germany winning a hypothetical upcoming match. Here is my own account of this topic. As in the first example, we see two sources of uncertainty. The skill of each team is uncertain -- we cannot hope to capture the notion of "skill" in a single number and calculate that number exactly. And then (even if we could) the result of a match would still be "stochastic in nature" because of the multitude of individual interactions.

So these two examples can indeed be neatly dissected into epistemic and aleatoric components. But how representative are they?


I have two comments.

(1) In the context of "decisions under uncertainty" the Fox quote is certainly reasonable. But this is only a small part of our perception of chance (see list of contexts and why do we care about probabilities). One could expand the scope by distinguishing three settings in which we might care about probabilities.
Textbook setting where there is explicit limited relevant information.
Observer setting where you have the ability to seek further relevant information.
Control setting where you can also take actions to change probabilities.

(2) I juxtaposed the two quotes because at first sight they seem contradictory. Is this (as Taleb suggests) "a distinction without a difference"?

In seeking to estimate the chance that (given he was there) the bin Laden raid would be successful, one can only compare with previous similar operations, as Obama implied. But exactly the same is true for estimating the chance that bin Laden was there; presumably the intelligence services have considerable experience in trying to locate people trying to hide, so estimating the chance of success in this case must rely on results from previous similar efforts. From the viewpoint of estimating the overall probability of success, the distinction surely makes no difference. Turning to the football case, again consider how we actually deal with the two sources of uncertainty. Given a numerical measure of skill, we should use data from past matches to estimate the win-probability as a function of skill difference. But how do we calculate this numerical measure of skill? -- well, by some algorithm based on data on the results from past matches. So again, we are just using the same type of data for both types of uncertainty.

And this perhaps supports Taleb's viewpoint.


In classical frequentist statistics one can regard the likelihood function as describing the the aleatoric uncertainty and the parameters as epistemic uncertainty, implicitly indicating a distinction between them, whereas a Bayesian puts a prior distribution on the parameters, implicitly rejecting a distinction.

Ongoing research of Fox-Ülkümen seeks to study in detail "the psychological implications of reasoning under epistemic versus aleatory uncertainty" as it affects actual decision-making.