Bayes' Theorem Explained!

Holy shit!  I finally understand Bayes' theorem.  Thanks again, Nate Silver, you beautiful bastard.  Bayes' theorem provides an answer the question: what is the probability of an event occurring (given some other event and our prior knowledge of both)?  This is a bit abstract, so let's use Silver's example.

Let's say you find some suspicious underwear at home.  What is the probability that your significant other is cheating on you, given this evidence?  In statistical notation, we'll say the the probability your partner is cheating is P(A).  The probability of finding strange underwear in your house is P(B).  And the probability that your significant other is cheating given the evidence of the underwear is P(A|B).

To figure out this probability, you break the problem into three parts.  First, you estimate the chances that underwear is a sign of cheating, or P(B|A).  There probably aren't many good explanations for it, so let's say the probability is 75%.  Second, you estimate the chances that the underwear has nothing to do with cheating, or P(B|~A).  This is kind of hard to imagine, so let's assume the probability is 10%. Finally, you try to estimate the chances that your partner would cheat on you, before you found any panties, P(A).  According to studies, 4% of married spouses cheat in any given year, so we'll go with that.

Even though the underwear seems pretty damning, your prior estimation leads to a low probability of cheating.  Let's do the math.  Bayes formula is: P(A|B) = P(B|A) * P(A) / P(B).  That is, P(partner is cheating given the evidence of underwear) = P(partner leaves underwear around because they're cheating) * P(partner is cheating) / P(finding mysterious underwear in your house).  P(B) expands to P(B|A) * P(A) + P(B|~A) * P(~A), so we have (.75)*(.05)/((.75)*(.05) + (.10)*(.95)).  In the end, it's only 28%.

This answer may be surprising, given the high likelihood that underwear is a sign of cheating.  The reason for this is the very low estimation we gave to the chances of cheating prior to finding the underwear.  Of course, these were very rough estimates, but they help us ballpark a number.  They also show two important important things about Bayes' theorem.

First, Bayes' theorem is highly dependent on prior estimates and tests.  There is a school of statistics, usually called 'frequentist', which defines probability in terms of the frequency of an event in a large number of tests.  According to frequentists, probability doesn't refer to prior probabilities.  The assumption here is that there is a correct probability which can be found through a large enough sample size.  For Bayes, it's probabilities all the way down.  Any probability is based on prior estimates, which were based on prior estimates, etc.

If you don't get it, don't feel bad.  Try here.

Bayes' theorem also forces us to take heed of the possibility of false positives.  No test is 100% accurate.  For example, if a steroids test is right 95% of the time, it might provide false readings 15% of the time.  Given the fact that 10% of athletes use steroids, we cannot say that a positive test gives us anything like 95% probability of steroid use.  Instead, it's more like 40%.

Philosophically, what's interesting about Bayes' theorem is it provides a method for evaluating new knowledge.  The psychologist and philosopher William James generalized the way we evaluate new experiences in the following way:

The process here is always the same. The individual has a stock of old opinions already, but he meets a new experience that puts them to a strain. Somebody contradicts them; or in a reflective moment he discovers that they contradict each other; or he hears of facts with which they are incompatible; or desires arise in him which they cease to satisfy. The result is an inward trouble to which his mind till then had been a stranger, and from which he seeks to escape by modifying his previous mass of opinions. He saves as much of it as he can, for in this matter of belief we are all extreme conservatives. So he tries to change first this opinion, and then that (for they resist change very variously), until at last some new idea comes up which he can graft upon the ancient stock with a minimum of disturbance of the latter, some idea that mediates between the stock and the new experience and runs them into one another most felicitously and expediently. 

James here is describing the everyday process of learning new information.  But, as James knew better than anyone, we don't always do this well.  Sometimes we don't think about false positives; sometimes we want to believe things that are too good to be true; and sometimes we get caught up in the present moment and forget about past experience.

Even if you don't work out the probabilities every single time you learn something troubling to your understanding of the world, understanding Bayes' theorem can still be helpful.  At the very least, it warns us about making rash decisions when we learn something surprising.  Silver believes that if we all thought a bit more probabilistically about the world, it might be a better place.  We would understand the uncertainties of our forecasts and anticipate exceptions to our beliefs.  It's worth a shot, I guess.