1132: "Frequentists vs. Bayesians"

[rhomboidal]

Title Text: 'Detector! What would the Bayesian statistician say if I asked him whether the--' [roll] 'I AM A NEUTRINO DETECTOR, NOT A LABYRINTH GUARD. SERIOUSLY, DID YOUR BRAIN FALL OUT?' [roll] '... yes.'

I'm suspicious of any scientific equipment that could also function in a Vegas craps room.

[Djehutynakht]

I'm frequently haunted by Statistics.

[pareidolon]

Best way to make money ever: bet people the world isn't going to end.
If you win, you get their money.
If you lose, meh.
On what I assure you is an entirely unrelated topic,
anyone here believe the world's going to end on December 12th?

[Coyne]

A $50 bet with a 1-in-36 chance of winning? And never having to pay off if he loses? Clever, isn't he?

[Quicksilver]

Haha, Labyrinth. I actually bought the film because a DVD shop had it playing and that answer came up with the guards.

[margeman2k3]

At least it's not a neutrino detector that stabs people who ask tricky questions...

[Durandal_1707]

A $50 bet with a 1-in-36 chance of winning? And never having to pay off if he loses? Clever, isn't he?

The odds are far better than that; the guy on the left neglected to factor in the probability that the sun would just suddenly go nova five billion years early. I'm not sure what the odds of that are, but I'm sure they're a lot lower than 1/36.

[Aiwendil]

That's one really small neutrino detector!

[Alan]

No no no no.

This isn't clever because he doesn't pay off the bet if he loses.

The Bayesian statistician knows the probability of the sun going nova is very, very small. That makes the chance that the machine rolled double sixes a near certainty (either that or the machine malfunctions).

P(Nova) = 1/1000000000
P(NotNova) = 999999999/1000000000

P(YES|Nova) = 35/36
P(YES|NotNova) = 1/36

P(Nova|Yes) = ( P(Yes|Nova) * P(Nova) ) / (P(Yes|Nova) * P(Nova) + P(Yes|NotNova) * P(NotNova))

P(Nova|Yes) = (35/36000000000) / (35/36000000000 + 999999999/36000000000)

P(Nova|Yes) = (35/36000000000) / (1000000034/36000000000)

P(Nova|Yes) = 35/1000000034

[MetaThought]

That's one really small neutrino detector!

Duh! It uses the rate of radioactive decay to detect neutrinos.

[doogly]

Do frequentists exist anymore?

[MetaThought]

In this case, someone who knew nothing of probability could also safely bet $50 that the sun hasn't exploded. A little knowledge is a dangerous thing?

[bjowen]

Or you could just look up & see whether the moon and planets become an order of magnitude brighter over the next few minutes. If, that is, you didn't want to fleece the poor guy for his $50.

[domino14]

Or you could just look up & see whether the moon and planets become an order of magnitude brighter over the next few minutes. If, that is, you didn't want to fleece the poor guy for his $50.

Relevant: http://en.wikipedia.org/wiki/Inconstant_Moon

[bjowen]

Relevant: http://en.wikipedia.org/wiki/Inconstant_Moon

Cool, thanks! Will track it down... I was recently directed to a similar disaster scenario story, A Pail of Air, which is well worth a read.

[snowyowl]

And this is why you should always insist on p<0.01.

Code: Select all

[the point]



   o  <-- my head
  /|\
   |
  / \

[J L]

What a setup for a Labyrinth reference.

I always loved this riddle. Great fun in role-playing, too -- either the players know it (and even then some probably forgot the solution) or it's endless hours of fun for the game master.

[CatOfGrey]

On a separate note - did Niels Bohr and Robert Oppenheimer really wager a dollar on the first nuclear weapons test? The story I heard was that they thought the nuke might ignite the ozone layer, removing most life from the planet. If it is true, I would wager that Bohr was betting for the disaster, but that's just a guess...

By the way, I play my 'correlation vs. causation' comic reference here, noting that statistical modeling, like deep frying a whole turkey or base jumping, should only be attempted by trained experts with a proper array of safety equipment on hand.
http://xkcd.com/552/

[legionlabs]

Arguably the frequentist makes another error, he formulates his hypothesis *after* collecting data. A moot point perhaps, but if you're going to do something wrong, do it wrong right!

If my memory is working correctly, he should formulate his hypothesis, decide on an alpha value based on the amount of humiliation caused by Type I error, *then* ask the machine.

I think there are post-hoc tests, but much like multiple comparison corrections, they largely get swept under the rug because most journals only publish significant results (there are some exceptions). So in this case, the frequentist gets a publication and the bayesian only gets 50$

I love to rag on frequentist stats, but seriously they're OK for many situations.

[Iranon]

That mistake is depressingly common.

You just got mugged by a redhead in a dark alley. Chances are it wasn't a redhead.

You have a cold and consult your physician, because you're worried about uncharacteristically severe symptoms that are a telltale sign for something serious.
You need to compare (chance of a particularly nasty cold if you know you have one, say 1/10) to (chance of getting something exotic times chance of showing typical symptoms, say 1/10000 * 9/10).
Overwhelmingly probable that it's just a cold, but you may not realise and they may still feel the need to cover their behind.

If you consider that judges don't necessarily have a sense for this, things get very very scary...

[aimaz]

Anyone who wants to know more about the history of frequentism vs Bayes would probably enjoy the book "The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from two centuries of controversy". It's a little pop-sci, but pretty interesting.

[314159]

The example I heard relating to this was of a very rare disease that affected 1 in a million people, and a test for that disease that's 99% accurate. If you get diagnosed with the disease, what are the odds you have it? Somewhere in the region of 1 in ten thousand, as you're most likely in the misdiagnosed disease free group.

That's always the problem with prescribing medical tests, which unfortunately most patients don't understand. And it's why doctors are trained in Bayesian statistics.

[da Doctah]

Haha, Labyrinth. I actually bought the film because a DVD shop had it playing and that answer came up with the guards.

It also came up in an episode of The Powerpuff Girls.

On what I assure you is an entirely unrelated topic,
anyone here believe the world's going to end on December 12th?

Far more likely event: on December 11th, flying saucers land and disgorge hundreds of Mayan astronomers, apologizing for taking so long to get back with the updated calendars, but the traffic was just murder.

[bisurge]

Haha, Labyrinth. I actually bought the film because a DVD shop had it playing and that answer came up with the guards.

It also came up in an episode of The Powerpuff Girls.

On what I assure you is an entirely unrelated topic,
anyone here believe the world's going to end on December 12th?

Far more likely event: on December 11th, flying saucers land and disgorge hundreds of Mayan astronomers, apologizing for taking so long to get back with the updated calendars, but the traffic was just murder.

I hope they brought the ones with the cute dogs on it.
I wonder if the detector could tell me the probability that they did...

[Bromskloss]

Is this the riddle you are talking about? https://www.youtube.com/watch?v=2dgmgub8mHw

Note: Odds are not the same thing as probability. When in doubt, go for probability. More people will understand you that way too.

[Degenx3]

The example I heard relating to this was of a very rare disease that affected 1 in a million people, and a test for that disease that's 99% accurate. If you get diagnosed with the disease, what are the odds you have it? Somewhere in the region of 1 in ten thousand, as you're most likely in the misdiagnosed disease free group.

That's always the problem with prescribing medical tests, which unfortunately most patients don't understand. And it's why doctors are trained in Bayesian statistics.

Keep in mind that this is only true if it's a disease that everybody is tested for. If only 1 in 1000 people show the warning signs of such a disease and are specifically tested for it, then your chance of actually having the disease after a positive reading is 1/10

[iserp]

I am baffled. Frequentist statisticians also believe in Bayes theorem. It is mathematically correct. But to apply it, you need to know lots of information before... e.g. The probability of the sun exploding, the efficiency of the detector, the probability of having a false positive (the detector lying).

More often than not, you know very little of this distributions. And this is where people can get very imaginative with bayesian statistics, to the point of fundamentally changing their results just changing their prior distributions.

Not that bayesian statistics is wrong, but you have to know how to apply known information, and you can do that the frequentist way (just not shown in the comic).

[lfmcm]

I just want to thank Randall so much for writing this comic. I've been reading xkcd for about 9 or 10 years now, and this one made me happier than anything except Purity (http://xkcd.com/435/)

I did an undergraduate degree in mathematics, focusing on very pure mathematics and highly impractical theoretical physics, and ignored all the stats stuff because it was taught so theoretically that it seemed boring and pointless. Then I got a job, worked with stats, and discovered Bayesian statistics, and realised that there was fun to be had in statistics after all.

Anyway, thanks Randall!

[J Thomas]

The example I heard relating to this was of a very rare disease that affected 1 in a million people, and a test for that disease that's 99% accurate. If you get diagnosed with the disease, what are the odds you have it? Somewhere in the region of 1 in ten thousand, as you're most likely in the misdiagnosed disease free group.

That's always the problem with prescribing medical tests, which unfortunately most patients don't understand. And it's why doctors are trained in Bayesian statistics.

Keep in mind that this is only true if it's a disease that everybody is tested for. If only 1 in 1000 people show the warning signs of such a disease and are specifically tested for it, then your chance of actually having the disease after a positive reading is 1/10

So, say the test costs $200. If it gives a positive result then 99% of the time it's right, and if it gives a negative result then 99% of the time it's right. You give it to a million people and it costs $200 million and you haven't learned much at all. But then if there are warning signs, and one person out of a thousand who has those warning signs has the disease. So you give the test to 1000 people and it costs $200,000, and the result is that it tells you that 10 of them have the disease. Probably one of them really does.

This is worthless.

So, say a competing test comes out that's 99.99% effective. It costs $400. You give the test to 1000 at-risk people for $400,000 and the result is that it tells you that one of them has the disease. There's around a 90% chance he really does, and there's a 10% chance that another of those 1000 has it and the test missed.

Still worthless.

What this tells MDs is to ignore rare diseases. If 0.1% of the possible treatments cover 90% of the actual problems, then you will be 90% effective if you never consider anything else. If 1% of the possible treatments cover 99.9% of the actual problems, then you will be 99.9% effective if you never consider anything else. And there are so many other ways to go wrong that it's very hard for human beings to be 99% effective. So it makes sense for med students to forget all the rare stuff right after the test.

[Red Hal]

Heh. Five paragraphs...

Anyway, the point of a positive test is to investigate further. Apply the test twice, (or a different test) but only to those that presented positive the first time. You'll very quickly get those odds down. More troubling than false positives are false negatives. Those are the ones to avoid.

[Someguy945]

Title Text: 'Detector! What would the Bayesian statistician say if I asked him whether the--' [roll] 'I AM A NEUTRINO DETECTOR, NOT A LABYRINTH GUARD. SERIOUSLY, DID YOUR BRAIN FALL OUT?' [roll] '... yes.'

http://xkcd.com/246/

[EpicanicusStrikes]

What this tells MDs is to ignore rare diseases.

Mindsets like this are lethal. My father died because his lung cancer was treated as pneumonia and not caught until it was too late. My uncle died because his esophageal cancer was treated as GERD until it was too late. My mother died because even though they caught her breast cancer in time, she developed pyoderma gangrenosum which was treated as vasculitis.

The treatment for vasculitis is debridement, which is the best way possible to ensure the spread of PG. But of course they didn't realize that until, once again, it was too late.

No actual comment on the comic. I just really fucking hate the medical industry.

[david_h]

Or you could just look up & see whether the moon and planets become an order of magnitude brighter over the next few minutes. If, that is, you didn't want to fleece the poor guy for his $50.

Relevant: http://en.wikipedia.org/wiki/Inconstant_Moon

Except that, of course,

Spoiler:

it turned out the sun hadn't gone nova.

David

[JediMaster012]

My first thought was that the need to ask the question of the neutrino detector was an indication that there was reason to suspect the sun exploding. i.e. The probability of sun explosion wasn't necessarily small. I actually found it funny because of the clever bet interpretation (that he wouldn't pay if he lost).

After reading the comments talking about the slim chance of explosion, why would it need to be a "Bayesian Statistician?" Wouldn't the average person tell you there was no chance of the sun having exploded regardless of what someone or something told you? Isn't that just common sense?

[david_h]

What this tells MDs is to ignore rare diseases.

Mindsets like this are lethal.

I hope you won't take offence, and I'm sorry for your losses and the circumstances surrounding them, but to say this mindset is lethal seems to me to be a generalisation along the same lines as "cars are lethal" or declaring the justice system irretrievably useless based on a handful of mistrials. Isn't this (by which I mean the use of Bayesian statistics to guide decision-making in general, not your personal experiences about which I know nothing) just being practical?

David

[peewee_RotA]

So this is like calculating the probability that drawing a chance card will make grandma go crazy and flip the table, monopoly pieces and all. Or is it like determining a winner right after that happens. That part of the game is not covered in the rule book.

[Iranon]

The example I heard relating to this was of a very rare disease that affected 1 in a million people, and a test for that disease that's 99% accurate. If you get diagnosed with the disease, what are the odds you have it? Somewhere in the region of 1 in ten thousand, as you're most likely in the misdiagnosed disease free group.

That's always the problem with prescribing medical tests, which unfortunately most patients don't understand. And it's why doctors are trained in Bayesian statistics.

Keep in mind that this is only true if it's a disease that everybody is tested for. If only 1 in 1000 people show the warning signs of such a disease and are specifically tested for it, then your chance of actually having the disease after a positive reading is 1/10

So, say the test costs $200. If it gives a positive result then 99% of the time it's right, and if it gives a negative result then 99% of the time it's right. You give it to a million people and it costs $200 million and you haven't learned much at all. But then if there are warning signs, and one person out of a thousand who has those warning signs has the disease. So you give the test to 1000 people and it costs $200,000, and the result is that it tells you that 10 of them have the disease. Probably one of them really does.

This is worthless.

So, say a competing test comes out that's 99.99% effective. It costs $400. You give the test to 1000 at-risk people for $400,000 and the result is that it tells you that one of them has the disease. There's around a 90% chance he really does, and there's a 10% chance that another of those 1000 has it and the test missed.

Still worthless.

What this tells MDs is to ignore rare diseases. If 0.1% of the possible treatments cover 90% of the actual problems, then you will be 90% effective if you never consider anything else. If 1% of the possible treatments cover 99.9% of the actual problems, then you will be 99.9% effective if you never consider anything else. And there are so many other ways to go wrong that it's very hard for human beings to be 99% effective. So it makes sense for med students to forget all the rare stuff right after the test.

In a multi-tiered system the initial tests don't have to be good. "Might as well" is good enough.
Not in a risk group/showing symptoms? Don't bother, too many false positives to be useful.
In a risk group or showing symptoms? Do a cheap, easy test even if it has plenty of false positives.
That returns a positive? Do a proper test (which you don't do by default because it's expensive, has its own health risks or involves sticking uncomfortable things somewhere personal).

Some cancer screening works this way. Doesn't make sense to test 20-year-olds in the first place because they're vanishingly unlikely to have that type of cancer even if the test were to have a positive result. Test returns a positive for a 60-year-old?
Probably still a false alarm, but enough to warrant a more reliable (but more expensive/unpleasant/time-consuming) procedure.
You need to know the context.

[Pingouin7]

On what I assure you is an entirely unrelated topic,
anyone here believe the world's going to end on December 12th?

Why would anyone believe that?

[EpicanicusStrikes]

What this tells MDs is to ignore rare diseases.

Mindsets like this are lethal.

I hope you won't take offence, and I'm sorry for your losses and the circumstances surrounding them, but to say this mindset is lethal seems to me to be a generalisation along the same lines as "cars are lethal" or declaring the justice system irretrievably useless based on a handful of mistrials. Isn't this (by which I mean the use of Bayesian statistics to guide decision-making in general, not your personal experiences about which I know nothing) just being practical?

David

Maybe, but more practical would be actually doing some research on the conditions being presented. When a doctor tried to diagnose my sinus polyps as TMJ Disorder, I walked away and found someone willing to actualy scan my head because I knew he was wrong. He just refused to admit it.

Once it became obvious that the debridement was only spreading my mother's condition over a period of months, until half her upper body had been eaten down to the ribcage, alternative solutions needed to be sought out. But they refused to change their tactics. Even their head of the infectious diseases department refused to listen to my suggestions that it might be USA 300, because he apparently doesn't read the CDC's EID Journal and had no clue that it was being tracked across the state.

He just wanted to push patients through his hyperbaric chamber like it was a some panacea drive-through. The surgeon in charge had never even heard of PVL toxins. Of course, it turned out I was wrong, but at least I knew the proper treatment was not to just keep ripping pieces off of her until she either got better or she died.

By the time PG was actually tested for, found to be postive and a treatment of immunosuppressants was started, her health was trashed to the point where she lived another year-and-a-half in constant agony.

Plus there was the constant vancomycin resistant MRSA bacteremia she had to deal with. Every time she tested positive, they went straight for the stuff that she claimed never worked on it, her charts claimed never worked on it and I claimed never worked on it. But of course they wouldn't deviate from dictation until, duh, it ended up not working.

Or the fractured back she eventually developed which they didn't find until an x-ray just happened to spot it, despite her constant claims that something serious was wrong. One step further, the persistant bacteremia produced a cyst in the fracture. But, again, they just thought she was being whiny about the pain and wouldn't listen or investigate further until she refused to attend any further therapy sessions.

So what's my point? Well, go ahead and start with the basics. That makes most sense. But for god's sake, change your damn mind when continued evidence shows that you are wrong. Listen to your patients. Listen to their family. Read their charts. The facts are there. Don't wait until someone is coughing up blood (like my father), or crapping out blood (like my uncle), before you consider that a new diagnoses may be in order. And when the necrotizing fasciitis you're spreading throughout a patient eats an entire pectoral muscle group, consider that you are not dealing with capillary damage from radiation therapy that was well documented to be within safe guidelines.

Blindly following the most likely scenario and being trained to ignore darkhorse diagnoses is a bad thing.

[benhowt]

If you consider that judges don't necessarily have a sense for this, things get very very scary...

More worrying is this; turns out one judge has decided that the assumption is so worrying that they won't accept its use in court...even people who get it don't get it...

http://www.guardian.co.uk/law/2011/oct/02/formula-justice-bayes-theorem-miscarriage