## 0882: "Significant"

This forum is for the individual discussion thread that goes with each new comic.

Moderators: Moderators General, Prelates, Magistrates

philip1201
Posts: 201
Joined: Tue Nov 03, 2009 6:16 am UTC

### Re: 0882: "Significant"

The scientists and people are actually the same people. The actual scientists were too busy playing minecraft, so the girl just undoes her hair and the guy wears Science Goggles, so the news just thought they would be scientists.
Once I was subscribed to a popular science magazine. Then I noticed they did this. Now I'm no longer subscribed*.
Also, while it's necessary for the joke, there is no way they would post the margin of error in a newspaper.

(* because my free yearly subscription ran out. See what happens when you assume causation from correlation?)

nico
Posts: 16
Joined: Fri Aug 22, 2008 6:53 am UTC

### Re: 0882: "Significant"

wackojacko1138 wrote:Say it with me kids:
"The p-value represents the likelihood of the evidence given the null hypothesis, not the likelihood of the null hypothesis given the evidence."
If you know, understand, and live that sentence, then you understand statistics better than most journalists and some scientists.

Thomas does not agree with you, Sir

PS: this is totally a non plausible situation. With a result like that, newspapers would have written that all candies are fatal AND increase your risk of atherosclerosis by 57%.

Pfhorrest
Posts: 5487
Joined: Fri Oct 30, 2009 6:11 am UTC
Contact:

### Re: 0882: "Significant"

So is the joke that the scientists, egged on by the annoying people, tested (all?) 20 flavors of jelly bean to conclusively show "no, there really isn't any link", but then made a typo with their inequality sign on one, and together with the number of other tests run that one (false, due to typo) result becomes much more significant than it would be taken in isolation? I confess I don't completely understand the math at work here, but I would think the first test would have to have demonstrated a link for that to work, i.e. "Hmm, jelly beans appear linked with acne. I wonder if it's just one color? [20 tests, only green shows link] Yep, seems to be just green."
Forrest Cameranesi, Geek of All Trades
"I am Sam. Sam I am. I do not like trolls, flames, or spam."
The Codex Quaerendae (my philosophy) - The Chronicles of Quelouva (my fiction)

A_S00
Posts: 3
Joined: Mon Sep 20, 2010 8:16 pm UTC

### Re: 0882: "Significant"

In similar (if slightly old) news: http://www.wired.com/wiredscience/2009/09/fmrisalmon/ - Neuroscience's infamous dead fish experiment (another cautionary tale about correcting for multiple comparisons)

Re: The Bonferroni discussion above, Bonferroni correction is an ugly thing to do to your statistical power, to be avoided at all costs. If the experimental design allows it, I'm a fan of Fisher-protected t-tests. Pretty good rate of type 1 error control on the multiple comparison front, with no cost whatsoever to the power of the pairwise comparisons (the only loss of power for the experiment as a whole is from the possibility of a type 2 error when testing the significance of the multiple R...and if your multiple R is small enough that you care about your power when trying to detect it, your effect sizes are probably small enough that nobody cares).

VectorZero
Posts: 471
Joined: Fri Nov 02, 2007 7:22 am UTC
Location: Kensington

### Re: 0882: "Significant"

Now if the claim was "jelly beans cure acne" and the scientists were "jelly bean manufacturer in-house scienticians" ...

@pforrest: short answer, no. p<0.05 means there is a less than 5% chance the result is due to chance. If you do a single test with an a priori expectation of a result (say, by detailing a physiological pathway by which a pharmaceutical agent would act on a system), then p<0.05 is accepted as 'significantly' likely to be a genuine result (it's an arbitrary figure). If you do 20 tests, with no a priori reason to expect an outcome and post hoc pull out the 'significant' results then you are less justified in claiming statistical significance. It still might be true. It is probably due to chance.
Last edited by VectorZero on Wed Apr 06, 2011 7:08 am UTC, edited 1 time in total.
Van wrote:Fireballs don't lie.

nico
Posts: 16
Joined: Fri Aug 22, 2008 6:53 am UTC

### Re: 0882: "Significant"

Pfhorrest wrote:So is the joke that the scientists, egged on by the annoying people, tested (all?) 20 flavors of jelly bean to conclusively show "no, there really isn't any link", but then made a typo with their inequality sign on one, and together with the number of other tests run that one (false, due to typo) result becomes much more significant than it would be taken in isolation? I confess I don't completely understand the math at work here, but I would think the first test would have to have demonstrated a link for that to work, i.e. "Hmm, jelly beans appear linked with acne. I wonder if it's just one color? [20 tests, only green shows link] Yep, seems to be just green."

This is an example of multiple comparisons fallacy.

You may want to have a look at this:
http://en.wikipedia.org/wiki/Multiple_comparisons

The problem is that the p-value does NOT tell you that acne and green jelly beans are correlated 95% of times.
It tells you that you have <5% (1/20, the number of panels is well chosen) probability that the correlation in the set of data you gathered is just due to chance.

cjdrum
Posts: 113
Joined: Sun Dec 12, 2010 4:51 am UTC
Location: BACK

### Re: 0882: "Significant"

Oops, I just skimmed through reading the colours... Then I got to the bottom frame and was like "Wait..." *checks*

Posts: 51
Joined: Mon Apr 12, 2010 12:26 am UTC
Location: My used book store near Seattle WA

### Re: 0882: "Significant"

Qwerty.55 wrote:

I enjoy recursive signatures.

I like your new recursive sig even better than before.

Me too!
"As the size of an Explosion increses the number of social situations it is incapable of resolving approches zero"
-Vaarsuvius, The Order of the Stick
Spoiler:

Qwerty.55 wrote:I like your new recursive sig even better than before.

syko_lozz
Posts: 53
Joined: Fri Jan 11, 2008 5:30 am UTC
Location: Oz

### Re: 0882: "Significant"

ummmm.... ok I am one of those life-scientists that totally doesnt get stats. I really dont see whats wrong with this.
So thinking out loud... yeah if I was doing the experiment I'd put them all onto one graph and tell my software to do an anova (rather than a whole bunch of separate t-tests).... but what if they were all independant experiments and didnt share a common control?

of course if the p values were all close to 0.05 then yeah its dodgy but what if the green one was way below 0.05 and really clear with a really big sample and a huge fold difference? and all the others had really high p-values?

is the point just that you cant rely on p-values alone? arent we missing a whole bunch of other details about the experiment to really judge whether the newspaper article was stupid or not?

these sorts of experiments are done all the time, for example, the effect of various drugs on growth of a cell line. (drug A, no effect, try the next one, drug B, no effect, drug C, ooh effect!) but this is spread over years and it seems silly to consider it all as one huge data set.

teeeaaach meeeee!

edit- will read article, but still, teach me!
Debate politics with a fern. If you lose, refuse to water it.

soveraign
Posts: 3
Joined: Sun Mar 21, 2010 12:59 am UTC

### Re: 0882: "Significant"

- Think to self, "I'm a scientist, I've heard of minecraft, what is this?"

paravatar
Posts: 9
Joined: Sun Oct 10, 2010 8:12 am UTC
Location: Ulm, Germany

### Re: 0882: "Significant"

thank you for this one!

yesterday I tried to explain the dangers of multiple comparisons to a coworker!

… but the comic strip lacks a proper "table one" and it gives no information if the trial was randomized…
with science like that you'll never make it to the New England Journal
…is listening to the swooshing sound of deadlines passing by

Scarlet
Posts: 15
Joined: Wed Jan 14, 2009 12:18 am UTC

### Re: 0882: "Significant"

Oh, science!
Current project
Honours Bachelor of Science
Computational Chemistry
University of Waterloo
Sept 2007 - Apr 2012

Turing Machine
Posts: 98
Joined: Fri Feb 19, 2010 5:48 am UTC

### Re: 0882: "Significant"

haha SMBC was funny when it did this

so long ago

VectorZero
Posts: 471
Joined: Fri Nov 02, 2007 7:22 am UTC
Location: Kensington

### Re: 0882: "Significant"

Also @pfhorrest: the initial test of all jelly bean colors doesn't have to show a difference. In fact, the correlation of the green jelly beans would have to be very strong indeed to show through the noise of 19 other colors that don't have a correlation with the expected outcome.

@syko_lozz: you're right, there is insufficient information to comment fully on the validity of the study. Yet that didn't stop the journos from claiming the correlation to be proven causation, did it?
Van wrote:Fireballs don't lie.

darkspork
Posts: 532
Joined: Tue Sep 23, 2008 12:43 am UTC
Location: Land of Trains and Suburbs

### Re: 0882: "Significant"

GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.
This has been roughly my response to everything for a week.
Shameless Website Promotion: Gamma Energy
My new esoteric programming language: GLOBOL
An experiment to mess with Google Search results: HARDCORE PORNOGRAPHY HARDCORE PORNOGRAPHY

Philyphil
Posts: 3
Joined: Mon Jan 24, 2011 9:29 am UTC

### Re: 0882: "Significant"

This comic is great. This is how, often old, scientific information is actually eminated. Nice work as ever. Loving the reccursive sig floating about today too!

glasnt
Posts: 539
Joined: Fri Jan 25, 2008 5:18 am UTC
Location: SQUEE!

### Re: 0882: "Significant"

This is now my excuse for everything:

"But.. miiiiiinecraft... "

HI JOEE

le_sacre
Posts: 35
Joined: Wed Jun 20, 2007 11:24 pm UTC
Location: stanford, ca
Contact:

### Re: 0882: "Significant"

Love the comic, but how could you pass up the opportunity for a "Bonferroni"-related punchline in the alt-text??

To the people (including scientists!) who say they don't get it: read it again! The way to think about p-values is to say, for p < .05, "You'd expect to get an effect at least this large just by chance 5% of of the time (if the null hypothesis were true)."

5% of the time = 1 out of 20.

Most of the brilliant, accurate, low-effort heuristics that have been evolutionarily selected into the human mind deal with single cases or very simple proportions. Probability and statistics is something a lot of us struggle to deal with intuitively. However, I think the basics can be grasped by anyone if they're taught well and early enough. In my grand agenda, stats will be a major and required part of both math and science curricula starting in grade school; it's the mathematics of decision-making, which is arguably far more vital in our world than the mathematics of mechanics or architecture, which are more specialty topics.

lrossouw
Posts: 3
Joined: Wed Feb 03, 2010 5:10 am UTC

### Re: 0882: "Significant"

Good stats commic.
Spoiler:
For those that don't get it. Basically the p value (5% = 1/20) of the green test says there is a one in twenty chance that there is no link on green. However they effectively repeated the test 20 times so if none of the colours are linked you expect on average one false positivie for a link in 20 tests. So their results are in reality broadly as expected for the scenario where no link is made.

To really test the validity of the result they need a combined test that takes all the data into account. Though I would not know exactly what is best in this case.

Tetrinity
Posts: 3
Joined: Wed Jan 19, 2011 7:42 am UTC

### Re: 0882: "Significant"

I'm no statistician, but I'm glad I knew enough to get this one. I must admit I skimmed past the colours first reading too...

And upon re-reading, noticed that they tested yellow jelly beans twice, in panels 13 and 18. Presumably a mistake on Randall's part, but I suppose there's nothing wrong (scientifically speaking, at least) with repeating the experiment.

spekkeh
Posts: 17
Joined: Wed Nov 18, 2009 11:13 am UTC

### Re: 0882: "Significant"

Diadem wrote:Unfortunately, this is exactly how many scientists (not all, luckily, but too many) do statistics. They really do tests on dozens of variables at once, with several different statistical tests, and then publish the ones that are interesting.

For a single experiment you should always divide your allowed margin of error by the number of variables you are testing for. So if you test n different colours of jelly beans, you must require p < 0.05/n before considering a result statistically significant. But this is often ignored.

....unless you write an article about each experiment separately, in which case you won't have to do it anymore. The comic would still apply, because basically that's what p<0.05 says, that once every 20 times you will get a false positive, so if that's your alpha, dividing it by twenty doesn't make any sense. Even in the case where you perform multiple tests on the same data (of which this is not an example), Bonferroni adjustments are pretty iffy, cf.

Nakagawa, S. (2004). A farewell to Bonferroni: the problems of low statistical power and publication bias. Behavioral Ecology, 15(6), 1044-1045. doi: 10.1093/beheco/arh107.

graatz
Posts: 87
Joined: Thu Oct 29, 2009 7:24 pm UTC

### Re: 0882: "Significant"

So in other words, 1 out of every 20 studies that use a 95% CI is wrong. The number of the wrong ones that get reported by popular media is overrepresented.

(Yes, I get the fact that the joke is that testing different jelly bean colors is essentially running the same test over and over, thus the expected 1 anomaly in 20, but it's nice to be periodically reminded how big of a number 5% actually is and how arbitrary a choice it is for hypothesis testing.)

eidako
Posts: 126
Joined: Wed Apr 06, 2011 10:24 am UTC

### Re: 0882: "Significant"

NASA Scientist: Minecraft Increases Your IQ

Cranium Talksalot (CNN) -- Worry that your kids play too many video games? You shouldn't. A surprising report by NASA roboticist and computer programmer Randall Munroe shows a trend between employment in intellectually demanding fields and playing games. Statisticians and chemists alike have been shown to play Minecraft, the wildly popular sandbox game by indie developer Mojang, even while at work. Shortly after the report's release, Munroe's website was abuzz with fellow scientists, engineers and academics remarking that they play the game, or if they haven't, greatly anticipate doing so. Munroe is an avid player himself. He admits to having played for days on end while recovering from a dental operation, so absorbed in the complex virtual world that he was able to completely ignore any pain.

He is not alone. Over 5.8 million users have registered on the game and 1.7 million have purchased it. There are approximately 1.3 million scientists in the United States, no small coincidence. The other 400,000? One of them just might be your child.

The next time your child wants to stay up playing Minecraft, maybe you should think twice before sending them to bed. That pixelated pickaxe just might be their ticket to an exciting career in science.
Last edited by eidako on Wed Apr 06, 2011 12:00 pm UTC, edited 1 time in total.

ksquared129
Posts: 1
Joined: Wed Apr 06, 2011 11:12 am UTC

### Re: 0882: "Significant"

chaucer345 wrote:See, what they should have done this is run a single factor ANOVA with the colors as the levels and then performed a Tukey MCP to determine which of the levels were significantly different from each other... Or would that have helped? Maybe I'm thinking of this the wrong way.

This is exactly what I thought after reading the comic. Poor scientists, they would have been done so much more quickly (to get back to Miiiiinecraft), without that mistake about the green jelly beans.

eidako
Posts: 126
Joined: Wed Apr 06, 2011 10:24 am UTC

### Re: 0882: "Significant"

(post merged with previous)
Last edited by eidako on Wed Apr 06, 2011 12:01 pm UTC, edited 1 time in total.

KShrike
Posts: 129
Joined: Wed Dec 15, 2010 5:47 am UTC

### Re: 0882: "Significant"

But... We're playing Minecraft (don't ever get this game. It will suck your life away with it's awesomeness)

Another statistics joke. Two in a row. He must really hate statistics this week (the practice thereof)
On 10/10/10, My Little Pony started to appeal to adult males. Ya rly!
Forget the /b/ memes, and go watch the show and see what I mean. After all, the /b/ memes exist because the show itself is so cleverly written. Check it out!

Posts: 137
Joined: Mon Mar 03, 2008 8:57 pm UTC
Contact:

### Re: 0882: "Significant"

Someone wrote:
"The p-value represents the likelihood of the evidence given the null hypothesis, not the likelihood of the null hypothesis given the evidence."

Which is why it is fortunate we have Bayes' Theorem and related techniques to let us use prior information to ascertain an estimate of the plausibility of the hypothesis given the evidence, which is what people who've kept up with the world have been doing for the past decade or more.

The sad fact is that far more than 5% of papers published against the rather modest standard of p < 0.05 turn out to be reporting on results due to random chance. This comic explains why.

No real experiment exists in isolation, and if your analytical techniques do not provide ways of accounting for prior knowledge, you are using the wrong analytical techniques.
Coming on Midsummer's Day to a Web Browser Near You: http://www.songsofalbion.com

KShrike
Posts: 129
Joined: Wed Dec 15, 2010 5:47 am UTC

### Re: 0882: "Significant"

darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.
This has been roughly my response to everything for a week.

Haha... me too. Last Thursday too. "What's Minecraft?...... I .... can't.... stop!!!!!!

This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Mark Twain wrote:paraphrased: There are three kinds of fibs: lies, damned lies, and statistics.

Oh, Mark Twain. If only you'd understand. Being an engineer is hard in the moral sense, really. I can prove it too (p < .05).
On 10/10/10, My Little Pony started to appeal to adult males. Ya rly!
Forget the /b/ memes, and go watch the show and see what I mean. After all, the /b/ memes exist because the show itself is so cleverly written. Check it out!

SirMustapha
Posts: 1302
Joined: Mon Jul 21, 2008 6:07 pm UTC

### Re: 0882: "Significant"

darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.

And then people don't understand what I mean with pandering.

KShrike wrote:This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Why? Randall's practice of namedropping nerd culture items to cause reactions like the one above is extremely common, and this is the second time he has used Minecraft.

Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

jthetzel
Posts: 2
Joined: Wed Apr 06, 2011 1:08 pm UTC

### Re: 0882: "Significant"

Diadem wrote:For a single experiment you should always divide your allowed margin of error by the number of variables you are testing for. So if you test n different colours of jelly beans, you must require p < 0.05/n before considering a result statistically significant. But this is often ignored.

THE CURSE OF BONFERRONI!!!

Balesk Baj, Timeburner
Posts: 76
Joined: Sun Aug 30, 2009 11:07 am UTC
Location: Wherever parallel dimension I'm currently living in...

### Re: 0882: "Significant"

Having learned some Statistics in my third year of high school, I love this comic and can relate to it. I believe it perfectly shows how journalists and reporters sometimes either twist their reports to make it more interesting or misinterpret them due to not understanding them much.

SirMustapha wrote:
darkspork wrote:GET OUT OF MY HEAD, RANDALL! I finally caved in (no pun intended) and bought Minecraft last Thursday.

And then people don't understand what I mean with pandering.

KShrike wrote:This is the second (i think) comic that he mentions Minecraft. I saw the word in the comic and I was surprised.

Why? Randall's practice of namedropping nerd culture items to cause reactions like the one above is extremely common, and this is the second time he has used Minecraft.

Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

I find it baffling that you are complaining about Randall's small mention of Minecraft in this comic and you're not complaining on the topic of the comic itself. But what I find stranger is that you usually don't bash comics containing scientific, mathematical, or programming elements that normal people usually don't know, and when you do, you usually:

a.) complain on small, unimportant stuff in it,
b.) point out that Randall's overall presentation of the comic is sloppy, or
c.) say that Randall only posts obscure nerdy topics to gain more attention and to make himself (and us XKCD fans) feel smarter.

</trollfeeding>
Magnanimous wrote:I'd have a giant I Tetris piece shoved down my throat, after which my neck will flash and disappear.

ribbonsofnight
Posts: 8
Joined: Wed Apr 06, 2011 1:26 pm UTC

### Re: 0882: "Significant"

SirMustapha wrote:
Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

We all love hearing from perfect trolls

TheDoctor
Posts: 11
Joined: Mon Apr 04, 2011 5:15 pm UTC

### Re: 0882: "Significant"

I loved reading this comic! It was my 30 second break from minecraft. Must destroy creepers.....

Cecilff2
Posts: 114
Joined: Fri Oct 03, 2008 9:10 pm UTC

### Re: 0882: "Significant"

Hooray statistics!

I'm a poisson distribution!

I'm a poisson distribution!

I'm a poisson distribution!

savanik
Posts: 345
Joined: Fri Jan 29, 2010 6:10 am UTC
Contact:

### Re: 0882: "Significant"

Cecilff2 wrote:Hooray statistics!

I'm a poisson distribution!

I'm a poisson distribution!

I'm a poisson distribution!

Hmmm... given the small sample size, I would have to conclude it's still fairly likely that you're not a poisson distribution.
"If it were up to the copyright lobby, owning a pen would be punishable by fines." ---Arancaytar

Cecilff2
Posts: 114
Joined: Fri Oct 03, 2008 9:10 pm UTC

### Re: 0882: "Significant"

I'm totally a poisson distribution.

jozwa
Posts: 144
Joined: Fri Sep 11, 2009 3:16 pm UTC
Location: Finland

### Re: 0882: "Significant"

Wait...

neoliminal
Posts: 626
Joined: Wed Feb 18, 2009 6:39 pm UTC

### Re: 0882: "Significant"

NEWS: Scientist find correlation between acne and minecraft!
http://www.amazon.com/dp/B0073YYXRC
Read My Book. Cost less than coffee. Will probably keep you awake longer.
[hint, scary!]

Angua
Don't call her Delphine.
Posts: 5943
Joined: Tue Sep 16, 2008 12:42 pm UTC
Location: UK/[St. Kitts and] Nevis Occasionally, I migrate to the US for a bit

### Re: 0882: "Significant"

Did the comic change between last night and this morning? I didn't see green jelly beans referenced until the end.
Crabtree's bludgeon: “no set of mutually inconsistent observations can exist for which some human intellect cannot conceive a coherent explanation, however complicated”
GNU Terry Pratchett

Posts: 350
Joined: Wed Oct 08, 2008 1:58 am UTC
Location: Guarding the Super Missile
Contact:

### Re: 0882: "Significant"

I was JUST eating jelly beans not 2 minutes ago

So, the public's attempts to simplify the findings of scientific studies or to use scientific studies to "prove" their hunches, gets in the way of true science?

I remember some news report from a couple years ago saying that studies showed blue M&M's could cure spinal injuries, and further details showed that the blue dye on them had been linked to improved spinal health or something. This reminds me a lot of that, I think they're in the same boat.
And, I think personally I would put the link between flu vaccines and autism in the same boat as well.

ribbonsofnight wrote:
SirMustapha wrote:
Interestingly, he is smart enough to choose his nerd items well, but way too sloppy to double-check his comic and correct the two awful mistakes people have pointed out before publishing it. Come on! He's been on this business for years and still commits such crude mistakes?

We all love hearing from perfect trolls

Welcome to the fora.
He's not trolling, that really is his opinion. Check the past 300+ comic threads, it's mostly similar reactions.
Last edited by SpringLoaded12 on Wed Apr 06, 2011 3:05 pm UTC, edited 1 time in total.
"It's easy to forget what a sin is in the middle of a battlefield." "Opposite over hypotenuse, dipshit."