0882: "Significant"

[Meem1029]

I think it changed very very quickly. I came on at 11:16 (12:16 eastern), so less than half an hour after it was posted and did not see a single thing about green in the first part.

[SirMustapha]

I find it baffling that you are complaining about Randall's small mention of Minecraft in this comic and you're not complaining on the topic of the comic itself.

Eh, the topic is completely bland and absolutely unsurprising, and Randall doesn't show any new twist to it at all; but if I complain about that, the reaction will be violent and ignorant anyway. The only interesting spin given on the story is the 20 nearly identical panels. Humour through repetition can be great, and I don't think Randall has ever tried that. It's the only fresh and interesting thing in the whole comic.

But what I find stranger is that you usually don't bash comics containing scientific, mathematical, or programming elements that normal people usually don't know

Eh, that's the whole schtick of xkcd, right? I don't have anything against Randall finding his own niche, really.

and when you do, you usually:

a.) complain on small, unimportant stuff in it,
b.) point out that Randall's overall presentation of the comic is sloppy, or
c.) say that Randall only posts obscure nerdy topics to gain more attention and to make himself (and us XKCD fans) feel smarter.

Ok, first, I don't agree that the small things are unimportant. When it comes to art, the small things can have a truly huge impact on the finished product -- a badly lighted scene on a film, a poorly recorded instrument on a song, and a stale "off-joke" in a comic? Those things may be small, but not at all unimportant. I hate the "Minecraft" reference. Why is this such a crime? A lot of people complain that Family Guy often just references pop culture items for comedic effect, instead of creating actual substance. Randall is doing the exact same thing here, only with nerd culture items. The joke there is that scientists like to play games in their spare time. It could be any game, it could be a generic game, it could be Tetris or anything. But no: Randall has to go for something absolutely topical, something that will attract attention to the reference itself and not to its effect on the joke! It's detracting, it's jarring, and it's nothing but cheap, obnoxious pandering! It has absolute potential for ruining anything, and it doesn't add any value to the comic. It is small, it is just a detail, but art is made of details. Ignoring the details is one of the most destructive attitudes one can have towards art.

As for the "overall presentation," well, are you willing to argue that the presentation is not important? I'm sorry, but I can't take seriously someone who argues that. A great presentation greatly enhances the effect of a joke, while a bad presentation can muffle the effect to the point of killing it. Besides, I'm not even complaining about the proper sloppiness of the presentation here, but for Randall's sloppiness as the editor and publisher of his own work. Occasional mistakes can be forgiven, but in the last few months, the amount of errors in xkcd was staggering, and are characteristic of an artist who doesn't care about his work enough to proof read it. The "double yellow" thing was minor enough to be passable, but the "green jelly" on the second panel anticipated the joke! No sleep-deprived sloppy editor would have forgiven that. Randall had the attitude of an artist that doesn't care, and that doesn't deserve respect, plain and simple.

As for the attention whoring, this topic has been nearly exhausted already, and it's completely off-topic here. I'll pass that one.

[jakerman999]

The update changed two things as far as a quick glance shows, the first reference to green being removed, and the second yellow being changed(is mauve now?) Do these changes affect the secret code?

[TheInsomniac]

No one who understands anything about statistics would use a p-value of 0.05 for a series of studies like this. It would require a bonferroni correction, which can be done simply by dividing 0.05 by the number of trials to correct for the 1/20 chance of a failure to reject the null.

[DesSidDes]

W00! Bonferroni/Sidak adjustments ftw!

[xkcdfan42]

Looking at the page where I usually read XKCD (his iGoogle gadget), I noticed that the last three cartoons had one-word titles that kind of made sense:

Significant
Probability
Headache

Since I'm not prone to be a conspiracy theorist, I estimate that p < 0.5 for the hypothesis that Randall chose these titles for this purpose, but it's still interesting.

[anthonyjerome]

Some commenters understand the issue and some don't. A simple explanation is, "if I try 20 tests that each have a 5% chance of being true by chance, I should not be surprised if one comes up true, nor make any grand claims from such a result." This is actually a serious stats problem that a lot of professionals get wrong, including in medicine. People overestimate the significance of their results without compensating for the problem of multiple comparisons. Science News ran a feature about this issue in March 2010 called "Odds Are, It's Wrong." The original article is now behind a paywall, but you can find discussion and excerpts by googling that name.

Randall may also have been inspired by this week's Supreme Court ruling about statistical significance, the only time I'm aware of that the high court has considered the meaning of p-values.

Great comic Randall!

[FourTael]

This reminds me of that House episode where they try to run Hep C tests until they get a false positive. Also the one where the senator has a false positive HIV test.

Anthonyjerome: Do you have a link to the majority decision or some news articles on the subject? I didn't hear about that case.

Edit: As an example of how poorly most people understand actual science, my sister quoted a study that found a correlation between college education and atheism and agnosticism. Aside from the problems of grouping those together, she cited that as proof that the more educated a person is, the more likely they are to be atheist or agnostic. This completely throws the idea of causation out the window (college culture, the background of people who go to college in the first place, religious statistics of people of different ages, etc). Of course, I recently read an article that showed a link between college education and belief in paranormal. Of course, they grouped things like psychic "phenomenon" in with things like Bigfoot, so that's another problem right there. Another problem is, of course, the possibility that the study was done incorrectly. Other studies I've read show an opposite correlation.

[RockoTDF]

The sad fact is that far more than 5% of papers published against the rather modest standard of p < 0.05 turn out to be reporting on results due to random chance. This comic explains why.

Most papers today actually report the best p value they can get, as given to them by their software. So yes, p <0.05 is the worst I'll take, but if SPSS, SAS, or R tells me p <0.0001 you bet your ass that is the number I put in the paper. Back in the day people just looked at a table and ran with p <0.05, even if it could be much better than that. For this reason I don't think this comic can argue that more than 5% of papers are published with chance results.

No one who understands anything about statistics would use a p-value of 0.05 for a series of studies like this. It would require a bonferroni correction, which can be done simply by dividing 0.05 by the number of trials to correct for the 1/20 chance of a failure to reject the null.

True, no one would use a ton of t-tests. But the Bonferroni correction is a bad idea because it affects the outcome of *all* of your experiments. Let's say you run 4 experiments that survive your Bonferroni, and realize that you need a fifth. Suddenly, *all* of your results are null because you had to adjust your correction. The Bonferroni correction is way way too conservative to use with more than a few experiments, and especially in the types of studies that generally don't give you miniscule p values.

[Dason]

Since I'm not prone to be a conspiracy theorist, I estimate that p < 0.5 for the hypothesis that Randall chose these titles for this purpose, but it's still interesting.

You use a very large alpha.

[meat.paste]

My first thought was "What is minecraft?". I googled it, and clicked over to their website. Our corporate IT department has blocked access. So, there is a >95% chance the game is worth playing.

[anthonyjerome]

Anthonyjerome: Do you have a link to the majority decision or some news articles on the subject? I didn't hear about that case.

http://online.wsj.com/article/SB10001424052748703712504576235683249040812.html
http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-closeup-1050/

[NumberFourtyThree]

I once read something saying this is a problem for genetic studies where they test hundreds of genes to find if one of them is linked to something.

[Gobo]

Minecraft has had a statistically significant impact on my dissertation at p<0.05.*


*(Actually this is fine. It's just that I don't work on it at home anymore. Work-life balance and leaving your job at the office is important, even if it's a virtual life stacking cubes.)

[FourTael]

Ah, I remembered the logical fallacy related to this comic!

It's called "The Prosecuter's Fallacy." Basically, if you run a DNA test that has a .1% chance of getting a false match against a database of 1,000 people, your chance of getting a false match is not .1%, but rather 63.23%. If you run 20 tests that each have a 5% chance of coming to the wrong conclusion, your chance of getting AT LEAST ONE wrong conclusion is actually 64.15%, not 5%. And, when the scientistis are distracted, annoyed, and want to play Minecraft, well... that probability is probably higher.

Edit: Sorry, The Prosecutor's Fallacy only applies to law. This is actually a base rate fallacy (or at least a cousin of base rate fallacy).

[katylava]

This helped me: http://www.graphpad.com/articles/pvalue.htm

[philip1201]

Maybe I'm failing at logic here, but a 5% chance of your data being caused by something else than your model (which is what p<0.05 means, right?) doesn't mean there's a 5% chance of your model appearing out of random data.

If you have a simple measurement of a model and the projected value falls within a margin of error of 5%, then you can say p<0.05. But that doesn't mean the chance of verifying the model by accident is 5%. If the value can vary between any number, the chance of getting the projected value verified is far less than 5%.

More practically, let's say you have a car crash. Observational data reveals a 5% chance that the driver was exceeding a speed of 200 km/h. That does in no way mean that 5% of car crashes involve drivers exceeding 200 km/h. That's just not statistically true.

The observation that people are guilty of ignoring X amount of evidence and then hail the 1-in-an-X chance as significant is still valid, of course. Even Dr. Manhattan is guilty of it.

[dotancohen]

I can't believe that nobody* has mentioned this yet:
http://www.phdcomics.com/comics.php?f=1174

* Alright, I only read the first page, so it might be buried on the second or third.

[Dason]

Most papers today actually report the best p value they can get, as given to them by their software. So yes, p <0.05 is the worst I'll take, but if SPSS, SAS, or R tells me p <0.0001 you bet your ass that is the number I put in the paper. Back in the day people just looked at a table and ran with p <0.05, even if it could be much better than that. For this reason I don't think this comic can argue that more than 5% of papers are published with chance results.

I don't think your logic is quite right. Just because we can get more accurate p-values doesn't mean that people suddenly use a lower alpha. If they get anything lower than .05 they'll publish. Sure you'll publish what the p-value actually is (p < .0001) but you would have been just as happy to publish if that would have to say (p = .049). So the alpha rate is still .05 and if they're doing multiple comparisons they're probably going to get some results by chance if they don't correct for it.

[Maurits]

Continuing the "wrong color" meme, "grey" is more usually spelled "gray" in the US.

[xmrsmoothx]

"But Miiiiinecraft" made this comic for me.

[roman cilicia]

I highly doubt this situation would unfold IRL, seeing as scientists are not usually autistic and would thusly not be playing minecraft (aka the "aspergers simulator")

[zAlbee]

Maybe I'm failing at logic here, but a 5% chance of your data being caused by something else than your model (which is what p<0.05 means, right?) doesn't mean there's a 5% chance of your model appearing out of random data.

I think you're confused as to what p-value means. As I understand it, it DOES mean the second part of what you wrote, and NOT the first part.

Since it's confusing enough as it is, I'll just link to http://en.wikipedia.org/wiki/P-value and http://www.graphpad.com/articles/pvalue.htm and hope they explain it better.

The p-value is not the probability that a finding is "merely a fluke."
As the calculation of a p-value is based on the assumption that a finding is the product of chance alone, it patently cannot also be used to gauge the probability of that assumption being true. This is different from the real meaning which is that the p-value is the chance of obtaining such results if the null hypothesis is true.

[A_of_s_t]

"But Miiiiinecraft."

Best.

Quote.

Ever.

[smowton]

I like to think of p = 0.05 as meaning:

If I were to step into a universe where there is no link between A and B, and carry out a bunch of experiments investigating a link between A and B, then 5% of them would come back suggesting a spurious link. Come up with a stronger result, the p value falls, and less of my experiments show the nonsense link. Of course what I should really do is amalgamate all the results from my experiments to produce one massive one with much higher certainty that nothing's going on.

[syko_lozz]

I love this. I feel like I am both procrastinating and studying at the same time

So tell me, how do repetitions affect the reliability of the p-value? If I repeat the experiment 3 times and each time I get a p-value <0.05, does this kinda divide the probability of it being due to chance by 3?

[RockoTDF]

I don't think your logic is quite right. Just because we can get more accurate p-values doesn't mean that people suddenly use a lower alpha. If they get anything lower than .05 they'll publish. Sure you'll publish what the p-value actually is (p < .0001) but you would have been just as happy to publish if that would have to say (p = .049). So the alpha rate is still .05 and if they're doing multiple comparisons they're probably going to get some results by chance if they don't correct for it.

Not "more accurate" but "smaller." If all of my p vals are .001 and all of your p vals are .049, you stand a better chance of getting results by chance than I do - in one experiment or 17 experiments. Alpha rate is entirely irrelevant, the p values are what actually matter. If my p vals are .001 and I have an alpha of .05 (corrected to .01 for each experiment by Bonferroni, let's say) my results are still significant. If you correct to .01 you are screwed and I am not. Therefore, in order for 5% of studies to have been by chance, all of those studies would need actual p values around .05, and in order for this to be 1% all studies would need to have a p val around .01. Basically, your argument is saying that the odds of me getting a result by chance depend more on what cutoff I use than on the actual p values themselves. That doesn't make sense.

[xtifr]

A lot of people complain that Family Guy often just references pop culture items for comedic effect, instead of creating actual substance.

Yes, but they don't usually do it day after day after day after day after day. Some people like Family Guy, others don't. The ones that don't usually go do something else instead of watching it. The tiny percentage that insist on nagging their friends and family over and over and over about how "it's not really funny" usually end up without friends or family. Heck, even the friends that share their dislike of Family Guy may get sick of the obsessive whining.

[Dason]

For this reason I don't think this comic can argue that more than 5% of papers are published with chance results.

It was really this sentence that I was refuting. If everybody uses an alpha of .05 still and they don't do a multiple comparisons adjustment then you can make the argument that more than 5% of published papers have results due to chance alone.

[SirMustapha]

Yes, but they don't usually do it day after day after day after day after day. Some people like Family Guy, others don't. The ones that don't usually go do something else instead of watching it.

Yes, they watch something else, and they also lock themselves away in a bubble so they don't hear their classmates and workmates babbling about Family Guy, and they don't read all the commentary on the Internet them, and they don't receive e-mails or Facebook messages pointing to "hilarious" videos of Family Guy on YouTube -- that is, they can isolate themselves perfectly well from the thing they hate so much. Same thing with music: if you don't like Justin Bieber, you can block your ears completely so you will never, not even by accident, hear Justin Bieber again, therefore you will never feel frustration and the need to vent it.

Here's another good one: three men walk into a bar...

EDIT: You think I'm kidding, right? Last week, during a class in college, one of my classmates was wearing an xkcd t-shirt. Yeah, I can't stand the goddamn thing, and even then someone has to rub it on my face. Okay, I can try to live with that. Today, at work, what do I see? A colleague, that is, another person entirely, wearing another xkcd t-shirt. THE GODDAMN THING IS EVERYWHERE.

And that's not all! What is stuck to the back of the door of my workplace? The Exploits of a Mom comic! And who pasted it there? I DID.

[Eternal Density]

Given a large enough sample size, it's pretty much given that someone will have thought or done something related to the latest xkcd, just by random chance. Hence the GOOMHR phenomenon.

Also, are there any members of the Bayesian Conspiracy here?

[Dason]

Also, are there any members of the Bayesian Conspiracy here?

By that do you mean practitioners of Bayesian statistics?

[hofmic]

Since I'm not prone to be a conspiracy theorist, I estimate that p < 0.5 for the hypothesis that Randall chose these titles for this purpose, but it's still interesting.

I estimate p > 0.5 (queue the whoa).

Before I get ranted on, I'll slowly step out of the statistics game... and go back to playing mine craft.

[silent.one]

I apologize for breaking up the wonderful discussion about statistics, but I felt like this had to be posted.

http://i.imgur.com/YKTLf.png/ Notch's reaction to this comic (Notch created Minecraft).

I just screen capped this, so you can visit his twitter page to verify. That's all, have a good evening/morning/afternoon.
(And yes, i created a forum account just to relay this)

[weckar2]

This came out on the day of my final stats exam, after which I spent the whole night playing minecraft.... Then in the morning I saw this comic.... Wow...

[biodomino]

No real experiment exists in isolation, and if your analytical techniques do not provide ways of accounting for prior knowledge, you are using the wrong analytical techniques.

Well, that's a fine ideal, only there's a ton of original research for which there are either no prior quantitative figures to apply, or it would be inappropriate to do so. Particularly in the social sciences, accounting for prior knowledge in your analysis (outside of a mention in a literature review or discussion section) is often not practical or even possible.

Realistically, the p value should be selected according to your data and hypotheses. There are situations where a p of .2 is appropriate, and situations where .01 is too lenient. However, in this case it's not necessarily fallacious to fail to correct using Bayesian analysis or other methods when considering that green jellybeans could contain an agent that other jellybeans don't.

[StClair]

Ladies and gentlemen, I present to you SirMustapha, our resident Stop Having Fun Guy.

[NeoThermic]

I wonder if this takes the prize for most panels in an XKCD comic (I know there have been comics larger than this, but I am refering specifically to the number of panels)

The 2008 Christmas Special had more panels, though a lot of them were just black.
XKCD Loves the Discovery Channel had the same number of panels as this comic, and far less repetition.

If we're not letting the 2008 Christmas special win due to the blank panels, then The Man Who Fell Sideways is the winner, larger by a whole extra row over XKCD Loves the Discovery Channel!

[Ezbez]

A great read which is loosely correlated with this comic is the recent New Yorker article The Truth Wears Off. It's a scary article for all us science folks who place great weight on the findings of well-executed studies.

[philip1201]

A great read which is loosely correlated with this comic is the recent New Yorker article The Truth Wears Off. It's a scary article for all us science folks who place great weight on the findings of well-executed studies.

Reminds me of this xkdc strip - Nature doesn't want to be perfectly known, so there's a conservation of knowledge of reality. We try to find the Higgs boson, so all other observations are becoming less true.
I do wonder if that effect is limited to the human-related sciences.