sourmìlk wrote:But yeah, that process is always fun. The sad thing though is that awareness of the Dunning-Kruger effect doesn't actually seem to diminish it. I was reading on the scientifically rigorous website Cracked.com about a study in which people were asked to rate some characteristic of themselves. Predictably, they all rated themselves above average. When the scientists gathered all the subjects together, told them that this was a mathematical impossibility, and asked if anybody might want to revise their opinions, nobody did.
Without a strong reference to exactly what the subjects were supposed to rate, it is impossible to tell if they were likely wrong. Without explicitly giving a definition of the direction "above average", they are simply asking the individuals if they feel they have remotely optimized these aspects of themselves. To take the classic example of driving, some people's goal is to minimize driving time, others wish to minimize risk of crashing, while still others may consider allowing traffic to maintain flow above all. Each driver sees themselves as "driving correctly" and anybody attempting to maximize some other goal as being "below average". Thus all but a few people consider themselves "above average".
Thus leaving the likely explanation is that the Dunning-Kruger effect has instead infected the scientists, making them feel they have constructed a far better experiment than they have (or they have, but following the classic example of scientist -> university PR flac -> tech journalist "breaking the news" -> hack journalist who re-reported the news you actually saw -> word of mouth [repeat] -> here, have mangled the report beyond all accuracy). [I suspect that experimental design is similar to software design. You can't find the errors in the most clever experiments you can design so you must limit yourself to far more simple experiments. Our scientists may already be out of their depth in this simple thing.]