What-If 0002: SAT Guessing

[BAReFOOt]

What if everyone who took the SAT guessed on every multiple-choice question? How many perfect scores would there be?

The whole thing reminds me of that thing Einstein supposedly said, that if only engineers would do research, we would now have perfectly working oil lamps.
The whole SAT thing builds upon the wrong assumption, that you should grade the answers that students are giving.
While in reality, what’s important, is that you understand the topic. And the answers are “surprisingly” unrelated to that.

The best teacher I ever had, wrote all the answers on the chalkboard. He even gave good grades when you had wrong answers.
Instead, he looked at the approach you took. The understanding you had. And since he was really good at making people understand, and giving things a purpose, classes were always fun and educating. It was the only class I really looked forward to, like the next episode of a good show. (He even explained mathematics to us. Something our math teacher failed horribly at, because she did teach math like teaching a language at a Chinese factory. Without any fun or practical application.)

So the whole discussion about the SAT is pointless.
Because right now, it’s designed to create memorizing serfs. The same drones you get at a phone hotline, that seem to only have a fixed set of input patterns they can react to. With a fixed set of pre-programmed functions. Critical thinking… Focusing on generalized concepts instead of memorizing lists of cases… Let alone finding your own solutions… and building your own sense of reality, right and wrong… all those things are nowhere to be found.
That system can’t be fixed. It is FUBAR.

The system must change as a whole.
First of all, it must be built with games. The mother of all education, art, sports and entertainment. Games that are fun and motivating. And revolve around becoming good at what you’re supposed to learn. And I don’t mean “just sit in front of your computer, alone”. I mean augmented reality… with the whole class… and teachers and parents in new roles. And everyone looking forward to them, because they are that awesome, inspiring and useful.
Games. That is a non-negotiable condition.

But of course this won’t happen, as long as we got shit like this epic failure coming from those who make the decisions:
Texas GOP Educational Platform Opposes Teaching Critical Thinking Skills

[gorcee]

Are you arguing that the math in the "What If" is wrong, or that the fundamental premise underlying the SAT is flawed?

In the latter case, does that really have anything to do with What If?

Third, your whole argument regarding games is naive, and in places wrong. Read Glued to Games by Scott Rigby, if you're really interested in applications of serious gaming.

Finally, there's a whole thread on standardized testing in the School forum.

[Роберт]

If you didn't use critical thinking skills on the SAT, I wonder how you scored...

[david_h]

My quibble with this What If is this bit:

it’s a statistical certainty that there would be no perfect scores on any of the three sections.

How is a statistical certainty different from an actual certainty (which this isn't)? Is there a particular definition for this phrase?

[Noc]

actual certainty

To answer your question, you can start by defining this phrase. Assuming that you have determined that something is an "actual certainty," how have you arrived at this conclusion?

You could conclude that something is a "statistical certainty" by running the numbers and determining that the probability of your event not occurring is negligible. Or you could conclude that something is a "logical certainty" by determining that your event must logically follow from the given premise. I suppose you could also conclude that something is an "empirical certainty" simply by consulting the available evidence and noting that the event occurs in every documented case. (Though this method admittedly involves the use of induction, which makes it mutually exclusive to being logically certain blah blah blah arguments beyond the scope of this discussion.)

But what process do you use to conclude that something is an "actual certainty?" Once you answer this question, the difference between it and the above should be fairly obvious.

[amennen]

"1/2.7∗10^110 (That's one in twenty-seven quinquatrigintillion.)"

Nope. Quinquatrigintillion looks like it probably means 10^(3*35 + 3) = 10^108, so 2.7*10^110 would be two hundred seventy quinquatrigintillion.

[Zanzaben]

The math on this is wrong depending on how you define a perfect score because the SAT is curved based on how hard it is. Because of this if every person where to guess it at random than the average would be around a 20% which is way lower than normal so the curve would come into effect then to make that 20% more like real normal. About .00002% of students get a perfect score each year (quick google search) and that number would remain the same in theory so the real question is what is the probability that any 1 student 99.99998% better than all the other students that averaged a 20%.
These are rough numbers and actual research would have to be done about the specifics of the SAT curve to get an exact answer

[david_h]

What I meant by "actual certainty" was really just "certainty." I was only using the word "actual" to distinguish it from "statistical certainty," the definition of which I'm still unclear on - unless the word "statistical" is used here to mean "almost, but not quite a"

No-one getting a perfect score is definitely not an absolute certainty, because it is possible - but extremely unlikely.

[Noc]

Getting a perfect score on the SAT isn't all that unlikely at all! Getting a perfect score on the SAT by guessing randomly is, which is the point of the article.

You still haven't answered my question: how do you determine that something is just a regular certainty? Do you just know?

A statistical certainty is something that you're certain about because of statistics. As I mentioned above, a logical certainty is something that you're certain about because of logic. If you want me to tell you how these are different from a "regular" certainty, you're going to have to qualify for me how you became "regularly" certain about something.

[SurgicalSteel]

Getting a perfect score on the SAT isn't all that unlikely at all! Getting a perfect score on the SAT by guessing randomly is, which is the point of the article.

What article? The one about Texas?

[Noc]

This article, which the OP was in response to.

[SurgicalSteel]

Ooh, thanks. There should probably be a link in the OP.

[gmalivuk]

There is now. It's just unfortunate that the first post had to be an uninformed standardized testing rant rather than actual commentary on the article.

[Turing Machine]

I think it's hilarious someone thinks the SAT can be gamed by memorization.

People with low IQs tend to hate things that correlate strongly with IQ. Damn you, g-factor! :shakes fist:

[Pfhorrest]

You still haven't answered my question: how do you determine that something is just a regular certainty? Do you just know?

Determining whether something is a member of a set is different from defining that set. For a topical example: we have a clear definition of what a Higgs Boson would be, given that that name is an invented term used to describe a mathematically precise hypothetical entity and not some loose natural language term. We've recently found something which... might be one. We know for certain what one would be, but we're still working on determining whether what we've found is one or not.

A certainty simpliciter (an "actual certainty" or "regular certainty" if you like) is something of which there is no doubt; or if we're talking about presuming you have and understand the relevant information, we might say "of which there can be no doubt", given that information. If you want to get technical it is the De Morgan dual of epistemic possibility; something is certain if its negation is not epistemically possible, if you know, for the strongest sense of that term, that it couldn't not be so. There is 0 (an absolute zero, 0.00 repeating) epistemic probability that it is not the case.

By contrast, a perfect score on the SATs by guessing is possible, logically, metaphysically, physically, epistemically, every kind of possible. It is just exceedingly improbable. Its probability is non-zero, and it is therefore not strictly impossible; but that probability is only negligibly larger than zero.

What I believe is being asked is, does "statistical certainty" hinge on an event's epistemic probability crossing a threshold of a specific proximity to zero, or is something "statistically certain" if its epistemic probability is "negligible" for some unspecified meaning of that term?

To my knowledge the answer is the latter, but I'm not a mathematician so I'd like to know if there is a mathematical meaning of "statistical". (I do know there is a mathematical meaning of "almost" which seems like it might be relevant here).

[allauthors]

By contrast, a perfect score on the SATs by guessing is possible, logically, metaphysically, physically, epistemically, every kind of possible. It is just exceedingly improbable. Its probability is non-zero, and it is therefore not strictly impossible; but that probability is only negligibly larger than zero.

I presume that there have been no other comments for 2 hours because this comment is filled with so much win that there is no point in any further comments. However, I will comment further because I feel compelled to point out this amazing win (I was going to post essentially this, but Pfhorrest said it better than I possibly could have), thus negating my point about the lack of point and thus making this comment pointless (which creates an unending self-referential logical loop).

[Fire Brns]

The link was particularly cruel near the end.

(Everyone survives but Alan Tudyk and Ron Glass)

[Arariel]

I believe the SAT is adjusted based on raw score distribution, though (how well other people score). Miss one question on reading and you can still get 800, but miss one on maths you might get 770.

[david_h]

You still haven't answered my question: how do you determine that something is just a regular certainty? Do you just know?

A certainty simpliciter (an "actual certainty" or "regular certainty" if you like) is something of which there is no doubt...

By contrast, a perfect score on the SATs by guessing is possible, logically, metaphysically, physically, epistemically, every kind of possible. It is just exceedingly improbable. Its probability is non-zero, and it is therefore not strictly impossible; but that probability is only negligibly larger than zero.

Exactly. My point is it's not certain that no-one will get all the questions right by any strict definition of certain. Therefore "statistical certainty" must be something else, and I'd like to know what it is. Does it mean "more likely to happen than not"? That would make a lot of things statistically certain. p<0.01? 0.00001? 0.0000000000000001?

David

[Alltat]

Exactly. My point is it's not certain that no-one will get all the questions right by any strict definition of certain. Therefore "statistical certainty" must be something else, and I'd like to know what it is. Does it mean "more likely to happen than not"? That would make a lot of things statistically certain. p<0.01? 0.00001? 0.0000000000000001?

"Actual" certainty is usually used when something can't happen. Generally speaking; there's no formal definition.

Statistical certainty is when it won't happen. The actual probability required to make such a statement depends on the context. It could still happen, logically speaking, but it will not. Not now, not ever.

It's like turning your computer on in the morning and finding that cosmic radiation has imprinted the complete works of Shakespeare in .pdf format onto your hard drive and placed a convenient shortcut to it on your desktop. It could happen. But it won't. You can claim this with statistical certainty, and you could dismiss anyone arguing against as an idiot too stuck in hypothetical scenarios to see the real world in front of them.

[blowfishhootie]

You still haven't answered my question: how do you determine that something is just a regular certainty? Do you just know?

A certainty simpliciter (an "actual certainty" or "regular certainty" if you like) is something of which there is no doubt...

By contrast, a perfect score on the SATs by guessing is possible, logically, metaphysically, physically, epistemically, every kind of possible. It is just exceedingly improbable. Its probability is non-zero, and it is therefore not strictly impossible; but that probability is only negligibly larger than zero.

Exactly. My point is it's not certain that no-one will get all the questions right by any strict definition of certain. Therefore "statistical certainty" must be something else, and I'd like to know what it is. Does it mean "more likely to happen than not"? That would make a lot of things statistically certain. p<0.01? 0.00001? 0.0000000000000001?

David

It means, "so incredibly unlikely that Monroe feels comfortable asserting that it will never happen."

There are very, very few things that have an actual probability of zero, because they are possible in some theoretical sense. You are being absurdly super-literal when it's not really necessary. The article very clearly explains that there is a greater-than-zero chance of it happening, so I'm not sure how much more of an explanation you could possibly require to understand the point.

[Pfhorrest]

As I said earlier, I believe the point of his question was "is there some specific criterion that constitutes 'statistical certainty', or is it just a rough statement of an undefined extremely high probability?"

I believe his differentiation from "actual certainty" was just an observation of the fact you note that it is not literally certain, as in, there is a nonzero probability of it not happening. If there is some chance it might not be so, then it is not literally "certain" in the ordinary use of that term. So "statistically certain" might just be hyperbole (as in, so likely it may as well be certain), or it might be some specific mathematical term. That is the question.

Other people got hung up on the "what do you mean by 'actual certainty'" thing (when the meaning of that should have been patently obvious), and ignored the real question: is 'statistical certainty' a term of art with a precise meaning or does it just mean "exceedingly likely"?

"Almost every member of this set" sounds like innocuous enough English too, but turns out to have a specific precise mathematical meaning besides the rough ordinary meaning of "almost", so it wouldn't be surprising at all if "statistically certain" did too.

The point is not to harp on "but it's not really certain!" but to ask, given the obvious fact that is is not literally certain, whether the phrase "statistically certain" means something special or is just ordinary hyperbole.

[blowfishhootie]

As I said earlier, I believe the point of his question was "is there some specific criterion that constitutes 'statistical certainty', or is it just a rough statement of an undefined extremely high probability?"

I believe his differentiation from "actual certainty" was just an observation of the fact you note that it is not literally certain, as in, there is a nonzero probability of it not happening. If there is some chance it might not be so, then it is not literally "certain" in the ordinary use of that term. So "statistically certain" might just be hyperbole (as in, so likely it may as well be certain), or it might be some specific mathematical term. That is the question.

Other people got hung up on the "what do you mean by 'actual certainty'" thing (when the meaning of that should have been patently obvious), and ignored the real question: is 'statistical certainty' a term of art with a precise meaning or does it just mean "exceedingly likely"?

"Almost every member of this set" sounds like innocuous enough English too, but turns out to have a specific precise mathematical meaning besides the rough ordinary meaning of "almost", so it wouldn't be surprising at all if "statistically certain" did too.

The point is not to harp on "but it's not really certain!" but to ask, given the obvious fact that is is not literally certain, whether the phrase "statistically certain" means something special or is just ordinary hyperbole.

Regarding at which point something can be said to be "statistically certain," as has been stated, it should be plainly obvious that there is no hard-and-fast answer to that question, as it will depend on specific contexts. Also, it is irrelevant to the point of the article, which is why the author didn't take the time to address it. It just doesn't matter. "Statistical certainty" as it is used in this article is just this person's interpretation of this specific scenario at this specific time. That's all.

And I disagree totally with your claim that this falls outside the ordinary uses of the word "certain." If someone wants to be obnoxiously literal for the sake of being argumentative, which is what is happening here, then almost nothing is actually "certain," ergo virtually all uses of the word "certain" are about things that David would argue aren't actually so.

[SerMufasa]

The link was particularly cruel near the end.

(Everyone survives but Alan Tudyk and Ron Glass)

I originally misread it as everyone dies but Alan Tudyk and Ron Glass and that made me happy. Then I reread it and I was sad.

[marsman57]

SAT guessing was a huge disappointment after how cool Relativistic Baseball was.

[blowfishhootie]

SAT guessing was a huge disappointment after how cool Relativistic Baseball was.

I thought the exact opposite. I thought this one was interesting and at least a (very) slight bit relevant. There are people who guess blindly on all the questions on standardized tests. Not everyone, and not even close to enough to make the question realistically applicable, but it was still interesting to me to know what a person's chances of doing outstanding when guessing on the exam are.

On the other hand, a billion years from now, there will still have never been a human who could throw a baseball anywhere near the speed of light (you might say it's a ... certainty), so it just struck me as kind of pointless. Pointless stuff can still be cool, but this wasn't that for me.

[rmsgrey]

I suspect there's no formal definition of "statistical certainty" - it's closer to a mathematician's way of saying (after calculating the odds) "if that happens, I'll eat my hat!"

For the SATs, I'm happy to call it a statistical certainty that no human being will ever get a perfect score on the multiple choice sections just by guessing uniformly at random from among the available options (so long as the format of the papers doesn't change significantly).

[Роберт]

If it's significantly more likely that you're actually in a Matrix-like environment being fed lies than it is for you to get all the questions right by random guessing, it's sensible to say that it's impossible to get all the question right by random guessing.

[mikewhite]

I think a major point missed by the article is that the SAT is not scored on a completely linear scale, and a perfect score does not correlate with having all the answers correct. This flaws the whole article, it's answering a different question (a much easier one I might add) than originally asked.

Several factors are taken into account, not just how many questions an individual got correct but also how everyone else preformed. I think it would have been interesting to investigate how the bell curve for the random scores would be fitted given random answers (would it be a Normal distribution by central limit thm?) , what is the probability is that an individual would be given a perfect score (1600 or whatever it is) given the random answers, and then also suppose everyone else in the world only supplied random answers to the test, what would I need to have scored in the normal SAT test to have achieved a perfect score in that scenario (I'd imagine anything greater than a 1000 in normal conditions would probably get you there, but it would be fun to learn the actual answer!).

I feel this was a halfhearted attempt at the question at best, but given my background in Math and statistics I probably hold it to higher standards. Also it would probably take a whole days worth of work by a both intelligent and motivated person to get all that info, if it was even possible to get in the first place.

I know people have said this earlier in the thread, but I thought I'd flesh it out with suggestions of what I would have wanted to see. Sadly I'm too busy with work to hunt down the answers myself...

[JudeMorrigan]

I believe the SAT is adjusted based on raw score distribution, though (how well other people score). Miss one question on reading and you can still get 800, but miss one on maths you might get 770.

I can personally verify this. I missed one on each section, which netted me an 800 verbal and 790 math score.

[firechicago]

I believe the SAT is adjusted based on raw score distribution, though (how well other people score). Miss one question on reading and you can still get 800, but miss one on maths you might get 770.

I can personally verify this. I missed one on each section, which netted me an 800 verbal and 790 math score.

Even weirder: when I took the test (just over ten years ago now) I got either 2 wrong on the math section and got a 770, but on my particular version of the test, if I'd gotten only one wrong I would have had an 800.

[Qbr12]

I thought it was interesting.

I don't come to XKCD looking for every comic or feature to be a profound and new insight into the world; I come to see the funny pictures, and if I see something truly profound I'll simply be even happier. This one fell into the "funny pictures" catagory.


Also, did anyone else notice that some of the "re:" labels of the response posts in this thread say "What it" instead of "What if"? It's been giving my inner OCD a pain in the ass.

[blowfishhootie]

I think a major point missed by the article is that the SAT is not scored on a completely linear scale, and a perfect score does not correlate with having all the answers correct. This flaws the whole article, it's answering a different question (a much easier one I might add) than originally asked.

This is just a *bit* of an overreaction. The conclusions of the article would be the same. If you want to figure it out and prove otherwise, I stand to be corrected, but without some numbers I'm not inclined to think accounting for this weighting of performances would change the numbers by anywhere even remotely close to a significant amount. Certainly not significant enough to suddenly make it likely to ever, ever happen.

[Adam H]

I think a major point missed by the article is that the SAT is not scored on a completely linear scale, and a perfect score does not correlate with having all the answers correct. This flaws the whole article, it's answering a different question (a much easier one I might add) than originally asked.

This is just a *bit* of an overreaction. The conclusions of the article would be the same. If you want to figure it out and prove otherwise, I stand to be corrected, but without some numbers I'm not inclined to think accounting for this weighting of performances would change the numbers by anywhere even remotely close to a significant amount. Certainly not significant enough to suddenly make it likely to ever, ever happen.

Yeah, even if you're 1,000,000 times more likely to get a perfect score because of the curving, it's still roughly the same odds as "every living ex-President and every member of the main cast of Firefly all being independently struck by lightning on the same day, except Bill Clinton.

Maybe. I'm not too good at the mathematical things.

[gmalivuk]

Yeah, even if you're 1,000,000 times more likely to get a perfect score because of the curving, it's still roughly the same odds as "every living ex-President and every member of the main cast of Firefly all being independently struck by lightning on the same day, except Bill Clinton.

Yeah, pretty much. Since each lightning strike is on the order of 1/1,000,000, just removing one of the strikes counters that increase in odds.

[VectorZero]

The link was particularly cruel near the end.

(Everyone survives but Alan Tudyk and Ron Glass)

I originally misread it as everyone dies but Alan Tudyk and Ron Glass and that made me happy. Then I reread it and I was sad.

I too misread the stinger. My mind demands justice for Wash and Book.

[conlinism]

Correct me if I am wrong, but wouldn't the probably of guessing them all correctly be 1/5? Because the questions are independent of one another, so the probability of each is 1/5. The probability would not decrease as you progress through the questions...

[jpers36]

Correct me if I am wrong, but wouldn't the probably of guessing them all correctly be 1/5? Because the questions are independent of one another, so the probability of each is 1/5. The probability would not decrease as you progress through the questions...

Each individual question's probability does not decrease. But each one affects the probability of getting them all right. Do you really think that getting 44+67+47 questions randomly right is just as likely as getting one question randomly right?

[ivnja]

Correct me if I am wrong, but wouldn't the probably of guessing them all correctly be 1/5? Because the questions are independent of one another, so the probability of each is 1/5. The probability would not decrease as you progress through the questions...

On the first question, 1/5 of the respondents will guess correctly. That eliminates 4/5 of the respondents from the possibility of having a perfect score. On the second question, of the 1/5 that are still eligible for a perfect score, only 1/5 will guess correctly (that's 1/25 of the original pool). 1/5 of the incorrect respondents from the first question will also guess correctly, but they've already been eliminated. On the third question, only 1/5 of the still-eligible 1/25 will guess correctly, so now we're down to just 1/125, and so on as the number of questions increases.

[blowfishhootie]

Correct me if I am wrong, but wouldn't the probably of guessing them all correctly be 1/5? Because the questions are independent of one another, so the probability of each is 1/5. The probability would not decrease as you progress through the questions...

If you have two questions, each with five possible answers, there are 25 possible combinations of answers:

Q1: A, Q2: A
Q1: A, Q2: B
Q1: A, Q2: C

...

Q1: E, Q2: C
Q1: E, Q2: D
Q1: E, Q2: E

Only one of these combinations can get both questions correct, so there is a 1/25 chance of getting both multiple choice questions right. If the entire quiz is made up of multiple choice questions with five answers, then the odds of getting them all right is 1/(5^n) where n is the total number of questions.