Is All Evidence Statistical?

[tomandlu]

My son and I were having a (friendly) dispute over whether all evidence for a theory is essentially statistical. He maintains that it is, even if the sample-size required is only 1 (e.g. the 1919 eclipse proving Einstein was correct). Whilst I concede that he has a point, this strikes me as making the category of 'statistic' so broad that it becomes meaningless.

Is there a simple answer to this debate?

[Zohar]

I don't know if the term "statistical evidence" is a thing. Ideally, all theories would be supported by empirical (i.e. measured) evidence. In many cases we'd want to perform statistical analysis to see if the results we have are statistically significant.

[tomandlu]

I don't know if the term "statistical evidence" is a thing. Ideally, all theories would be supported by empirical (i.e. measured) evidence. In many cases we'd want to perform statistical analysis to see if the results we have are statistically significant.

Wouldn't something like smoking and cancer rates be statistical evidence of correlation, and, assuming the correct controls were in place, evidence of causation as well?

Also (now disproved), wasn't the fact that all snails found had the same orientation of spiral taken as statistical evidence that it was true for all snails? And all it took was a sample of 1 to kill that theory. I guess this is where my dispute with the idea lies. That sample of 1 could have been a sample of a 1,000 - it would still have had the same effect. I find it hard to contemplate something being statistical when the sample size has no relation to confidence.

[doogly]

Let's say all evidence for a theory is statistical, but evidence against can be simple counterexamples?

[Zohar]

There's also a difference between what we consider sufficient evidence in physics compared to medicine. If I let go of a stone 10,000 from the top of a building and each time it falls straight down to the ground, but the 10,001 time it just drifts down like a feather, that's very different from medication working the same 10,000 time and being different the 10,001 time (that would, in general, be considered a pretty effective medicine actually).

[doogly]

Though the equivalent for the medicine would be more like, the 10,001th patient turns into a frog.

[Zohar]

Yeah I found it difficult to find a good example for physics that's clear...

[cyanyoshi]

Though the equivalent for the medicine would be more like, the 10,001th patient turns into a frog.

Side effects may include headaches, nausea, ... , and turning into a frog.

[gmalivuk]

My son and I were having a (friendly) dispute over whether all evidence for a theory is essentially statistical. He maintains that it is, even if the sample-size required is only 1 (e.g. the 1919 eclipse proving Einstein was correct). Whilst I concede that he has a point, this strikes me as making the category of 'statistic' so broad that it becomes meaningless.

Is there a simple answer to this debate?

A single observation during one eclipse didn't "prove Einstein was correct", first of all, because there was so much more to his theory than that.

Secondly, there's still some statistics and probability involved in even saying it proved one part of General Relativity. Measuring the position of a star has a margin of error (which can be calculated by doing statistics on a large number of similar measurements) and you could call that particular observation a hypothesis test, where given that margin of error, the measurement would be far less likely if the actual position was where Newton said it would be than if it was where Einstein said it would be.

Let's say all evidence for a theory is statistical, but evidence against can be simple counterexamples?

Isn't this basically the definition of falsifiable theories? That a counterexample would falsify the theory.

Badly formulated theories, such as the "theory" that Russel's teapot exists, are provable with one example but only disprovable statistically.

[tomandlu]

Badly formulated theories, such as the "theory" that Russel's teapot exists, are provable with one example but only disprovable statistically.

Would you define (or consider it reasonable to define) that one example as a statistic?

[gmalivuk]

No.

It's an inversion of doogly's statement. Evidence for falsifiable theories is statistical, but a single (non-statistical) counterexample could falsify it. For certain types of non-falsifiable theories, it's the opposite.

[doogly]

But I do think that ultimately even the counterexamples are statistical, because I am going to do a Bayes to them. If someone sense "Dude, this particular vial of aspirin turned me into a newt! ...then I got better." I am going to not overhaul my theory of what aspirin does at all, because of a Bayes. If they take another pill and turn back into a newt to show me what for, I am probably just going to highly doubt that it was aspirin, rather than modifying my theory of what aspirin does.

[Thesh]

"Asprin linked to shapeshifters reverting to reptilian form."

[doogly]

But then really, what reason do you have for using statistics?

At the root, all evidence is aesthetic. Tomandlu, I promise your kid isn't too young for Feyerabend.

[tomandlu]

But then really, what reason do you have for using statistics?

At the root, all evidence is aesthetic. Tomandlu, I promise your kid isn't too young for Feyerabend.

Ah, we've moved on now to Panpsychism (fwiw we both think it's reasonable).

[EDIT] That aside, I shall pass on the reference. He's juggling at the moment, but heading towards maths and philosophy at university. That said, he's very absolute by virtue of nature and age, so I suspect that Feyerabend might appeal more to me than him (yay, another debate!).

[gmalivuk]

But I do think that ultimately even the counterexamples are statistical, because I am going to do a Bayes to them. If someone sense "Dude, this particular vial of aspirin turned me into a newt! ...then I got better." I am going to not overhaul my theory of what aspirin does at all, because of a Bayes. If they take another pill and turn back into a newt to show me what for, I am probably just going to highly doubt that it was aspirin, rather than modifying my theory of what aspirin does.

True. One genuine counterexample would falsify a theory, but in practice you still need to do statistics to decide if it really is a counterexample.

[Soupspoon]

(Blatantly ripping the following off of a random website, to avoid having to type it from memory...)

An engineer, a physicist, and a mathematician (and others) were on a train heading north, and had just crossed the border into Scotland.

The engineer looked out of the window and said "Look! Scottish sheep are black!"
The physicist said, "No, no. Some Scottish sheep are black."
The mathematician looked irritated. "There is at least one field, containing at least one sheep, of which at least one side is black."
The statistician proclaims, "It's not significant. We only know there's one black sheep"
The computer scientist, is taken aback. "Oh, no! A special case!"

[tomandlu]

BTW does anyone recall, or have a reference for, the old-time quote about it being hard to draw the absolute line between night and day, and yet it being tolerably easy to tell the two apart in general?

The Google has failed me so far, but I'm sure I'm not making it up.

[Eebster the Great]

The reference I heard was basically equivalent. It's hard to draw a bright line between hot and cold weather, but we can still all agree that 0 C is cold and 40 C is hot. It's useful for lots of cases. For instance, there was no "first human," even though there are humans now, there was a time when there were no humans, and apes are born one at a time. Similarly, it is neither true that "the chicken came before the (chicken) egg" nor that "the (chicken) egg came before the chicken," even though all chickens come from eggs, all (chicken) eggs come from chickens, and neither existed fifty million years ago.

This also seems like a reasonable response to the "paradox of the heap."

[jewish_scientist]

Karl Popper proved that empirical evidence can increase the probability that a statement is true, but can never conform it. For example, if I discover 9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999 white swans, then the probability that the statement 'All swans are white,' is very high. However, it is still not 1 because there could still be a black swan somewhere that I have not found yet. The field of mathematics used to calculate exactly what the probability that a statement is correct is called statistics.

It is worth pointing out that Popper was only addressing empirical evidence, meaning evidence based on observations. Non-empirical evidence (e.g. formal logic) can prove a statement correct.

[doogly]

Karl Popper proved

oh did he now

[Eebster the Great]

Actually, Popper alleged that corroborating a theory by failing to falsify it did not increase the probability that it was true. He was not after truth. He was just after effective theories. The reason was that the inductive validation model used by positivists relies on the fallacy of affirming the consequent, whereas the deductive falsification model relies on modus tollens.

There are some issues with falsificationism to be sure, but I think it was better than the positivist ideals that came before it.

[Quizatzhaderac]

@tomandlu
Statistics is a set of techniques.

It is a technique that can be applied to any evidence, but is not necessarily applied to analyze all evidence. I see three cases:

1. The evidence is gathered with intend to subject it to statistical analysis.
2. The evidence would benefit from statistical analysis.
3. You could subject the evidence to a statistical framework, but doing so yields no benefit.

An example of three: would be one kissed frog turned into a prince, zero kissed frogs failed to turn into princes, and zero other frogs observed during evidence gathering.

We could apply the Laplace–Bayes estimator, which would tell us that there is a 2/3 chance kissing a frog will make it a prince, but that number is fairly arbitrary and really the only conclusion we can draw is "kissing frogs can turn them into princes, but I haven't done a scientific study"

But I do think that ultimately even the counterexamples are statistical, because I am going to do a Bayes to them. If someone sense "Dude, this particular vial of aspirin turned me into a newt! ...then I got better." I am going to not overhaul my theory of what aspirin does at all, because of a Bayes. If they take another pill and turn back into a newt to show me what for, I am probably just going to highly doubt that it was aspirin, rather than modifying my theory of what aspirin does.

True. One genuine counterexample would falsify a theory, but in practice you still need to do statistics to decide if it really is a counterexample.

Well, also in this case even after we were sure that we found a counter-example, our working theory afterwards of what aspirin does would include "It very rarely changes whomever took it into a frog"

[Shufflepants]

All evidence is inherently statistical because it took statistical inference to learn anything at all. When you first start thinking, whenever that may be, the only thing you know is whatever biologically determined heuristics evolution has programmed into us. From that point, everything has to be inferred. Even the fact that things you see don't just up and cease existing the moment you stop seeing them is a statistically inferred fact. Really young babies don't have object permanence. They have to learn through many examples of things appearing, disappearing, coming back, and sometimes not coming back that things don't generally cease to exist the moment you stop looking at them. Thus any piece of evidence you might have relies on (among probably many others) this base statistical inference. After all, maybe the evidence you thought you had ceased to be the second after it happened or maybe it never really happened at all.

At the root of this, is of course uncertainty. If there's any possibility of uncertainty, you've got a statistical problem on your hands. As pointed out by Kierkegaard, even math and logic have some amount of uncertainty to them insofar as maybe there's a math genie who is warping all of reality at all times to make you think 2+2=4. Or maybe you're really just a Boltzman Brain. Now, you can say we can just assume that these sorts of problems are not the case and then some things become non-statistical. But that's just an assumption (which has a very high statistical likelihood of being true).

It certainly isn't often a useful distinction, and it's very useful most of the time to pretend like it's not true as a time saving heuristic, but I don't see any way around it unless you believe in some mystical 'soul' that can "Know" things with absolute certainty and achieve capital T Truth.

But is it really so bad operating from a point of view where many of the things we take as set in stone, irrefutable facts are really just things with a 99.9...a few hundred more 9's...999% chance of being correct?