From: utzoo!henry
Newsgroups: net.misc
Title: Re: Re: ESP & Elementary Probability 
Article-I.D.: utzoo.2848
Posted: Tue Mar  1 16:29:01 1983
Received: Tue Mar  1 16:29:01 1983

   "Most researchers do not claim "truth" based on a single experiment
   unless the probability is extremely small (p < 10e-9 say).
   The usual process is to look at other independent tests,
   calculate the overall statistic, and then make a judgement.
   Most laymen are statistically naive..."

Researchers who follow this procedure are also statistically naive.
It does not matter how many existing results your theory can explain;
if it cannot *predict* new results, then what you are doing is not
science.  It's easy to fiddle with a theory until it explains data you
already know about;  the acid test is new data, not considered in the
formulation of the theory.

Obviously, this causes trouble if you aren't sure what you're after.
There are ways around this, but only the more conscientious parapsy-
chology researchers use them.  Said conscientious researchers are also
seldom heard claiming positive results.

In case people are curious, at least one of said methods is very simple.
You divide your data into two halves along some arbitrary line, agreed
on before the experiment.  (E.g., first n/2 tests and last n/2 tests.)
You put the second half away and don't look at it.  You then examine
the first half in any way you please, looking for correlations.  Once
you have determined what phenomenon appears to be manifesting itself,
you settle on a specific data-analysis procedure and specific criteria
for success and failure.  Then, you dig out the second half of the data
and apply that exact procedure and those exact criteria.  You are not
allowed to fiddle with the analysis method once you have started to
examine the second half.  If you get positive results on the second half,
using the method and criteria derived from the first half, THEN you have
a positive result for the experiment.  If not, you have a negative result
and no further fudging is allowed.

Show me positive results obtained with this method, or some other way
of making sure you don't outwit yourself, and I will be impressed.
Statistical analysis of parapsychology data without such safeguards
is worthless.

					Henry Spencer
					U of Toronto