Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10.2 9/18/84; site LaBrea.ARPA
Path: utzoo!watmath!clyde!bonnie!akgua!whuxlm!harpo!decvax!decwrl!Glacier!LaBrea!mann
From: mann@LaBrea.ARPA
Newsgroups: net.physics
Subject: Re: Parapsychology Experiment Now
Message-ID: <123@LaBrea.ARPA>
Date: Sat, 6-Jul-85 00:37:14 EDT
Article-I.D.: LaBrea.123
Posted: Sat Jul  6 00:37:14 1985
Date-Received: Fri, 12-Jul-85 02:48:45 EDT
References: <2896@decwrl.UUCP>
Organization: Stanford University
Lines: 23

> The highest score over the 95% confidence level will be awarded a
> tasteful brass plaque.
 
As I'm sure you recognize, if you get 100 responses, the expected number of
scores that are significantly high at the 95% confidence level is 5, even if
no one has any ESP at all.  So I hope you have a good source for a cheap (but
tasteful) brass plaque -- you'll be awarding it.

More seriously, this points up a real pitfall that many ESP researchers have
fallen into.  They know enough about statistics and scientific practice to
get excited about a result that is "significant at the 95% level", and even
more excited about one that's significant at the 99% level.  So they do
hundreds of experiments, and test each one's significance individually.  Lo
and behold, about 1 in 100 experiments is "significant" at the 99% level, so
they report it.  Of course, this is just what would be expected by chance.
"Significant at the 99% level" means there is only 1 chance in 100 of the
given outcome having occured entirely by chance.  If most or all your
experiements are significant at this level, or (far better) if all your
experiments taken together are significant at this level, you are really 
on to something.  If only 1 in 100 is "significant", you have merely
observed the laws of probability in action.

	--Tim