Yes that is more precise. In my post to the query I only noted that the variance in significance levels across multiple permutation tests decreases as the number of iterations increases. Joe's post provides the equation for the expected value of that variance; mine provided reference to an empirical example (Adams and Anthony, 1996).
Dean Dr. Dean C. Adams Professor Department of Ecology, Evolution, and Organismal Biology Department of Statistics Iowa State University www.public.iastate.edu/~dcadams/ phone: 515-294-3834 -----Original Message----- From: R-sig-phylo [mailto:r-sig-phylo-boun...@r-project.org] On Behalf Of Joe Felsenstein Sent: Monday, June 8, 2015 1:29 AM To: Dennis E. Slice; r-sig-phylo mailman Subject: Re: [R-sig-phylo] [MORPHMET] Re: Stability of p-values (physignal and testing for morphological integration) A number of people have suggested that P values should stabilize after a number of samples (in a permutation test) that depends on the data set. I suspect that these were unintended misstatements. As Dennis Slice has mentioned, one can regard each permutation in the permutation test as a random sample from a distribution. Comparing a test statistic X to its value in the data (say, Y), each permutation draws from a distribution in which there is a probability P that X exceeds Y. So each permutation is (to good approximation) a coin toss with probability P of Heads. There obviously no number of tosses beyond which the fraction of Heads "stabilizes". The fraction of heads after N tosses will depart from the true value P by an amount which has expectation 0, and variance P(1-P)/N. This is a fairly slow approach of the fraction of Heads to the true value. So to get twice as close to the true P value, one needs 4 times as many permutations. And this need for more and more samples continues indefinitely. There is no sudden change as one reaches a threshold number of permutations. But that's what you really meant, right? Joe ------- Joe Felsenstein j...@gs.washington.edu [[alternative HTML version deleted]] _______________________________________________ R-sig-phylo mailing list - r-sig-ph...@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-phylo Searchable archive at http://www.mail-archive.com/r-sig-phylo@r-project.org/ -- MORPHMET may be accessed via its webpage at http://www.morphometrics.org To unsubscribe from this group and stop receiving emails from it, send an email to morphmet+unsubscr...@morphometrics.org.
