A number of people have suggested that P values should stabilize after a
number of samples (in a permutation test) that depends on the data set.
I suspect that these were unintended misstatements. As Dennis Slice has
mentioned, one can regard each permutation in the permutation test as a
random sample from a distribution. Comparing a test statistic X to its
value in the data (say, Y), each permutation draws from a distribution in
which there is a probability P that X exceeds Y.
So each permutation is (to good approximation) a coin toss with probability
P of Heads. There obviously no number of tosses beyond which the fraction
of Heads "stabilizes". The fraction of heads after N tosses will depart
from the true value P by an amount which has expectation 0, and variance
P(1-P)/N. This is a fairly slow approach of the fraction of Heads to the
true value.
So to get twice as close to the true P value, one needs 4 times as many
permutations. And this need for more and more samples continues
indefinitely. There is no sudden change as one reaches a threshold number
of permutations.
But that's what you really meant, right?
Joe
-------
Joe Felsenstein j...@gs.washington.edu
[[alternative HTML version deleted]]
_______________________________________________
R-sig-phylo mailing list - R-sig-phylo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-phylo
Searchable archive at http://www.mail-archive.com/r-sig-phylo@r-project.org/
