Dear Tsung, The reason the p-value may have decreased as you increased the number of iterations with your own data set may simply be because you are increasing the denominator when you calculate the p-value. For example, if you run 5 iterations, the lowest p-value you obtain is 1/5 = 0.2. If you run 100, then 0.01, and so on. Therefore, if you have a strong signal against the null hypothesis, then the p-value will decrease as you increase the number of iterations (i.e., 1/"no. of iter" ).
Cheers, Aki On Tue, Jun 2, 2015 at 8:33 AM, Adams, Dean [EEOBS] <[email protected]> wrote: > Tsung, > > > > This property of permutation tests has been known for some time. Generally > speaking, as the number of permutations increases the variation in > significance levels obtained from repeated runs decreases. Earlier work in > the statistics literature from the 70s and 80s by Edgington and others > discusses the issue. > > > > You will also find a figure demonstrating the property for behavioral data > in Adams and Anthony 1996 (Fig. 3). > > > > Best, > > > > Dean > > > > Dr. Dean C. Adams > > Professor > > Department of Ecology, Evolution, and Organismal Biology > > Department of Statistics > > Iowa State University > > www.public.iastate.edu/~dcadams/ > > phone: 515-294-3834 > > > > *From:* Tsung Fei Khang [mailto:[email protected]] > *Sent:* Sunday, May 31, 2015 12:38 PM > *To:* [email protected] > *Subject:* [MORPHMET] Stability of p-values (physignal and testing for > morphological integration) > > > > Dear community, > > > > I would like to share my experience with using some (really cool) > computational tools for phylogenetic signal and morphological integration > analysis. > > > > I am using physignal (geomorph R package) and the Phylo.Morphol.PLS > function provided in the paper by Adams and Felice (2014; PLoS ONE, > 9:e94335) in my work. I noticed that if the same analysis is rerun for a > particular number of iterations, the results may vary. Additionally, I > observed that increasing the number of iterations, up to some critical > point, may push down the p-value, depending on data set (didn't happen with > the plethspecies (9 species) data, but happened in my data set - 13 > species, not salamanders). I attach runs (10 times) for both data sets for > iterations of 100, 1000, 10000 and 100000 here for Phylo.Morphol.PLS. > Note that some kind of stable results is attained after 1000 iterations > (default) for the plethspecies data, but for my case, which needs 10000. > > > > I think the notion that p-values returned from a permutation method are > actually realizations of random variables with a certain mean and variance > may not be familiar to many biologists, who are accustomed to expect a > reproducible p-value when the same data set is rerun using common > statistical tests. Perhaps in a future version the authors of the code can > implement a checker within the functions that checks the number of > iterations for attaining "convergence", so that a more stable p-value is > returned? > > > > > > > > > > > > > > " PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan bersamanya > ("Mesej") adalah ditujukan hanya untuk kegunaan penerima(-penerima) yang > termaklum di atas dan mungkin mengandungi maklumat sulit. Anda dengan ini > dimaklumkan bahawa mengambil apa jua tindakan bersandarkan kepada, membuat > penilaian, mengulang hantar, menghebah, mengedar, mencetak, atau menyalin > Mesej ini atau sebahagian daripadanya oleh sesiapa selain daripada > penerima(-penerima) yang termaklum di atas adalah dilarang. Jika anda telah > menerima Mesej ini kerana kesilapan, anda mesti menghapuskan Mesej ini > dengan segera dan memaklumkan kepada penghantar Mesej ini menerusi balasan > e-mel. Pendapat-pendapat, rumusan-rumusan, dan sebarang maklumat lain di > dalam Mesej ini yang tidak berkait dengan urusan rasmi Universiti Malaya > adalah difahami sebagai bukan dikeluar atau diperakui oleh mana-mana pihak > yang disebut. > > > > DISCLAIMER: This e-mail and any files transmitted with it ("Message") is > intended only for the use of the recipient(s) named above and may contain > confidential information. You are hereby notified that the taking of any > action in reliance upon, or any review, retransmission, dissemination, > distribution, printing or copying of this Message or any part thereof by > anyone other than the intended recipient(s) is strictly prohibited. If you > have received this Message in error, you should delete this Message > immediately and advise the sender by return e-mail. Opinions, conclusions > and other information in this Message that do not relate to the official > business of University of Malaya shall be understood as neither given nor > endorsed by any of the forementioned. " > > -- > MORPHMET may be accessed via its webpage at http://www.morphometrics.org > > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > -- > MORPHMET may be accessed via its webpage at http://www.morphometrics.org > > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > -- Aki Watanabe Doctoral Candidate Division of Paleontology Richard Gilder Graduate School American Museum of Natural History Central Park West at 79th Street New York, NY 10024 website: https://sites.google.com/site/akinopteryx/ twitter: @akiopteryx Google Glass project: https://www.youtube.com/playlist?list=PLrfcruGtplwHCoLvvxKq1Bq-XkbO3tclr -- MORPHMET may be accessed via its webpage at http://www.morphometrics.org To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
