There was a similar concern (about arbitrary p-values in simulations) expressed in a nice, recent Oikos paper, too:
White, J. W., A. Rassweiler, J. F. Samhouri, A. C. Stier, and C. White. 2014. Ecologists should not use statistical significance tests to interpret simulation model results. Oikos 123(4):385-388. I think some of the concerns are also wise to heed for those doing resampling. It's always worth doing a quick sensitivity analysis (running the resampling algorithm for increasing numbers of replicates) to identify the point at which p-values (or other stats you're interested in) become stable. But there's no need to run huge replicates if they're not needed. See p. 40 in the paper below (focused more on general paleontological applications than morphometrics) for discussion and recommendations on this issue, with some advice from the literature. The online supplement at http://www.paleosoc.org/shortcourse2010/Resampling_KandN-G_Appendices_Oct11.doc has some (clunky) sample R code (section 2.8) demonstrating this. Kowalewski, M., and P. Novack-Gottshall. 2010. Resampling methods in paleontology. Pp. 19-54. In J. Alroy, and G. Hunt, eds. Quantitative Methods in Paleobiology. Short Courses in Paleontology 16. Paleontological Society and Paleontological Research Institute, Ithaca, NY. Cheers, Phil On 6/2/2015 11:47 PM, Tsung Fei Khang wrote: Dear community, Many thanks to everyone who responded with your opinions and also references. I think the set.seed solves the reproducibility problem, and for practicality, I would just set seed, make a single run at a high number of replicates such as 10,000, and then report a reasonable upper bound for the p-value (e.g. p-value < 0.01 if I get something like 0.0068). @Aki: Thank you. 1/#iterations is problematic because one could then get arbitrarily small p-values... should converge to some value (however small) as the number of iterations exceeds some threshold, which is dependent on data set. On Tuesday, June 2, 2015 at 3:48:37 PM UTC+8, Tsung Fei Khang wrote: Dear community, I would like to share my experience with using some (really cool) computational tools for phylogenetic signal and morphological integration analysis. I am using physignal (geomorph R package) and the Phylo.Morphol.PLS function provided in the paper by Adams and Felice (2014; PLoS ONE, 9:e94335) in my work. I noticed that if the same analysis is rerun for a particular number of iterations, the results may vary. Additionally, I observed that increasing the number of iterations, up to some critical point, may push down the p-value, depending on data set (didn't happen with the plethspecies (9 species) data, but happened in my data set - 13 species, not salamanders). I attach runs (10 times) for both data sets for iterations of 100, 1000, 10000 and 100000 here for Phylo.Morphol.PLS. Note that some kind of stable results is attained after 1000 iterations (default) for the plethspecies data, but for my case, which needs 10000. I think the notion that p-values returned from a permutation method are actually realizations of random variables with a certain mean and variance may not be familiar to many biologists, who are accustomed to expect a reproducible p-value when the same data set is rerun using common statistical tests. Perhaps in a future version the authors of the code can implement a checker within the functions that checks the number of iterations for attaining "convergence", so that a more stable p-value is returned? " PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan bersamanya ("Mesej") adalah ditujukan hanya untuk kegunaan penerima(-penerima) yang termaklum di atas dan mungkin mengandungi maklumat sulit. Anda dengan ini dimaklumkan bahawa mengambil apa jua tindakan bersandarkan kepada, membuat penilaian, mengulang hantar, menghebah, mengedar, mencetak, atau menyalin Mesej ini atau sebahagian daripadanya oleh sesiapa selain daripada penerima(-penerima) yang termaklum di atas adalah dilarang. Jika anda telah menerima Mesej ini kerana kesilapan, anda mesti menghapuskan Mesej ini dengan segera dan memaklumkan kepada penghantar Mesej ini menerusi balasan e-mel. Pendapat-pendapat, rumusan-rumusan, dan sebarang maklumat lain di dalam Mesej ini yang tidak berkait dengan urusan rasmi Universiti Malaya adalah difahami sebagai bukan dikeluar atau diperakui oleh mana-mana pihak yang disebut. DISCLAIMER: This e-mail and any files transmitted with it ("Message") is intended only for the use of the recipient(s) named above and may contain confidential information. You are hereby notified that the taking of any action in reliance upon, or any review, retransmission, dissemination, distribution, printing or copying of this Message or any part thereof by anyone other than the intended recipient(s) is strictly prohibited. If you have received this Message in error, you should delete this Message immediately and advise the sender by return e-mail. Opinions, conclusions and other information in this Message that do not relate to the official business of University of Malaya shall be understood as neither given nor endorsed by any of the forementioned. " " PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan bersamanya ("Mesej") adalah ditujukan hanya untuk kegunaan penerima(-penerima) yang termaklum di atas dan mungkin mengandungi maklumat sulit. Anda dengan ini dimaklumkan bahawa mengambil apa jua tindakan bersandarkan kepada, membuat penilaian, mengulang hantar, menghebah, mengedar, mencetak, atau menyalin Mesej ini atau sebahagian daripadanya oleh sesiapa selain daripada penerima(-penerima) yang termaklum di atas adalah dilarang. Jika anda telah menerima Mesej ini kerana kesilapan, anda mesti menghapuskan Mesej ini dengan segera dan memaklumkan kepada penghantar Mesej ini menerusi balasan e-mel. Pendapat-pendapat, rumusan-rumusan, dan sebarang maklumat lain di dalam Mesej ini yang tidak berkait dengan urusan rasmi Universiti Malaya adalah difahami sebagai bukan dikeluar atau diperakui oleh mana-mana pihak yang disebut. DISCLAIMER: This e-mail and any files transmitted with it ("Message") is intended only for the use of the recipient(s) named above and may contain confidential information. You are hereby notified that the taking of any action in reliance upon, or any review, retransmission, dissemination, distribution, printing or copying of this Message or any part thereof by anyone other than the intended recipient(s) is strictly prohibited. If you have received this Message in error, you should delete this Message immediately and advise the sender by return e-mail. Opinions, conclusions and other information in this Message that do not relate to the official business of University of Malaya shall be understood as neither given nor endorsed by any of the forementioned. " -- MORPHMET may be accessed via its webpage at http://www.morphometrics.org To unsubscribe from this group and stop receiving emails from it, send an email to morphmet+unsubscr...@morphometrics.org<mailto:morphmet+unsubscr...@morphometrics.org>. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Phil Novack-Gottshall Associate Professor Department of Biological Sciences Benedictine University 5700 College Road Lisle, IL 60532 pnovack-gottsh...@ben.edu<mailto:pnovack-gottsh...@ben.edu> Phone: 630-829-6514 Fax: 630-829-6547 Office: 332 Birck Hall Lab: 316 Birck Hall http://www1.ben.edu/faculty/pnovack-gottshall ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- MORPHMET may be accessed via its webpage at http://www.morphometrics.org To unsubscribe from this group and stop receiving emails from it, send an email to morphmet+unsubscr...@morphometrics.org.
