Tsung,
This property of permutation tests has been known for some time. Generally
speaking, as the number of permutations increases the variation in significance
levels obtained from repeated runs decreases. Earlier work in the statistics
literature from the 70s and 80s by Edgington and others discusses the issue.
You will also find a figure demonstrating the property for behavioral data in
Adams and Anthony 1996 (Fig. 3).
Best,
Dean
Dr. Dean C. Adams
Professor
Department of Ecology, Evolution, and Organismal Biology
Department of Statistics
Iowa State University
www.public.iastate.edu/~dcadams/<http://www.public.iastate.edu/~dcadams/>
phone: 515-294-3834
From: Tsung Fei Khang [mailto:[email protected]]
Sent: Sunday, May 31, 2015 12:38 PM
To: [email protected]
Subject: [MORPHMET] Stability of p-values (physignal and testing for
morphological integration)
Dear community,
I would like to share my experience with using some (really cool) computational
tools for phylogenetic signal and morphological integration analysis.
I am using physignal (geomorph R package) and the Phylo.Morphol.PLS function
provided in the paper by Adams and Felice (2014; PLoS ONE, 9:e94335) in my
work. I noticed that if the same analysis is rerun for a particular number of
iterations, the results may vary. Additionally, I observed that increasing the
number of iterations, up to some critical point, may push down the p-value,
depending on data set (didn't happen with the plethspecies (9 species) data,
but happened in my data set - 13 species, not salamanders). I attach runs (10
times) for both data sets for iterations of 100, 1000, 10000 and 100000 here
for Phylo.Morphol.PLS. Note that some kind of stable results is attained after
1000 iterations (default) for the plethspecies data, but for my case, which
needs 10000.
I think the notion that p-values returned from a permutation method are
actually realizations of random variables with a certain mean and variance may
not be familiar to many biologists, who are accustomed to expect a reproducible
p-value when the same data set is rerun using common statistical tests. Perhaps
in a future version the authors of the code can implement a checker within the
functions that checks the number of iterations for attaining "convergence", so
that a more stable p-value is returned?
" PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan bersamanya ("Mesej")
adalah ditujukan hanya untuk kegunaan penerima(-penerima) yang termaklum di
atas dan mungkin mengandungi maklumat sulit. Anda dengan ini dimaklumkan bahawa
mengambil apa jua tindakan bersandarkan kepada, membuat penilaian, mengulang
hantar, menghebah, mengedar, mencetak, atau menyalin Mesej ini atau sebahagian
daripadanya oleh sesiapa selain daripada penerima(-penerima) yang termaklum di
atas adalah dilarang. Jika anda telah menerima Mesej ini kerana kesilapan, anda
mesti menghapuskan Mesej ini dengan segera dan memaklumkan kepada penghantar
Mesej ini menerusi balasan e-mel. Pendapat-pendapat, rumusan-rumusan, dan
sebarang maklumat lain di dalam Mesej ini yang tidak berkait dengan urusan
rasmi Universiti Malaya adalah difahami sebagai bukan dikeluar atau diperakui
oleh mana-mana pihak yang disebut.
DISCLAIMER: This e-mail and any files transmitted with it ("Message") is
intended only for the use of the recipient(s) named above and may contain
confidential information. You are hereby notified that the taking of any action
in reliance upon, or any review, retransmission, dissemination, distribution,
printing or copying of this Message or any part thereof by anyone other than
the intended recipient(s) is strictly prohibited. If you have received this
Message in error, you should delete this Message immediately and advise the
sender by return e-mail. Opinions, conclusions and other information in this
Message that do not relate to the official business of University of Malaya
shall be understood as neither given nor endorsed by any of the forementioned. "
--
MORPHMET may be accessed via its webpage at http://www.morphometrics.org
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]<mailto:[email protected]>.
--
MORPHMET may be accessed via its webpage at http://www.morphometrics.org
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
