Dear community,
Many thanks to everyone who responded with your opinions and also
references. I think the set.seed solves the reproducibility problem, and
for practicality, I would just set seed, make a single run at a high
number of replicates such as 10,000, and then report a reasonable upper
bound for the p-value (e.g. p-value < 0.01 if I get something like 0.0068).
@Aki: Thank you. 1/#iterations is problematic because one could then get
arbitrarily small p-values... should converge to some value (however small)
as the number of iterations exceeds some threshold, which is dependent on
data set.
On Tuesday, June 2, 2015 at 3:48:37 PM UTC+8, Tsung Fei Khang wrote:
>
> Dear community,
>
> I would like to share my experience with using some (really cool)
> computational tools for phylogenetic signal and morphological integration
> analysis.
>
> I am using physignal (geomorph R package) and the Phylo.Morphol.PLS
> function provided in the paper by Adams and Felice (2014; PLoS ONE,
> 9:e94335) in my work. I noticed that if the same analysis is rerun for a
> particular number of iterations, the results may vary. Additionally, I
> observed that increasing the number of iterations, up to some critical
> point, may push down the p-value, depending on data set (didn't happen with
> the plethspecies (9 species) data, but happened in my data set - 13
> species, not salamanders). I attach runs (10 times) for both data sets for
> iterations of 100, 1000, 10000 and 100000 here for Phylo.Morphol.PLS.
> Note that some kind of stable results is attained after 1000 iterations
> (default) for the plethspecies data, but for my case, which needs 10000.
>
> I think the notion that p-values returned from a permutation method are
> actually realizations of random variables with a certain mean and variance
> may not be familiar to many biologists, who are accustomed to expect a
> reproducible p-value when the same data set is rerun using common
> statistical tests. Perhaps in a future version the authors of the code can
> implement a checker within the functions that checks the number of
> iterations for attaining "convergence", so that a more stable p-value is
> returned?
>
>
>
>
>
>
> " PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan bersamanya
> ("Mesej") adalah ditujukan hanya untuk kegunaan penerima(-penerima) yang
> termaklum di atas dan mungkin mengandungi maklumat sulit. Anda dengan ini
> dimaklumkan bahawa mengambil apa jua tindakan bersandarkan kepada, membuat
> penilaian, mengulang hantar, menghebah, mengedar, mencetak, atau menyalin
> Mesej ini atau sebahagian daripadanya oleh sesiapa selain daripada
> penerima(-penerima) yang termaklum di atas adalah dilarang. Jika anda telah
> menerima Mesej ini kerana kesilapan, anda mesti menghapuskan Mesej ini
> dengan segera dan memaklumkan kepada penghantar Mesej ini menerusi balasan
> e-mel. Pendapat-pendapat, rumusan-rumusan, dan sebarang maklumat lain di
> dalam Mesej ini yang tidak berkait dengan urusan rasmi Universiti Malaya
> adalah difahami sebagai bukan dikeluar atau diperakui oleh mana-mana pihak
> yang disebut.
>
>
> DISCLAIMER: This e-mail and any files transmitted with it ("Message") is
> intended only for the use of the recipient(s) named above and may contain
> confidential information. You are hereby notified that the taking of any
> action in reliance upon, or any review, retransmission, dissemination,
> distribution, printing or copying of this Message or any part thereof by
> anyone other than the intended recipient(s) is strictly prohibited. If you
> have received this Message in error, you should delete this Message
> immediately and advise the sender by return e-mail. Opinions, conclusions
> and other information in this Message that do not relate to the official
> business of University of Malaya shall be understood as neither given nor
> endorsed by any of the forementioned. "
>
--
" PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan bersamanya ("Mesej")
adalah ditujukan hanya untuk kegunaan penerima(-penerima) yang termaklum di
atas dan mungkin mengandungi maklumat sulit. Anda dengan ini dimaklumkan
bahawa mengambil apa jua tindakan bersandarkan kepada, membuat penilaian,
mengulang hantar, menghebah, mengedar, mencetak, atau menyalin Mesej ini
atau sebahagian daripadanya oleh sesiapa selain daripada
penerima(-penerima) yang termaklum di atas adalah dilarang. Jika anda telah
menerima Mesej ini kerana kesilapan, anda mesti menghapuskan Mesej ini
dengan segera dan memaklumkan kepada penghantar Mesej ini menerusi balasan
e-mel. Pendapat-pendapat, rumusan-rumusan, dan sebarang maklumat lain di
dalam Mesej ini yang tidak berkait dengan urusan rasmi Universiti Malaya
adalah difahami sebagai bukan dikeluar atau diperakui oleh mana-mana pihak
yang disebut.
DISCLAIMER: This e-mail and any files transmitted with it ("Message") is
intended only for the use of the recipient(s) named above and may contain
confidential information. You are hereby notified that the taking of any
action in reliance upon, or any review, retransmission, dissemination,
distribution, printing or copying of this Message or any part thereof by
anyone other than the intended recipient(s) is strictly prohibited. If you
have received this Message in error, you should delete this Message
immediately and advise the sender by return e-mail. Opinions, conclusions
and other information in this Message that do not relate to the official
business of University of Malaya shall be understood as neither given nor
endorsed by any of the forementioned. "
--
MORPHMET may be accessed via its webpage at http://www.morphometrics.org
To unsubscribe from this group and stop receiving emails from it, send an email
to morphmet+unsubscr...@morphometrics.org.
