Andrea,
I agree that one must consider both statistical significance and
biological meaningfulness in evaluating patterns. Considering one of
these without the other can often get one into trouble.
Your post concerned the inability to statistically detect differences
due to sample size limitations, and the possibility of concluding
homogeneity from this result when it may not be the case. But as Mike
mentioned, the opposite is also a concern. In fact, one might recall a
discussion some months ago on Morphmet on this very issue; where large
samples afforded the ability to discern allometric differences between
groups, but where those statistical differences may not be
biologically important. In both cases, critical thinking and a merger
of statistical result and biological knowledge of the system are
required to arrive at a well-reasoned understanding of the patterns in
the data.
Best,
Dean
Dr. Dean C. Adams
Professor
Department of Ecology, Evolution, and Organismal Biology
Department of Statistics
Iowa State University
www.public.iastate.edu/~dcadams/
<http://www.public.iastate.edu/%7Edcadams/>
phone: 515-294-3834
*From:*Mike Collyer [mailto:[email protected]]
*Sent:* Monday, December 12, 2016 8:34 AM
*To:* andrea cardini <[email protected]>
*Cc:* [email protected]
*Subject:* Re: brief comment on non-significance Re: [MORPHMET]
procD.allometry with group inclusion
Andrea,
My opinion on this is that the researcher who has collected the data
must retain at all times a biological wisdom that supersedes a
suggested course of action based on results from a statistical test.
If the purpose of a study is to assess the allometric pattern of
shape variation within populations, then maybe the results of a
homogeneity of slopes test can be an unnecessary burden. If a
researcher wants to compare the mean shapes of different groups but is
concerned that allometric variation might differ among groups, then a
homogeneity of slopes test could be an important first step, but I
agree that a non-significant result should not spur the researcher to
immediately conclude a common allometry or no allometry is
appropriate. Sample size, variation in size among groups, and
appropriate distributions of specimen size within groups might all be
things to think about.
The point you make about a potential type II error is a real concern.
The opposite problem is also a real concern. One might have very
large sample sizes and sufficient statistical power to suggest that
allometric slopes are heterogeneous. However, the coefficient of
determination and/or effect size for size:group interaction might be
quite small. Just because there is a low probability of finding as
large of an effect based on thousands of random permutations, is one
ready to accept that different groups have evolved unique allometric
trajectories? It is easy to forget that the choice of “significance
level” - the a priori acceptable rate of type I error - is arbitrary.
Making strong inferential decisions based on a binary decision for an
arbitrary criterion is probably not wise. I would argue that instead
of focusing on a P-value, one could just as arbitrarily, but perhaps
more justifiably, choose a coefficient of determination of R^2 = 0.10
or an effect size of 2 SD as a criterion for whether to retain or omit
the interaction coefficients that allow for heterogenous slopes.
*** Warning: pedantic discussion on model selection starts here. Skip
if unappealing.
One could also turn to model selection approaches. However, I think
multivariate generalization for indices like AIC is an area lacking
needed theoretical research for high-dimensional shape data. There
are two reasons for this. First, the oft-defined AIC is model
log-likelihood + 2K, where K is the number of coefficients in a linear
model (rank of the model design matrix) + 1, where the 1 is the
dimension of the value for the variance of the error. This is a
simplification for univariate data. The second half of the equation
is actually 2[pk + 0.5p(p+1)], where p is the number of shape
variables and k is the rank of the design matrix. (One might define p
as the rank of the shape variable matrix - the number of actual
dimensions in the tangent space, also equal to the number of principal
components with positive eigen values greater than 0 from a PCA - if
using high-dimensional data or small samples.) Notice that
substituting 1 for p in this equation gets one back to the 2K, as
defined first. The pk part of the equation represents the dimensions
of linear model coefficients; the 0.5p(p+1) part represents the
dimensions of the error covariance matrix. The reason this is
important is that one might have picked up along the way that a delta
AIC of 1-2 means two models are comparable (as if with equal
likelihoods, they differ by around 1 parameter or less). This rule of
thumb would have to be augmented with highly multivariate data to 1*p
to 2*p, which makes it hard to have a good general sense of when
models are comparable, unless one takes into consideration how many
shape variables are in use.
Second, the log-likelihood involves calculating the determinant of the
error covariance matrix, which is problematic for singular matrices,
like might be found with high-dimensional shape data. Recently,
colleagues and I have used plots of the log of the trace of error
covariance matrices versus the log of parameter penalties - the 2[pk +
0.5p(p+1)] part - as a way of scanning candidate models for the one or
two that have lower error relative to the number of parameters in the
model. Such an approach allows one to have no allometric slope, a
common allometric slope, and unique allometric slopes, in combination
with other important factors, and consider many models at once. But
again, there is a certain level of arbitrariness to this.
*** End pedantic discussion
There are other issues that can be quite real with real data. For
example, if one wishes to consider if there are shape differences
among groups but first wishes to address if there is meaningful
allometric shape variation, and whether there might be different
allometries among groups, a homogeneity of slopes test might be done.
But what if it is revealed that one group has all small specimens and
one group has all large specimens? The researcher knows better than
anyone else whether this is sampling error or a biological phenomenon.
How to proceed should not rest solely on an outcome from a
statistical test. For example, if the specimens are adult organisms
and represent large individuals within populations, one might want to
discuss shape differences without adjusting for allometry, as well as
discuss size differences. A discussion of allometries in this case
might obscure what is really most important, that maybe two
populations evolved size and shape differences because of some
ecologically meaningful reason, for example.
So I agree with you, and more. “No significance” or “significance” is
only part of the evaluation. Effect sizes and assessment of sampling
errors, biases, or limitations should also be considered. And no
matter what, careful communication that reveals the researcher’s logic
needs to be made in published articles.
Just my opinion,
Mike
On Dec 12, 2016, at 2:40 AM, andrea cardini <[email protected]
<mailto:[email protected]>> wrote:
Dear All,
if I can, I'd add a brief comment on the interpretation of
non-significant results. I'd appreciate this to be checked by
those with a proper understanding and background on stats (which I
haven't!).
I use Mike's sentence on non-significant slopes as an example but
the issue is a general one, although I find it particularly tricky
in the context of comparing trajectories (allometries or other)
across groups. Mike wisely said "approximately ("If not
significant, than the slope vectors are APPROXIMATELY parallel").
With permutations, one might be able to perform tests even when
sample sizes are small (and maybe, which is even more problematic,
heterogeneous across groups): then, non-significance could simply
mean that samples are not large enough to make strong statements
(rejection of the null hp) with confidence (i.e., statistical
power is low). Especially with short trajectories (allometries or
other), it might happen to find n.s. slopes with very large angles
between the vectors, a case where it is probably hard to conclude
that allometries really are parallel.
That of small samples is a curse of many studies in taxonomy and
evolution. We've done a couple of exploratory (non-very-rigorous!)
empirical analyses of the effect of reducing sample sizes on
means, variances, vector angles etc. in geometric morphometrics
(Cardini & Elton, 2007, Zoomorphol.; Cardini et al., 2015,
Zoomorphol.) and some, probably, most of these, literally blow up
when N goes down. That happened even when differences were
relatively large (species separated by several millions of years
of independent evolution or samples including domestic breeds
hugely different from their wild cpunterpart).
Unless one has done power analyses and/or has very large samples,
I'd be careful with the interpretations. There's plenty on this in
the difficult (for me) statistical literature. Surely one can do
sophisticated power analyses in R and, although probably and
unfortunately not used by many, one of the programs of the TPS
series (TPSPower) was written by Jim exactly for this aim
(possibly not for power analyses in the case of MANCOVAs/vector
angles but certainly in the simpler case of comparisons of means).
Cheers
Andrea
On 11/12/16 19:17, Mike Collyer wrote:
Dear Tsung,
The geomorph function, advanced.procD.lm, allows one to
extract group slopes and model coefficients. In fact,
procD.allometry is a specialized function that uses
advanced.procD.lm to perform the HOS test and then uses
procD.lm to produce an ANOVA table, depending on the results
of the HOS test. It also uses the coefficients and fitted
values from procD.lm to generate the various types of
regression scores. In essence, procD.allometry is a function
that carries out several analyses with geomorph base
functions, procD.lm and advanced.procD.lm, in a specified way.
By comparison, the output is more limited, but one can use
the base functions to get much more output.
In advanced.procD.lm, if one specifies groups and a slope, one
of the outputs is a matrix of slope vectors. Also, one can
perform pairwise tests to compare either the correlation or
angle between slope vectors.
Regarding the operation of the HOS test, it is a permutational
test that does the following: calculate the sum of squared
residuals for a “full” model, shape ~ size + group +
size:group and the same for a “reduced” model, shape ~ size +
group. (The sum of squared residuals is the trace of the
error SSCP matrix, which is the same of the sum of the summed
squared residuals for every shape variable.) The difference
between these two values is the sum of squares for the
size:group effect. If significantly large (i.e., is found
with low probability in many random permutations), one can
conclude that the coefficients for this effect are
collectively large enough to justify this effect should be
retained, as the slope vectors are (at least in part) not
parallel. If not significant, than the slope vectors are
approximately parallel, and the effect can be removed from the
model. A randomized residual permutation procedure is used,
which randomizes the residual vectors of the reduced model in
each random permutation to obtain random pseudo-values,
repeating the sum of squares calculations each time.
Regarding your final question, yes, you are correct. In a
case like this, one might conclude that logCS is not a
significant source of shape variation, and proceed with other
analyses that do not include it as a covariate. In either
case - whether is is retained as a covariate or excluded -
advanced.procD.lm will allow one to perform pairwise
comparison tests among groups.
Cheers!
Mike
On Dec 11, 2016, at 10:56 AM, Tsung Fei Khang
<[email protected] <mailto:[email protected]>> wrote:
Dear Mike,
Many thanks for the reply!
When the procD.allometry function performs HOS test with
multiple group labels given, does it compute the
regression vectors for each group, and then tests whether
the coefficients of these vectors were equal, using some
multivariate statistical test? If so, is there an option
that outputs the regression vectors? Given the high
frequency of the latter being discussed in the primary GM
literature, it seems important to be able to extract this
result from the function.
Finally, on the interpretation side - If group variation
is significant, but not logCS, then under the model
shape~size+group, does this imply that shape variation is
mainly explained by variation in species, and allometry is
absent?
Regards,
T.F.
On Thursday, December 8, 2016 at 6:08:17 PM UTC+8, Mike
Collyer wrote:
Dear Tsung,
The procD.allometry function performs two basic
processes when groups are provided. First, it does a
homogeneity of slopes (HOS) test. This test
ascertains whether two or more groups have parallel or
unique slopes (the latter meaning at least one
groups’s slope is different than the others). The HOS
test constructs two linear models: shape ~ size +
group and shape ~ size + group + size:group, and
performs an analysis of variance to determine if the
size:group interaction significantly reduces the
residual error produced. (Note: log(size) is a
possible and default choice in this analysis.)
After this test, procD.allometry then provides an
analysis of variance on each term in the resulting
model from the HOS test.
Regarding your question, if the HOS test reveals there
is significant heterogeneity in slopes, the
coefficients returned allow one to find the unique
linear equations, by group, which would be found from
separate runs on procD.allometry, one group at a time.
If the HOS test reveals that there is not significant
heterogeneity in slopes, the coefficients constrain
the slopes for different groups to be the same
(parallel).
Finally, and I think more to your point, the projected
regression scores are found by using for a (in the Xa
calculation you note) the coefficients that represent
a common or individual slope from the linear model
produced. The matrix of coefficients, B, is arranged
as first row = intercept, second row = common slope,
next rows (if applicable) are coefficients for the
group factor (essentially change the intercept, by
group), and finally, the last rows are the
coefficients for the size:group interaction (if
applicable), which change the common slope to match
each group’s unique slope. Irrespective of the
complexity of this B matrix, a is found as the second
row. If you run procD.allometry group by group, it is
the same as (1) asserting that group slopes are unique
and (2) changing a to match not the common slope, but
the summation of the common slope and the
group-specific slope adjustment. One could do that,
but would lose the ability to compare the groups in
the same plot, as each group would be projected on a
different axis.
Hope that helps.
Mike
On Dec 8, 2016, at 3:37 AM, Tsung Fei Khang
<[email protected] <mailto:[email protected]>> wrote:
Hi all,
I would like to use procD.allometry to study
allometry in two species.
I understand that the function returns the
regression score for each specimen as Reg.proj,
and that the calculation is obtained as:
s = Xa, where X is the nxp matrix of Procrustes
shape variables, and a is the px1 vector of
regression coefficients normalized to 1. I am able
to verify this computation from first principles
when all samples are presumed to come from the
same species.
However, what happens when we are interested in
more than 1 species (say 2)? I could run
procD.allometry by including the species labels
via f2=~gps, where gps gives the species labels.
Is there just 1 regression vector (which feels
weird, since this should be species-specific), or
2? If so, how can I recover both vectors? What is
the difference of including f2=~gps using all
data, compared to if we make two separate runs of
procD.allometry, one for samples from species 1,
and another for samples from species 2?
Thanks for any help.
Rgds,
TF
" PENAFIAN: E-mel ini dan apa-apa fail yang
dikepilkan bersamanya ("Mesej") adalah ditujukan
hanya untuk kegunaan penerima(-penerima) yang
termaklum di atas dan mungkin mengandungi maklumat
sulit. Anda dengan ini dimaklumkan bahawa
mengambil apa jua tindakan bersandarkan kepada,
membuat penilaian, mengulang hantar, menghebah,
mengedar, mencetak, atau menyalin Mesej ini atau
sebahagian daripadanya oleh sesiapa selain
daripada penerima(-penerima) yang termaklum di
atas adalah dilarang. Jika anda telah menerima
Mesej ini kerana kesilapan, anda mesti
menghapuskan Mesej ini dengan segera dan
memaklumkan kepada penghantar Mesej ini menerusi
balasan e-mel. Pendapat-pendapat, rumusan-rumusan,
dan sebarang maklumat lain di dalam Mesej ini yang
tidak berkait dengan urusan rasmi Universiti
Malaya adalah difahami sebagai bukan dikeluar atau
diperakui oleh mana-mana pihak yang disebut.
DISCLAIMER: This e-mail and any files transmitted
with it ("Message") is intended only for the use
of the recipient(s) named above and may contain
confidential information. You are hereby notified
that the taking of any action in reliance upon, or
any review, retransmission, dissemination,
distribution, printing or copying of this Message
or any part thereof by anyone other than the
intended recipient(s) is strictly prohibited. If
you have received this Message in error, you
should delete this Message immediately and advise
the sender by return e-mail. Opinions, conclusions
and other information in this Message that do not
relate to the official business of University of
Malaya shall be understood as neither given nor
endorsed by any of the forementioned. "
--
MORPHMET may be accessed via its webpage
athttp://www.morphometrics.org
<http://www.morphometrics.org/>
---
You received this message because you are
subscribed to the Google Groups "MORPHMET" group.
To unsubscribe from this group and stop receiving
emails from it, send an email
[email protected]
<http://morphometrics.org/>.
" PENAFIAN: E-mel ini dan apa-apa fail yang dikepilkan
bersamanya ("Mesej") adalah ditujukan hanya untuk kegunaan
penerima(-penerima) yang termaklum di atas dan mungkin
mengandungi maklumat sulit. Anda dengan ini dimaklumkan
bahawa mengambil apa jua tindakan bersandarkan kepada,
membuat penilaian, mengulang hantar, menghebah, mengedar,
mencetak, atau menyalin Mesej ini atau sebahagian
daripadanya oleh sesiapa selain daripada
penerima(-penerima) yang termaklum di atas adalah
dilarang. Jika anda telah menerima Mesej ini kerana
kesilapan, anda mesti menghapuskan Mesej ini dengan segera
dan memaklumkan kepada penghantar Mesej ini menerusi
balasan e-mel. Pendapat-pendapat, rumusan-rumusan, dan
sebarang maklumat lain di dalam Mesej ini yang tidak
berkait dengan urusan rasmi Universiti Malaya adalah
difahami sebagai bukan dikeluar atau diperakui oleh
mana-mana pihak yang disebut.
DISCLAIMER: This e-mail and any files transmitted with it
("Message") is intended only for the use of the
recipient(s) named above and may contain confidential
information. You are hereby notified that the taking of
any action in reliance upon, or any review,
retransmission, dissemination, distribution, printing or
copying of this Message or any part thereof by anyone
other than the intended recipient(s) is strictly
prohibited. If you have received this Message in error,
you should delete this Message immediately and advise the
sender by return e-mail. Opinions, conclusions and other
information in this Message that do not relate to the
official business of University of Malaya shall be
understood as neither given nor endorsed by any of the
forementioned. "
--
MORPHMET may be accessed via its webpage
athttp://www.morphometrics.org <http://www.morphometrics.org/>
---
You received this message because you are subscribed to
the Google Groups "MORPHMET" group.
To unsubscribe from this group and stop receiving emails
from it, send an email
[email protected]
<mailto:[email protected]>.
--
MORPHMET may be accessed via its webpage at
http://www.morphometrics.org <http://www.morphometrics.org/>
---
You received this message because you are subscribed to the
Google Groups "MORPHMET" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected]
<mailto:[email protected]>.
--
Dr. Andrea Cardini
Researcher, Dipartimento di Scienze Chimiche e Geologiche, Università di
Modena e Reggio Emilia, Via Campi, 103 - 41125 Modena - Italy
tel. 0039 059 2058472
Adjunct Associate Professor, School of Anatomy, Physiology and Human
Biology, The University of Western Australia, 35 Stirling Highway, Crawley WA
6009, Australia
E-mail address:[email protected]
<mailto:[email protected]>,[email protected]
<mailto:[email protected]>
WEBPAGE:https://sites.google.com/site/alcardini/home/main
FREE Yellow BOOK on Geometric
Morphometrics:http://www.italian-journal-of-mammalogy.it/public/journals/3/issue_241_complete_100.pdf
ESTIMATE YOUR GLOBAL
FOOTPRINT:http://www.footprintnetwork.org/en/index.php/GFN/page/calculators/
--
MORPHMET may be accessed via its webpage at
http://www.morphometrics.org <http://www.morphometrics.org/>
---
You received this message because you are subscribed to the Google
Groups "MORPHMET" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected]
<mailto:[email protected]>.
--
MORPHMET may be accessed via its webpage at
http://www.morphometrics.org <http://www.morphometrics.org>
---
You received this message because you are subscribed to the Google
Groups "MORPHMET" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
