Dear Lars, Pedro and all,
There should be no problem in using different thresholds for each of
these pairwise comparisons.
Let's consider the following extreme case: Suppose in one of these
comparisons you have a strong effect (say, "activation") that comprises
large brain regions. Suppose that for the 5 other comparisons, there is
no effect at all. If you pool all your six comparisons to calculate a
single threshold for all of them, due to the adaptiveness of the
Benjamini & Hochberg procedure, the calculated threshold will be such
that it will produce more liberal results (i.e. higher p-threshold) for
the 5 comparisons where there is no experimental effect, than would be
obtained by not pooling, implying that some vertices where the null is
true will be (falsely) declared as positive for these comparisons.
On the other hand, the vertices where there is no effect will also give
their contribution to compute this threshold, but their influence will
be in the opposite direction, producing more conservative results (i.e.
lower p-threshold) for the comparison where "activation" is present,
than would be without pooling, resulting in "active" vertices remaining
not detected (false negatives).
Both are clearly undesirable, as the amount of errors (both type I and
II) is increased by bleeding the effect/absence of effect from one
comparison into another.
In other words, when including in the same analysis different sets of
comparisons (which possibly includes different experimental hypotheses,
which definitely would preclude pooling), one will be losing one of the
nicest features of the B&H procedure: the weak control of FWE (i.e. when
the null is true everywhere, you are controlling FWE, even using an FDR
procedure) for each comparison if the null for any of these comparisons
is true everywhere.
This also means that, although FDR would still be controlled globally,
one cannot make inferences about each comparison individually, which I
believe was the whole point of making the comparisons initially.
Hope this helps!
Kind regards,
Anderson
Lars M. Rimol wrote:
Well, I believe there is a problem in principle here. FDR deals with
multiple comparisons across the surface (or brain volume), but how do
you deal with a series of such analyses? Of course, if you use a
different method of correction you avoid this problem but that's not
the point.
LMR
------------------------------------------------------------------------
Date: Tue, 24 Mar 2009 14:03:46 -0300
Subject: Re: [Freesurfer] FDR correction
From: p...@netfilter.com.br
To: lari...@gmail.com
CC: freesurfer@nmr.mgh.harvard.edu
Some links that may be helpful:
http://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/QdecMultipleComparisons
http://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/GroupAnalysis
http://surfer.nmr.mgh.harvard.edu/fswiki/MultipleComparisons
Hope it helps.
PPJ
-----------------------------------------------------------
Pedro Paulo de M. Oliveira Junior
Diretor de Operações
Netfilter & SpeedComm Telecom
On Tue, Mar 24, 2009 at 12:38, Lars M. Rimol <lari...@gmail.com
<mailto:lari...@gmail.com>> wrote:
Hi,
I have done an analysis involving three groups, so there are three
pairwise comparisons across two hemispheres = 6 p-maps. I want to
adjust for multiple comparisons (across the vertices), so I use
FDR. But since FDR determines the threshold basd on the actual
p-values, I get 6 different tresholds:
comparison 1: lh and rh, 0.016 and 0.028 (I can choose .01)
comparison 2: lh and rh, 0.01 and 0.001 (I can choose.001)
comparison 3 lh and rh, 0.001 and 0.0001 (I can choose .0001)
There are lots of significant vertices in comparison 1 and nothing
significant, after correction, in comparison 3. Is there anything
wrong with using different tresholds here, and concluding that in
comparison 1 there were extensive differences between the groups,
whereas in comparison 3 there were none? I'm not sure if this is a
problem, but I'm afraid some reviewers might have an issue with
it. Across the hemispheres, I can choose a conservative threshold
which covers both hemispheres, i.e. lower than both the
FDR-adjusted treshold for lh and rh. But between the comparisons
the tresholds differ even more, by a factor of 10 and 100. And if
I choose the most conservative of all the adjusted thresholds, I'm
afraid that I'll make a type II error in comparison 1.
From what I understand, the adjusted threshold for comparison 3 is
more conservative because of the actual empirical data (the
distribution of p-values), so that's an empirical argument for
using a more conservative threshold there.
And: What if I pooled all thre p-maps (sig.mgh) and did an FDR on
the whole thing, would that be a better approach? And does
Freesurfer use the Benjamini algorithm, and if you do, can I use
Tom Nichols' matlab function for FDR
(http://www.sph.umich.edu/~nichols/FDR/FDR.m
<http://www.sph.umich.edu/%7Enichols/FDR/FDR.m>) for pooling all
three p-maps?
Thank you!
--
yours,
LMR
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu>
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
------------------------------------------------------------------------
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
