Re: [Freesurfer] Multiple Comparison Question for surface-based analyses

[Douglas Greve]

On 10/17/16 2:16 AM, Ajay Kurani wrote:

Hi Doug,
  I had some additional questions regarding multiple comparisons in 
Freesurfer.


1) Do you correct the left and right hemispheres separately or combine 
both together for muliple comparison correction?
We correct each hemi independently and then bonferroni correct both. Eg, 
if you have a cluster in the left hemi that has a pvalue = .01, then we 
would multiply the pvalue by 2 to account for both hemis. When you run 
mri_glmfit-sim with --2space, it will do this for you automatically.


2) Say you are testing multiple contrasts in your model: A > B, A< B 
etc.  Do you correct for multiple contrasts and if not, is there any 
particular reason why not.
I think you should, but most people probably don't. This is an unsigned 
test. In FS, you would specify "abs" as the sign (absolute, vs pos 
(positive) or neg), and FS will do the right correction. In other 
packages, eg, FSL, you must specify an F-test (even then it might not do 
the right thing).


Thanks,
Ajay

On Wed, Aug 3, 2016 at 2:45 AM, Ajay Kurani > wrote:


Hi Doug,
   Thank you very much for your update regarding this issue.

1)Just curious, will LGI be included in this report as this is
another analysis of interest?

2)As for the cortical thickness I originally used 15mm in the
analysis so based on your email I think using 5-10mm may be more
prudent in order to minimize FPR.  From your email, I understand
that mris_surf2surf (command I use to convert individual subject
to fsaverage or template and smooth to 10-15mm) assumes an ACF
estimation of smoothness which DOES NOT take into account the long
tail distribution.  Does this mean that when using mri_mcsim on my
own template, the cluster extents for a given smoothness will be
undersampled due to the fact that the "true" smoothness is more
than what is estimated in the simulation, correct?  For instance,
when I select 15mm in qdec, it would point to the 21mm folder
(fwhm.dat=20.8mm estimate), and I would select a given cluster
extent for p=0.05.  However, in this case, 15mm may translate to a
larger FWHM than the estimated 21mm, correct?

3)You mentioned that I can use mri_glmfit-sim which is permutation
testing based.  I am struggling a bit in understanding how this
differs from the simulation ran with mri_mcsim/qdec?  Does qdec
monte carlo simulation option run mri_glmfit-sim in the background
to estimate the smoothness which looks up the cluster extent
within the mri_mcsim based on the estimated FWHM?  If so, is this
estimate incorrect due to the fact that the long tails are not
taken into account?


Thanks,
Ajay

On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani
> wrote:

Hello Freesurfer Experts,
   Recently there were two article published regarding
clusterwise simulations for volumetric fmri analyses and
potential errors for underestimating clusterwise extent
thresholds.

1)
http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes

2) biorxiv.org/content/early/2016/07/26/065862


One issue pointed out from these articles seems software
specific, however the second issue is determining the proper
clustersize. The heavy-tail nature of spatial smoothness seems
to be ignored and a gaussian shape is generally assumed,
leading to an underestimation of the spatial smoothness which
can affect cluster size calculations. The issues are
highlighted in the second article above.

I created my own monte carlo simulation in Freesurfer for a
specific brain template and I wanted to find out if these
concerns also apply to my surface based simulations?  I am not
sure if it does since the monte carlo tool is a GRF simulation
as opposed to an analytic equation, however given that these
articles were highlighted very recently, I wanted to ensure I
am running things appropriately for surface based cortical
thickness/dti analyses.

Thanks,
Ajay





___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance 

Re: [Freesurfer] Multiple Comparison Question for surface-based analyses

[Ajay Kurani]

Hi Doug,
  I had some additional questions regarding multiple comparisons in
Freesurfer.

1) Do you correct the left and right hemispheres separately or combine both
together for muliple comparison correction?

2) Say you are testing multiple contrasts in your model: A > B, A< B etc.
Do you correct for multiple contrasts and if not, is there any particular
reason why not.

Thanks,
Ajay

On Wed, Aug 3, 2016 at 2:45 AM, Ajay Kurani 
wrote:

> Hi Doug,
>Thank you very much for your update regarding this issue.
>
> 1)Just curious, will LGI be included in this report as this is another
> analysis of interest?
>
> 2)As for the cortical thickness I originally used 15mm in the analysis so
> based on your email I think using 5-10mm may be more prudent in order to
> minimize FPR.  From your email, I understand that mris_surf2surf (command I
> use to convert individual subject to fsaverage or template and smooth to
> 10-15mm) assumes an ACF estimation of smoothness which DOES NOT take into
> account the long tail distribution.  Does this mean that when using
> mri_mcsim on my own template, the cluster extents for a given smoothness
> will be undersampled due to the fact that the "true" smoothness is more
> than what is estimated in the simulation, correct?  For instance, when I
> select 15mm in qdec, it would point to the 21mm folder (fwhm.dat=20.8mm
> estimate), and I would select a given cluster extent for p=0.05.  However,
> in this case, 15mm may translate to a larger FWHM than the estimated 21mm,
> correct?
>
> 3)You mentioned that I can use mri_glmfit-sim which is permutation testing
> based.  I am struggling a bit in understanding how this differs from the
> simulation ran with mri_mcsim/qdec?  Does qdec monte carlo simulation
> option run mri_glmfit-sim in the background to estimate the smoothness
> which looks up the cluster extent within the mri_mcsim based on the
> estimated FWHM?  If so, is this estimate incorrect due to the fact that the
> long tails are not taken into account?
>
>
> Thanks,
> Ajay
>
> On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani 
> wrote:
>
>> Hello Freesurfer Experts,
>>Recently there were two article published regarding clusterwise
>> simulations for volumetric fmri analyses and potential errors for
>> underestimating clusterwise extent thresholds.
>>
>> 1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
>> 2) biorxiv.org/content/early/2016/07/26/065862
>>
>> One issue pointed out from these articles seems software specific,
>> however the second issue is determining the proper clustersize.  The
>> heavy-tail nature of spatial smoothness seems to be ignored and a gaussian
>> shape is generally assumed, leading to an underestimation of the spatial
>> smoothness which can affect cluster size calculations.  The issues are
>> highlighted in the second article above.
>>
>> I created my own monte carlo simulation in Freesurfer for a specific
>> brain template and I wanted to find out if these concerns also apply to my
>> surface based simulations?  I am not sure if it does since the monte carlo
>> tool is a GRF simulation as opposed to an analytic equation, however given
>> that these articles were highlighted very recently, I wanted to ensure I am
>> running things appropriately for surface based cortical thickness/dti
>> analyses.
>>
>> Thanks,
>> Ajay
>>
>
>
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Multiple Comparison Question for surface-based analyses

[Douglas Greve]

On 8/3/16 3:45 AM, Ajay Kurani wrote:

Hi Doug,
   Thank you very much for your update regarding this issue.

1)Just curious, will LGI be included in this report as this is another 
analysis of interest?
I was not planning to. The 809 subjects that I used for thickness do not 
have lGI run on them, but I think it is possible to run it on a subset 
without too much trouble. Not sure when I'll get to it.


2)As for the cortical thickness I originally used 15mm in the analysis 
so based on your email I think using 5-10mm may be more prudent in 
order to minimize FPR.  From your email, I understand that 
mris_surf2surf (command I use to convert individual subject to 
fsaverage or template and smooth to 10-15mm) assumes an ACF estimation 
of smoothness which DOES NOT take into account the long tail 
distribution.  Does this mean that when using mri_mcsim on my own 
template, the cluster extents for a given smoothness will be 
undersampled due to the fact that the "true" smoothness is more than 
what is estimated in the simulation, correct?  For instance, when I 
select 15mm in qdec, it would point to the 21mm folder 
(fwhm.dat=20.8mm estimate), and I would select a given cluster extent 
for p=0.05.  However, in this case, 15mm may translate to a larger 
FWHM than the estimated 21mm, correct?
Sort of. It is the Gaussian assumption that is incorrect, so there is no 
one FWHM that is correct (it is not a question of it simply being too 
small).


3)You mentioned that I can use mri_glmfit-sim which is permutation 
testing based.  I am struggling a bit in understanding how this 
differs from the simulation ran with mri_mcsim/qdec?  Does qdec monte 
carlo simulation option run mri_glmfit-sim in the background to 
estimate the smoothness which looks up the cluster extent within the 
mri_mcsim based on the estimated FWHM?  If so, is this estimate 
incorrect due to the fact that the long tails are not taken into account?
Permutation uses your data and permutes (ie, randomly swaps) the class 
label associated with a subject, the data with this new labeling are 
analyzed and clusters computed. Under the null, the label is irrelevant, 
so clusters are interpreted as false positives. After several thousand 
of these swaps, one builds a list of the probability of seeing a cluster 
of a certain size under the null, and this is used to generate the 
p-value.  Permutation will then naturally take into account all the 
non-Gaussian aspects of the data. The monte carlo (MC) simulation is 
similar, but it uses smoothed synthesized gaussian noise instead of the 
real data and so the gaussian assumption is  built into it.



Thanks,
Ajay

On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani > wrote:


Hello Freesurfer Experts,
   Recently there were two article published regarding clusterwise
simulations for volumetric fmri analyses and potential errors for
underestimating clusterwise extent thresholds.

1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
2) biorxiv.org/content/early/2016/07/26/065862


One issue pointed out from these articles seems software specific,
however the second issue is determining the proper clustersize. 
The heavy-tail nature of spatial smoothness seems to be ignored

and a gaussian shape is generally assumed, leading to an
underestimation of the spatial smoothness which can affect cluster
size calculations.  The issues are highlighted in the second
article above.

I created my own monte carlo simulation in Freesurfer for a
specific brain template and I wanted to find out if these concerns
also apply to my surface based simulations?  I am not sure if it
does since the monte carlo tool is a GRF simulation as opposed to
an analytic equation, however given that these articles were
highlighted very recently, I wanted to ensure I am running things
appropriately for surface based cortical thickness/dti analyses.

Thanks,
Ajay




___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Multiple Comparison Question for surface-based analyses

[Ajay Kurani]

Hi Doug,
   Thank you very much for your update regarding this issue.

1)Just curious, will LGI be included in this report as this is another
analysis of interest?

2)As for the cortical thickness I originally used 15mm in the analysis so
based on your email I think using 5-10mm may be more prudent in order to
minimize FPR.  From your email, I understand that mris_surf2surf (command I
use to convert individual subject to fsaverage or template and smooth to
10-15mm) assumes an ACF estimation of smoothness which DOES NOT take into
account the long tail distribution.  Does this mean that when using
mri_mcsim on my own template, the cluster extents for a given smoothness
will be undersampled due to the fact that the "true" smoothness is more
than what is estimated in the simulation, correct?  For instance, when I
select 15mm in qdec, it would point to the 21mm folder (fwhm.dat=20.8mm
estimate), and I would select a given cluster extent for p=0.05.  However,
in this case, 15mm may translate to a larger FWHM than the estimated 21mm,
correct?

3)You mentioned that I can use mri_glmfit-sim which is permutation testing
based.  I am struggling a bit in understanding how this differs from the
simulation ran with mri_mcsim/qdec?  Does qdec monte carlo simulation
option run mri_glmfit-sim in the background to estimate the smoothness
which looks up the cluster extent within the mri_mcsim based on the
estimated FWHM?  If so, is this estimate incorrect due to the fact that the
long tails are not taken into account?


Thanks,
Ajay

On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani 
wrote:

> Hello Freesurfer Experts,
>Recently there were two article published regarding clusterwise
> simulations for volumetric fmri analyses and potential errors for
> underestimating clusterwise extent thresholds.
>
> 1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
> 2) biorxiv.org/content/early/2016/07/26/065862
>
> One issue pointed out from these articles seems software specific, however
> the second issue is determining the proper clustersize.  The heavy-tail
> nature of spatial smoothness seems to be ignored and a gaussian shape is
> generally assumed, leading to an underestimation of the spatial smoothness
> which can affect cluster size calculations.  The issues are highlighted in
> the second article above.
>
> I created my own monte carlo simulation in Freesurfer for a specific brain
> template and I wanted to find out if these concerns also apply to my
> surface based simulations?  I am not sure if it does since the monte carlo
> tool is a GRF simulation as opposed to an analytic equation, however given
> that these articles were highlighted very recently, I wanted to ensure I am
> running things appropriately for surface based cortical thickness/dti
> analyses.
>
> Thanks,
> Ajay
>
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Multiple Comparison Question for surface-based analyses

[Douglas N Greve]

I have  been doing simulations similar to #1 (Eklund) using 
surface-based analysis on both thickness and fMRI. I'll prepare a report 
of the results, but the early indications are that the same effect is in 
play, though it does not look like the effects are as bad as in Eklund.

For thickness analysis using applied smoothing of 5 or 10 mm FWHM, for a 
voxel-wise threshold of .001, the false positives are appropriate (ie, 
5%). For a voxel-wise threshold of .01, the false positives is only a 
little off (about 7%); for a voxel-wise threshold of .05, the FPR is 
about 13%. If the data are not smoothed at all, then the false positive 
rates go way up. The reason appears to be the same as found in Eklund 
(ie, the autocorrelation function has a heavier-than-Gaussian tail). I 
did the analysis by randomly selecting 40 subjects from a homogeneous 
data set of 809 subjects aged 18-25. I then made two groups of 20 
subjects each and ran a two-group test, then found clusters significant 
based on our Monte Carlo (Gaussian) simulations. I repeated this several 
thousand times. Any significant clusters were interpreted as false 
positives. These results are much better than Eklund, but Eklund was 
analyzing fMRI data.

I'm still working on the fMRI data. It is much more complicated because 
the results depend on the assumed stimulus schedule (eg, 10 sec blocks 
vs 30 sec blocks) and whether a one-group or two-group anaysis is done; 
nuisance variables also play a role. At very low cluster-forming 
thresholds (ie, .05), the FPR is roughly 20-30%. At a threshold of .01, 
the FPR is about 3-13%. At a threshold of .001 are about 1-6%. This is 
all for an applied smoothing level of 5mm.

All of these results are preliminary, so don't take them as true and 
established yet. As a reminder, you can always do a permutation test 
using mri_glmfit-sim. Eklund found that permutation did pretty well in 
most cases.

doug


On 8/2/16 12:43 AM, Ajay Kurani wrote:
> Hello Freesurfer Experts,
>Recently there were two article published regarding clusterwise 
> simulations for volumetric fmri analyses and potential errors for 
> underestimating clusterwise extent thresholds.
>
> 1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
> 2) biorxiv.org/content/early/2016/07/26/065862 
> 
>
> One issue pointed out from these articles seems software specific, 
> however the second issue is determining the proper clustersize.  The 
> heavy-tail nature of spatial smoothness seems to be ignored and a 
> gaussian shape is generally assumed, leading to an underestimation of 
> the spatial smoothness which can affect cluster size calculations.  
> The issues are highlighted in the second article above.
>
> I created my own monte carlo simulation in Freesurfer for a specific 
> brain template and I wanted to find out if these concerns also apply 
> to my surface based simulations?  I am not sure if it does since the 
> monte carlo tool is a GRF simulation as opposed to an analytic 
> equation, however given that these articles were highlighted very 
> recently, I wanted to ensure I am running things appropriately for 
> surface based cortical thickness/dti analyses.
>
> Thanks,
> Ajay
>
>
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

[Freesurfer] Multiple Comparison Question for surface-based analyses

[Ajay Kurani]

Hello Freesurfer Experts,
   Recently there were two article published regarding clusterwise
simulations for volumetric fmri analyses and potential errors for
underestimating clusterwise extent thresholds.

1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
2) biorxiv.org/content/early/2016/07/26/065862

One issue pointed out from these articles seems software specific, however
the second issue is determining the proper clustersize.  The heavy-tail
nature of spatial smoothness seems to be ignored and a gaussian shape is
generally assumed, leading to an underestimation of the spatial smoothness
which can affect cluster size calculations.  The issues are highlighted in
the second article above.

I created my own monte carlo simulation in Freesurfer for a specific brain
template and I wanted to find out if these concerns also apply to my
surface based simulations?  I am not sure if it does since the monte carlo
tool is a GRF simulation as opposed to an analytic equation, however given
that these articles were highlighted very recently, I wanted to ensure I am
running things appropriately for surface based cortical thickness/dti
analyses.

Thanks,
Ajay
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.