A paper was recently published in the Proceedings of the National
Academy of Sciences (PNAS) that shows that the three software
packages typically used in analyzing fMRI images produces a
higher than expected false positive rate (Type I error rate). Some
of the popular science media outlets that cover this research
include Ars Technica: see::
http://arstechnica.com/science/2016/07/algorithms-used-to-study-brain-activity-may-be-exaggerating-results/
Quoting from the above article:
|The results were not good news for fMRI users. "In brief,"
|the authors conclude, "we find that all three packages have
|conservative voxelwise inference and invalid clusterwise
|inference." In other words, while they're likely to be cautions
|when determining whether a given voxel is showing activity,
|the cluster identification algorithms frequently assign activity
|to a region when none is likely to be present. How frequently?
|Up to 70 percent of the time, depending on the algorithm
|and parameters used.
|
|For good measure, a bug that has been sitting in the code
|for 15 years showed up during this testing. The fix for the bug
|reduced false positives by more than 10 percent. While good
|that it's fixed, it's a shame that all those studies have been
|published using the faulty version.
|
|The authors also found that some regions of the brain were
|more likely to have problems with false positives possibly
|because of assumptions the algorithms make about the
|underlying brain morphology.
|
|Is this really as bad as it sounds? The authors certainly think so.
|"This calls into question the validity of countless published fMRI
|studies based on parametric clusterwise inference." It's not clear
|how many of those there are, but they're likely to be a notable
|fraction of the total number of studies that use fMRI, which the
|authors estimate at 40,000.
To be fair, back in 1975 Wilkinson & Dallal found that most of the
commonly used statistical packages were subject to "overflow"
and "underflow" errors that affected how descriptive statistics,
correlations, and other statistics were calculated -- the problem
was that not enough computer memory was allocated to represent
the numbers used in the calculations, truncating them and subsequently
throwing off the calculations. One wonders how many dissertations
and published research articles presented wrongly calculated statistics.
The reference for W&D is:
Wilkinson, L., & Dallal, G. E. (1977). Accuracy of sample moments
calculations among widely used statistical programs. The American
Statistician, 31(3), 128-131.
NOTE: The package BMDP calculated the values correctly because
of the algorithms it used.
Wilkinson, who developed the software package Systat, continued
to do research on statistical software accuracy, for example:
Wilkinson, L. (1994). Practical guidelines for testing statistical
software.
Computational Statistics, 111-24.
For the complete article on the problems of statistical analysis of
fMRI images, see:
http://www.pnas.org/content/early/2016/06/27/1602413113.full
The reference for this article is:
Eklund, A., Nichols, T. E., & Knutsson, H. (2016). Cluster failure:
Why fMRI inferences for spatial extent have inflated false-positive
rates.
Proceedings of the National Academy of Sciences, 201602413.
-Mike Palij
New York University
m...@nyu.edu
---
You are currently subscribed to tips as: arch...@mail-archive.com.
To unsubscribe click here:
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=48987
or send a blank email to
leave-48987-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
