In article <[EMAIL PROTECTED]>,
Uplandcrow wrote:
> I teach research methods for social science at a small liberal arts
> college.
> The level of math in the class is low, I use Richard Black's "Doing
> Quantitative Research in the Soc. Sci." and excerpts from
> Gujarati's "Basic Econometrics."
> <SNIP>
> I am looking for examples of articles that use a stat procedure
> incorrectly.
Here is an error that is subtle, but very common. The statistical
test (multiple regression) was applied perfectly, but the
statistical inference was wrong.
My first reference to this type of error is in the classic,
but highly controversial, ecology treatise by Andrewartha & Birch
(1954): The distribution and abundance of animals, p. 580.
These Australian ecologists wanted to show that animal
populations aren't controlled by density-dependent factors like
competition or predation. They regress 14 years of thrip (an
insect) abundance vs weather variables. They considered weather a
density-independent factor (mortality from a storm or a hot day
isn't directly related to animal density).
They conclude, "...altogether, 78 per cent of the variance
<in thrip maximal abundance> was explained by four quantities which
were calculated entirely from meteorological records. This left
virtually no chance of finding any other systematic cause for
variation, because 22 per cent is a rather small residium to
be left as due to random sampling errors.
All the variation in maximal numbers from year to year may therefore
be attributed to causes that are not related to density: not only
did we not find a "density-dependent factor," but we also showed that
there was no room for one."
The logical/statistical flaw in the Australian thrip story was
published in Smith, F.E. (1961) Density dependence in the Australian
thrips. Ecology 42: 403-407. Since weather accounted for such a
high proportion of the variance in the data (78%), A&B assumed other
factors could not be important. This is a fallacy. Smith argues
that some density-dependent factors, unmeasured but probably correlated
with weather, must be acting to control abundances.
I have recently seen examples of the thrip fallacy in the op-ed
pages of the Boston Globe. Massachusetts has implemented
state-wide standardized testing and has increased state funding
for school districts with low test scores. Statistical analysis
reveals that Five or six socioeconomic factors
(parents educational level, annual salary, % two-parent households,
etc) account for over 90% of the variance in town-to-town K-12
standardized test scores. The implication is that only 10% of
the variance in mean test scores COULD be due to differences in
curriculum, teacher quality, or financing for the school (Take
that Teacher's Unions!). Some might conclude that spending
money on schools & teachers since only 10% of the town-to-town
variance in these scores could be due to factors outside the home.
This fallacy fails to consider that a high median income and
other socioeconomic factors often are strongly associated with
a better tax base, lower class sizes, better trained teachers,
more innovative curriculum etc.
This fallacy should have a name, but I don't know it. I point
my students to Wright's path analysis and structural
modeling approaches (LISREL, and AMOS) to show alternatives
to the misleading inference based on an R^2 in a multiple
regression equation.
One could experimentally demonstrate this fallacy by
transferring students from affluent communities to communities whose
schools have dismal standardized test scores. Somehow, I don't
think the parents would accept the statistical argument that the
their children's mean scores could decline at most 10% since 90% of
the variance was due to socio-economic variables.
--
Eugene D. Gallagher
ECOS, UMASS/Boston
Sent via Deja.com http://www.deja.com/
Before you buy.
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
