[EMAIL PROTECTED] wrote:
>
> In article <[EMAIL PROTECTED]>,
> > Chris: That's not what Jerry means. What he's saying is that if
> > your sample size is large enough, a difference may be statistically
> > significant (a term which has a very precise meaning, especially to
> > the Apostles of the Holy 5%) but not large enough to be practically
> > important. [A hypothetical very large sample might show, let us say,
> > that a very expensive diet supplement reduced one's chances of a heart
> > attack by 1/10 of 1%.]
>
> Firstly, I think we can thank publication pressures for the church of
> the Holy 5%. I go with Keppel's approach in suspending judgement for mid
> range significance levels (although we should do this for nonsignificant
> results anyway as they are inherently indeterminant).
>
> Wrt to your example, it seems that the decision you are making about
> practical importance is purely subjective.
What exactly do you mean by this? Are you saying that _my_ example
is purely subjective but that others are not, or that the entire concept
of practical significance is subjective? And, if so, so what? Does it
then follow that it is more "scientific" to ignore it entirely?
In any number of alternative
> situations a .01% effect could have major implications, practical and
> theoretical.
It might or it might not. I was referring to a hypothetical situation
in which it seemed reasonable to suppose that it didn't.
I regard this as less a fundamental flaw with hypothesis
> testing and more a question of expermental design and asking the right
> questions to begin with.
Fair enough: but I would argue that the right question is rarely "if
there were no effect whatsoever, and the following model applied, what
is the probility that we would observe a value of the following
statistic at least as great as what was observed?" and hence that a
hypothesis test is rarely the right way to obtain the right answer.
Hypothesis testing does what it sets out to do perfectly well- the only
question, in most cases, is why one would want that done.
> >Alternatively, in an imperfectly-controlled
> > study, it may show an effect that - whether large enough to be of
> > interest or not - is too small to ascribe a cause to. [A moderately
> > large study might show that some ethnic group has a 1% higher rate of
> > heart attacks, with amargin of error of +- .2% . But we might have, or
> > an effect of this size, no way of telling whether it's due to genes,
> > diet, socioeconomic factors, recreational drugs, or whatever.]
>
> Surely the ambiguity of this outcome is the result of the lack of
> experimental control. If the effects of genetics, diet etc. are not
> appropriately controlled, it doesn't matter what sample size is
> used - the outcome will be always be equivocal. What it does suggest is
> that, irrespective of sample size, we must be vigilant in controlling
> for extraneous variables. Is it fair to consider this a flaw of
> hypothesis testing? We can hardly blame the tools for not working
> properly if they are not used correctly.
Fair enough... I do not argue with your support of proper controls.
However,
in the real world, insisting on this would be tantamount to ending
experimental
research in the social sciences and many disciplines within the life
sciences.
(You may draw your own conclusions as the advisability of this <grin> -
I will
venture an opinion that it ain't a-gonna happen, advisable or no.)
There are always more experimental variables than we can control for,
and there are often explanatory variables of interest that it would be
impossible (eg, ethnic
background - unless we can emulate the aliens on the Monty Python
episode who
could turn people into Scotsmen!) or unethical to randomize. The best
that
one can hope to do in such situations is control for nuisance variables
whose
effects judged likely to produce a large effect, and accept that any
small effect
is of unknowable origin.
-Robert Dawson
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
