Dear all,
In light of the very interesting and highly appreciated response I
received in my mailbox, allow me to attempt to be more clear. First I
should say that I am not aware of the deep details of the study (it is
indeed someone else's and I am not trying to cover up my errors).
Ss are put into a half an hour or so deep relaxation state in a
soundproof environment and are asked to speak loud their thoughts.
After that they are presented with four photographs and asked to rate
them for their similarity to what they were thinking during the
relaxation stage. The photos are pre-selected by the computer and the
experimenter is blind to their content. The computer has also
pre-selected and registered the "winning" photo (again the researcher
knows only that this photo is one of the presented four, but not which
one). After these two sessions the subject fills in a few
questionnaires and the testing ends. What she is trying to see is
whether the Ss have an above chance "hit" on the winning photo (though
this is probably very simplistically stated). Apparently there is a
foundation behind this rationale, but I am not familiar with it, and
frankly I do not think it is relevant to the issue raised. Also there
may be more conditions in the study that I am not aware of, but once
again I do not think they matter here. Finally, I know that there was
a pilot prior to the study.
The intended sample of 200 was stated in an approved grand proposal
and in the proposal to the ethics committee. And as I mention in my
previous posting, potential Ss are very precious since they have to
fulfil certain requirements.
The potential flaw has to do with a condition during the relaxation
session. Drumbeats are used as a background noise because some
evidence suggests that this "harmonises" with ones heartbeat. However,
Ss tend to complain that it distracts their attention or that instead
of relaxing them, it produces the opposite effect (I trust the volume
levels have been checked). Should this be the case, then this may be
the (a) reason why she is not getting the expected effect. Therefore
she is considering changing this to simple and "well-tested" white
noise (don't ask me why, but there has to be some background sound).
Initially I had suggested to split-half her "intended" sample and
randomly assign half of the Ss to the drumbeat condition and the other
half to the white noise one. However, when I was told that 40 people
had already been sampled and that the results were really (and perhaps
surprisingly) discouraging, the issue shifted to the one I presented
here in my initial post.
I would like to make two additional comments/clarifications:
In response to a mail from Dennis who wrote: "if she has run 40 Ss
with no results ... that certainly will NOT get published ", I ensure
you that from what I know of that area (precognition, PK, etc.), "no
results" are as good (and indeed much more common) as "some results" -
though I will agree that one should perhaps be more concerned with
getting "positive" and replicable results. However, in this particular
instance (only) I find it hard to subscribe to the view that perhaps
it is still too early for an effect to show up and that perhaps she
should indeed meet the 200 target and check again. The thought of this
being an unnecessary waste of valuable "resources" really scares me.
Finally replying to Donald's mail who said:
>we keep being reminded that the null hypothesis is never actually
true, which
>implies that the ES is not exactly zero, which implies that with a
>sufficient sample size (maybe ten million or so?) the power curve
would
>indeed level out -- near "power = 1.0".) If one wanted to invoke a
>statistical argument (in the face of whatever logical argument and/or
>evidence exists of a design flaw and/or of an ES an order of
magnitude
>smaller than one had reason to expect in the beginning), it might be
more
>persuasive to show that an upper bound on ES (say, the top of a 95%
>confidence interval) would imply no practical value whatever for so
small
>an ES. (Presumably the presence of an interestingly large ES would
have
>implied some change, or recommendation for change, in practice
>somewhere.)
I do agree with all of this, and in fact, as I say to my students, if
I could have a million participants I would probably be able to detect
a significant association between, say, schizophrenia and banana
consumption. However, I was hoping in this particular instance that
the power curve would show signs of levelling already with a sample
size of, say, 100 Ss (perhaps too optimistic, I know).
Once again I do appreciate your input, and I hope you forgive my
unintended verbosity.
Best
Niko Tiliopoulos
Department of Psychology
The University of Edinburgh
7 George Square
Edinburgh, EH8 9JZ
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
