Dear all, In light of the very interesting and highly appreciated response I received in my mailbox, allow me to attempt to be more clear. First I should say that I am not aware of the deep details of the study (it is indeed someone else's and I am not trying to cover up my errors).
Ss are put into a half an hour or so deep relaxation state in a soundproof environment and are asked to speak loud their thoughts. After that they are presented with four photographs and asked to rate them for their similarity to what they were thinking during the relaxation stage. The photos are pre-selected by the computer and the experimenter is blind to their content. The computer has also pre-selected and registered the "winning" photo (again the researcher knows only that this photo is one of the presented four, but not which one). After these two sessions the subject fills in a few questionnaires and the testing ends. What she is trying to see is whether the Ss have an above chance "hit" on the winning photo (though this is probably very simplistically stated). Apparently there is a foundation behind this rationale, but I am not familiar with it, and frankly I do not think it is relevant to the issue raised. Also there may be more conditions in the study that I am not aware of, but once again I do not think they matter here. Finally, I know that there was a pilot prior to the study. The intended sample of 200 was stated in an approved grand proposal and in the proposal to the ethics committee. And as I mention in my previous posting, potential Ss are very precious since they have to fulfil certain requirements. The potential flaw has to do with a condition during the relaxation session. Drumbeats are used as a background noise because some evidence suggests that this "harmonises" with ones heartbeat. However, Ss tend to complain that it distracts their attention or that instead of relaxing them, it produces the opposite effect (I trust the volume levels have been checked). Should this be the case, then this may be the (a) reason why she is not getting the expected effect. Therefore she is considering changing this to simple and "well-tested" white noise (don't ask me why, but there has to be some background sound). Initially I had suggested to split-half her "intended" sample and randomly assign half of the Ss to the drumbeat condition and the other half to the white noise one. However, when I was told that 40 people had already been sampled and that the results were really (and perhaps surprisingly) discouraging, the issue shifted to the one I presented here in my initial post. I would like to make two additional comments/clarifications: In response to a mail from Dennis who wrote: "if she has run 40 Ss with no results ... that certainly will NOT get published ", I ensure you that from what I know of that area (precognition, PK, etc.), "no results" are as good (and indeed much more common) as "some results" - though I will agree that one should perhaps be more concerned with getting "positive" and replicable results. However, in this particular instance (only) I find it hard to subscribe to the view that perhaps it is still too early for an effect to show up and that perhaps she should indeed meet the 200 target and check again. The thought of this being an unnecessary waste of valuable "resources" really scares me. Finally replying to Donald's mail who said: >we keep being reminded that the null hypothesis is never actually true, which >implies that the ES is not exactly zero, which implies that with a >sufficient sample size (maybe ten million or so?) the power curve would >indeed level out -- near "power = 1.0".) If one wanted to invoke a >statistical argument (in the face of whatever logical argument and/or >evidence exists of a design flaw and/or of an ES an order of magnitude >smaller than one had reason to expect in the beginning), it might be more >persuasive to show that an upper bound on ES (say, the top of a 95% >confidence interval) would imply no practical value whatever for so small >an ES. (Presumably the presence of an interestingly large ES would have >implied some change, or recommendation for change, in practice >somewhere.) I do agree with all of this, and in fact, as I say to my students, if I could have a million participants I would probably be able to detect a significant association between, say, schizophrenia and banana consumption. However, I was hoping in this particular instance that the power curve would show signs of levelling already with a sample size of, say, 100 Ss (perhaps too optimistic, I know). Once again I do appreciate your input, and I hope you forgive my unintended verbosity. Best Niko Tiliopoulos Department of Psychology The University of Edinburgh 7 George Square Edinburgh, EH8 9JZ ================================================================= Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =================================================================