re: [tips] Reliability of the sacred scientific method?
On Tue, 28 Dec 2010 12:40:54 -0800, Joan Warmbold wrote:
> http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer
>
>Fascinating article in the New Yorker about the conflicting results
>researchers find despite all claiming they are using objective, scientific
>procedures in their research. Where are the likely sources for possible
>contamination of the results of the various studies discussed? I have
>some thoughts but would enjoy hearing others first.
Let me suggest examining the following article
Bernhard T Gehr1, Christel Weiss2 and Franz Porzsolt*3 (2006)
The fading of reported effectiveness. A meta-analysis of randomised
controlled trials
BMC Medical Research Methodology 2006, 6:25 doi:10.1186/1471-2288-6-25
PDF available at the following website
http://www.biomedcentral.com/content/pdf/1471-2288-6-25.pdf
Abstract
Background: The "real" effect size of a medical therapy is constant over
time. In contrast, the effect size reported in randomised controlled trials
(RCTs) may change over time because the sum of all kinds of bias
influencing the reported effectiveness is not necessarily constant. As this
would affect the validity of meta-analyses, we tested the hypothesis that
the reported effect size decreases over time. Furthermore, we tested
three hypotheses that would explain a possible change.
Methods: Because of well established outcome measures, the lipid-lowering
drugs Pravastatin and Atorvastatin (serum low-density lipoprotein cholesterol,
LDL-C) and the anti-glaucoma drugs Timolol and Latanoprost (intraocular
pressure, IOP) were chosen for this investigation. Studies were identified by
a standardized MEDLINE search. RCTs investigating the above identified
medications administered as monotherapy, and in defined dosages, were
included. Publication year, baseline (= pre-treatment value in the treatment
group of interest) and post intervention means, number of patients and the
assignment to experimental or control group were extracted for each study.
Results: A total of 625 citations were screened; 206 met the inclusion
criteria.
The reported effect size of Pravastatin (change of reported effect size in five
years: -3.22% LDL-C, P < .0001), Timolol (-0.56 mmHg, P < .0001) and
Latanoprost (-1.78 mmHg, P = .0074) decreased over time, while there was
no significant change for Atorvastatin (+0.31% LDL-C, P = .8618). Multiple
regression analysis showed that baseline values were the most important
influencing factor; study size or treatment group did not play a significant
role.
Conclusion: The effectiveness of medical therapies reported in RCTs decreases
over time in three of the four investigated pharmaceuticals, caused mainly by
baseline differences. We call this phenomenon "fading of reported
effectiveness".
Under this condition the validity of a meta-analysis may be impaired. Therefore
we propose to observe this phenomenon in future meta-analyses in order to
guarantee a maximum of transparency.
Published: 11 May 2006
A few additional points:
(3) As Gehr et al above make clear there are a variety of reasons for seeing
a reduced effect size especially in medical study. In a pretest-postest
design,
if there is a correlation between baseline values of the dependent
variable and the effect size, then studies with higher baseline values should
have higher effect sizes. That is, in this situation, people with higher levels
of cholestrol (i.e., "sicker" people) will show a greater effect of treatment
than
those people who have lower baseline levels. If earlier studies use people
that
are sicker than later those in latter studies, then we should expect to see
a decrease in effect size over time. This may explain why a "decline effect"
is seen in medical and psychotherapeutic studies; there may be other
reasons for other types of studies (e.g., insufficient power to detect an
effect).
(2) The Lehrer article is about the so-called "decline effect" which was a
term that Rhine apparently coined to explain why certain PSI effects were
initially strongly apparent but disappeared over time. At least one reviewer
suggested that this was a key feature of PSI phenomena (see:
Girden, E. {1962}. A review of psychokinesis (PK), Psychological bulletin
59{5},353--388.
Skeptics and even believers like Eynsenck challenged the Rhine results
though for different reasons (e.g., Eysenck and the British school of
psychics were alarmed at the number of psychically gifted people there
was in America because they were so rare in Britain).
Why Schooler has appropriated this term is "interesting" to say the least.
Why he gets such strange results from his experiments should probably be
investigated by a neutral party.
(3) Taken to its logical extreme, if one applied the decline effect to all
scientific
results (a foolish conclusion IMHO), then science to no more valid knowledge
than any other endeavor. However, this is not what Jonah Lehrer apparently
believes.
Re: [tips] Reliability of the sacred scientific method?
Jim Clark wrote in response to the NYT article about the "failure of science." "Just a couple of observations, not directly on Joan's interesting question about individual areas. > >1. The decline effect is nothing to worry about as it should disappear with >replication! >2. There are some truly egregious examples included here ... really,Rhine and >ESP illustrates the decline effect? >From the perspective of someone who worked in a private practice, which was >presided over by a psychiatrist, I learned that the placebo effect is built >into new medications. When drug reps visited our practice, we were regaled by >the hype that the company sent the rep to give us and, while sitting in on >some drug consults, I noticed that for a period of time, the psychiatrist >would feed all of the hype to the patients (as he called them). Over time, >however, the hype was gradually dropped. I believe that this is a perfect >demonstration of the placebo effect. Stage one: The doctor prescribed this new miracle drug for me Stage two: The doctor prescribed this new drug and told me it really worked well for many people Stage three: The doctor prescribed this drug and told me that some people found it effective. If you chart that out it suggests that in stage one we are dealing with effect of drug + effect of hype Well, you get it. I agree that including Rhine "studies" as part of should not have been included as evidence of the failure of the scientific method and have a number of other questions about some of the information provided in the article. . Robert W. Wildblood, PhD Adjunct Psychology Faculty Germanna Community College drb...@rcn.com --- You are currently subscribed to tips as: arch...@jab.org. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=7526 or send a blank email to leave-7526-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
RE: [tips] Reliability of the sacred scientific method?
Following up on Jim Clark's thoughtful comments, two additional thoughts about Lehrer's article.. (1) Lots and lots of meta-analyses code date of publication as a continuous moderator. Perhaps it's my own fallible memory at work here, but I certainly don't recall any clear-cut tendency across or within fields (at least within clinical and personality psychology, the fields I know reasonably well) for effect sizes to decline consistently over time. But even if my recollection here is faulty and tendentious, which is entirely possible, it shouldn't be terribly difficult to ascertain from published meta-analyses whether there is a general "law of initial results" (as it is sometimes known) whereby effect sizes tend to decline across psychological fields over time. Has anyone done this? If so, I've never seen it (see also Jim Clark's query # 3 below). (2) Admittedly, it's been a few weeks since I read Lehrer's piece, but I was bewildered by why the decline effect, to the extent that it's a dependable generalization (which remains to be seen), somehow suggests that "the scientific method" (whatever that is...I had thought that the ideal of a single, monolithic scientific method was no longer taken seriously among most philosphers of science) is problematic. After all, hasn't it been the fruits of well conducted scientific research that have permitted researchers to pinpoint this effect in the first place ? And won't it be scientific methodology that will ultimately allow researchers to find ways of measuring and potentially controlling for this effect (to the extent that it's genuine)? Much like the file-drawer effect, the decline effect, if it's real, will come to be known not as a dire threat to "the scientific method," but rather as still another source of error to be considered, and ideally controllled, in narrative and quantitative literature reviews. Scott From: Jim Clark [j.cl...@uwinnipeg.ca] Sent: Tuesday, December 28, 2010 6:41 PM To: Teaching in the Psychological Sciences (TIPS) Subject: Re: [tips] Reliability of the sacred scientific method? Hi Just a couple of observations, not directly on Joan's interesting question about individual areas. 1. The decline effect is nothing to worry about as it should disappear with replication! 2. There are some truly egregious examples included here ... really, Rhine and ESP illustrates the decline effect? 3. What proportion of scientific phenomena do the examples represent? That is, how ubiquitous is this effect? 4. There are innumerable areas of science where in fact replication did converge on a correct value for some physical quantity. Hence, is it not rather ridiculous to ask "Is there something wrong with the scientific method?" on the basis of some few (and unknown proportion of all studies) phenomena showing decline versus the almost unlimited array of successful science? 5. Closing provides solice to those who want to ignore science and believe whatever they want to believe. The ideologues will be thrilled. "The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that*s often not the case. Just because an idea is true doesn*t mean it can be proved. And just because an idea can be proved doesn*t mean it*s true. When the experiments are done, we still have to choose what to believe." 6. Of course, I'm only a regular guy, not like the author Lehrer whose internet blurbs refer to his "profound understanding of the human mind." Take care Jim James M. Clark Professor of Psychology 204-786-9757 204-774-4134 Fax j.cl...@uwinnipeg.ca >>> "Joan Warmbold" 28-Dec-10 2:40:47 PM >>> http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer Fascinating article in the New Yorker about the conflicting results researchers find despite all claiming they are using objective, scientific procedures in their research. Where are the likely sources for possible contamination of the results of the various studies discussed? I have some thoughts but would enjoy hearing others first. Joan jwarm...@oakton.edu --- You are currently subscribed to tips as: j.cl...@uwinnipeg.ca. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13251.645f86b5cec4da0a56ffea7a891720c9&n=T&l=tips&o=7522 or send a blank email to leave-7522-13251.645f86b5cec4da0a56ffea7a89172...@fsulist.frostburg.edu --- You are currently subscribed to tips as: slil...@emory.edu. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13509.d0999cebc8f4ed4eb54d5317367e9b2f&n=T&l=tips&o=7524 or send a blank email to leave-7524-13509.d0999cebc8f4ed4eb54d5317367e9...@fsulist.frostburg.edu This e-mail
Re: [tips] Reliability of the sacred scientific method?
Hi Just a couple of observations, not directly on Joan's interesting question about individual areas. 1. The decline effect is nothing to worry about as it should disappear with replication! 2. There are some truly egregious examples included here ... really, Rhine and ESP illustrates the decline effect? 3. What proportion of scientific phenomena do the examples represent? That is, how ubiquitous is this effect? 4. There are innumerable areas of science where in fact replication did converge on a correct value for some physical quantity. Hence, is it not rather ridiculous to ask "Is there something wrong with the scientific method?" on the basis of some few (and unknown proportion of all studies) phenomena showing decline versus the almost unlimited array of successful science? 5. Closing provides solice to those who want to ignore science and believe whatever they want to believe. The ideologues will be thrilled. "The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that*s often not the case. Just because an idea is true doesn*t mean it can be proved. And just because an idea can be proved doesn*t mean it*s true. When the experiments are done, we still have to choose what to believe." 6. Of course, I'm only a regular guy, not like the author Lehrer whose internet blurbs refer to his "profound understanding of the human mind." Take care Jim James M. Clark Professor of Psychology 204-786-9757 204-774-4134 Fax j.cl...@uwinnipeg.ca >>> "Joan Warmbold" 28-Dec-10 2:40:47 PM >>> http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer Fascinating article in the New Yorker about the conflicting results researchers find despite all claiming they are using objective, scientific procedures in their research. Where are the likely sources for possible contamination of the results of the various studies discussed? I have some thoughts but would enjoy hearing others first. Joan jwarm...@oakton.edu --- You are currently subscribed to tips as: j.cl...@uwinnipeg.ca. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13251.645f86b5cec4da0a56ffea7a891720c9&n=T&l=tips&o=7522 or send a blank email to leave-7522-13251.645f86b5cec4da0a56ffea7a89172...@fsulist.frostburg.edu --- You are currently subscribed to tips as: arch...@jab.org. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=7524 or send a blank email to leave-7524-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
[tips] Reliability of the sacred scientific method?
http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer Fascinating article in the New Yorker about the conflicting results researchers find despite all claiming they are using objective, scientific procedures in their research. Where are the likely sources for possible contamination of the results of the various studies discussed? I have some thoughts but would enjoy hearing others first. Joan jwarm...@oakton.edu --- You are currently subscribed to tips as: arch...@jab.org. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=7522 or send a blank email to leave-7522-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
