Mike Williams wrote that " Since all the dependent measures involve a judgement
by the patient or the investigator that the disorder got better or worse, they
are all influenced by the expectation bias that the treatment worked." The
assertion ("all of the dependent measures...") in the first half of Mike's
sentence is false. Many psychotherapy outcome studies in the child and adult
literature have used observer ratings, including ratings by family members and
friends (what Kazdin termed "social validation"), measures of out-of-session
behaviors, or both, and found positive and clinically robust results. See just
one example in the child literature:
http://www.springerlink.com/content/u1mg416h25n29405/fulltext.pdf
See their Table 1, which yielded larger effect sizes for psychotherapy for
indicators that were more objective (e.g., arrest data) than for self-reports
or objective ratings.
See also meta-analyses of the literature on psychotherapy for anger problems:
http://onlinelibrary.wiley.com/doi/10.1093/clipsy.10.1.70/pdf
which shows approximately equal effect sizes for self-versus observer reports,
as well as marked effects on physiological indicators. There are many other
examples. Again, there are plenty of interesting issues to raise here (e.g.,
degree of blindness of observers to treatment assignment, susceptibility of
physiological indicators to expectancy effects), but Mike's hypothesis that the
results are due simply to subject expectancies and/or demand characteristics
does not square with an enormous body of meta-analytic data on psychotherapy
outcome showing positive effects on observer ratings (which are often conducted
blindly in RCTs of psychotherapy outcome) and largely objective indicators of
real-world behavior.
Mike W. is right to raise useful questions regarding the internal validity
of psychotherapy designs, but I agree with Mike P. that he is wrong to
categorically dismiss all of them simply as "invalid." Surely, no study is
perfect, but many of them yield highly useful inferences. In addition to Mike
P.'s endorsement of Don Campbell's writings on internal validity, I'd like to
add Campbell's helpful principle of the heterogeneity of irrelevancies. The
most helpful inferences derive the consilience of multiple independent studies,
all with largely offsetting flaws.
Scott O. Lilienfeld, Ph.D.
Department of Psychology, Room 473
Emory University
36 Eagle Row,
Atlanta, Georgia 30322
[email protected]; 404-727-1125
-----Original Message-----
From: Mike Palij [mailto:[email protected]]
Sent: Tuesday, September 13, 2011 9:26 AM
To: Teaching in the Psychological Sciences (TIPS)
Cc: Mike Palij
Subject: RE:[tips] Clinical training: Boulder and Denver
Just a few points:
(1) I guess that before going to bed folks should read Campbell and Stanley
and/or Cook and Campbell and/or Shadish's updating of their material because
the issues raised by Mike Williams fall under the topic of "threats to internal
validity". Specifically,
(a) diffusion or imitation of treatments and (b) compensatory rivalry or
resentful demoralization. In both of these situations, knowledge of the
different treatments used in a between-subjects design, can affect how subjects
respond or fail to respond to treatments. It should be noted that drug
treatment studies can be conducted with within-subject designs such as
crossover designs where one group receives a drug treatment first and, after a
washout period, receives a placebo treatment. Another group has placebo first
and drug later. In any event, a competent researcher will make sure that the
design they use addresses threats to the different types of validity involved
in the study and try to make sure that their effect is negated or minimized.
Here is link to Pedhazur & Schmelkin's treatment of validity issues in their
text "Measurement, Design, and Analysis":
http://tinyurl.com/PedhazurIntValid
(2) It might be a somewhat useful to follow the research heuristic that "all
treatment/medication studies involving human are invalid"
but, as with all heuristics, there will be situations where it fails and
situations where it is right but for the wrong reasons. In the example that
Williams uses below, it is clear that in an inpatient/ward study, one has to
guard against the threats I identify above. But if one uses an outpatient
population where the participants have no contact with each other, it is hard
to see the merit in Williams' critique.
There are many different ways to conduct a drug/treatment/intervention study
and the quality of that research is dependent upon what the person in charge
knows about research design, the different types of validities involved as well
as threats to them, and the degree of control one has over the research
situation.
(3) A minor point: I would assert that though one's own personal experience
is, perhaps, a useful guide to think about things, it does not necessarily
constitute a valid guide. It is helpful to remember that "anecdotes do not
constitute data" (unless, of course, one is doing a qualitative study where
subject's statements are the data).
Williams does not provide the results of the study he was involved which is odd
because he suggests that the study should have produced statistically
significant results. If the study did, the logical follow-up question would be
was the effect obtained with other types of designs (e.g., out patient studies
where subjects did not have contact with each other). The objection that
Williams makes is limited to a particular research situation and does not hold
for other designs where the threats are not relevant. This one point
invalidates the "universality" of his claims that all drug studies are all
invalid.
(4) I have conducted the statistical analysis for a few drug studies as
represented in the following publications:
(a) The effectiveness of antianxiety medications (after participating receiving
a sodium lactate infusion to see if it provoked a panic attack):
http://tinyurl.com/Liebowitz1984
(b) Whether fenfluramine positively affects behavior and attention in young
children with autism:
http://www.springerlink.com/content/t152543684578671/
(c) The use of bromocriptine in the treatment of cocaine addiction:
http://onlinelibrary.wiley.com/doi/10.1111/j.1521-0391.1997.tb00392.x/abstract
One of the things that one learns from analyzing the data from such studies is
that subjects can have very different responses to treatment, sometimes
contrary to what "commonsense" would lead one to expect, such as finding that
the placebo condition has MORE side effects than the drug condition. One
should also keep in mind that though one can engage in a "guessing game" about
which treatment one is receiving, people are not 100% accurate in making such
predictions and there probably are drugs where the effects are not perceptible
or the effects build up slowly over time and the day to day changes are not
noticeable (think of the frog in pot on the stove where the heat is low but
eventually comes to a boil).
In summary, many research studies are flawed, some fatally flawed and useless
while the flaws in others are not fatal. One can always check the systematic
reviews on the Cochrane Collaboration website (www.cochrane.org ) to see how
many crappy treatment/intervention studies have been published because the
researchers did not address the different types of threats to the validity of
the study. Trying to claim that all studies are invalid or all studies are
valid is logically invalid from an inductive perspective -- it is as foolish as
claiming that "All swans are white". Those without experience with black swans
will swear the "all swans are white" if that has been their lifelong experience.
-Mike Palij
New York University
[email protected]
------------------ Original Messages ------------------------------ On Mon, 12
Sep 2011 23:24:29 -0700, Mike Williams wrote:
Hello All.
When I was a grad student, we were conducting a clinical trial of Imipramine vs
Xanex in the treatment of severe depression. The study was conducted on an
inpatient research unit in the hospital. The patients lived there and I
noticed that they would sit in the day room in the evenings and discuss their
treatment. Although the medications were assigned randomly and the researchers
did not know the assignment, the patients with dry mouth and constipation knew
they were taking the medications. Those given placebo knew this because they
did not suffer constipation and dry mouth (the anticholinergic side effects).
The patients knew which treatment they were receiving and they communicated
this to the investigators because the investigators constantly monitored the
side effects. The constant monitoring of side effects unblinds the study.
This happens in every clinical trial of psychotropic medications.
This problem is even more obvious in every clinical trial of psychotherapy.
All these studies are invalid.
I could explain why they are invalidated by referring to the gigantic
literature on expectation biases.
Since all the dependent measures involve a judgement by the patient or the
investigator that the disorder got better or worse, they are all influenced by
the expectation bias that the treatment worked. I think many subjects want to
help the researchers and they endorse small positive changes on the dependent
measures. The people who get placebo behave consistent with this because they
know they never got treatment.
All the investigators have to do is anonymously survey the subjects. The
results will blow their minds. To my knowledge, this obvious, simple
assessment has never been made.
Now you may be able to understand why the treatment effect size today for
antidepressants is the same as the placebo effect for some studies in the past
- its all noise.
Mike Williams
______________________________________________________
Hi Mike:
This is a very interesting point but I am not sure that I follow the argument
completely. Please expand your argument, dotting the 'i's and crossing the
't's.
Ken
On 9/12/2011 3:00 AM, Mike Wiliams wrote:
Clinical Psychology psychotherapy and psychotropic medication therapies will
never have sufficient empirical support simply because the subjects are never
blind to the treatment condition.
*************************
All the
investigators are doing is training the subjects to endorse change on the
dependent measures.
**************************
That's why the meta-analyses conclude that
any therapy is effective. I have never seen an analysis that addressed this
research problem. It's similar to the obesity researchers who never notice that
their entire field is based on the dieting behavior of young women.
Mike Williams
Drexel University
---------------------------------------------------------------
---
You are currently subscribed to tips as: [email protected].
To unsubscribe click here:
http://fsulist.frostburg.edu/u?id=13509.d0999cebc8f4ed4eb54d5317367e9b2f&n=T&l=tips&o=12615
or send a blank email to
leave-12615-13509.d0999cebc8f4ed4eb54d5317367e9...@fsulist.frostburg.edu
________________________________
This e-mail message (including any attachments) is for the sole use of
the intended recipient(s) and may contain confidential and privileged
information. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution
or copying of this message (including any attachments) is strictly
prohibited.
If you have received this message in error, please contact
the sender by reply e-mail message and destroy all copies of the
original message (including attachments).
---
You are currently subscribed to tips as: [email protected].
To unsubscribe click here:
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=12619
or send a blank email to
leave-12619-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
