Hi Stephen- I share your concern about using logistical regression as well as deciding, a posteriori, that .06 is sufficient grounds to reject the null. However, to play devil's advocate, there might be some justification for controlling for things like disruptiveness and parental occupation IFF you could show that: 1. Criminal record was a low base rate phenomenon and 2. There was prior evidence that showed a link between these factors and criminal record. That said, it looks to me like another case of data mining that probably won't hold up under replication.
-Don. Don Allen Dept. of Psychology Langara College 100 W. 49th Ave. Vancouver, B.C. Canada V5Y 2Z6 Phone: 604-323-5871 ----- Original Message ----- From: [EMAIL PROTECTED] Date: Monday, November 12, 2007 9:22 am Subject: [tips] Question for the stats-savvy To: "Teaching in the Psychological Sciences (TIPS)" <tips@acsun.frostburg.edu> > I'm pondering a study which purports to show that a specific early > intervention works in improving school performance and keeping > kids from > crime. The design is good, although the results, while positive, > are > weak. > > But there does appear to be some possibility of fiddle in the way > the > results are analyzed. A group of high-risk children were randomly > assigned to intervention and control groups. 15 years after > treatment, > two objective measures of success were obtained: high school > graduation, > and criminal record. > > I'd have gone with a simple independent test of proportions for > each > measure. When I did, high school graduation was significant, but > criminal > record was not (p=.09).They didn't do this. They used logistic > regression > in each case, for graduation controlling both for parental > occupation and > disruptiveness, and for criminal record, controlling only for > parental > occupation, > > The one for graduation was significant, but for criminal record it > was p > = .06, which they accepted as significant, "although marginal". > > That's not what's bugging me. What I want to know is if it's > justifiable > to control for things like parental occupation and disruptiveness > in a > randomized study. This is ok for correlational research, but why > would > you want to do it in a randomized study where such factors are > already > eliminated through randomization? > > It seems to me they may have done this because it got them close > enough > to the magic p= .05 to claim it anyway. If that's the only reason, > I > don't think it's right. Also, once you've controlled in that way, > wouldn't that somehow limit the generality of your findings, that > they're > now restricted to an artificial type of homogeneous population > resulting > from the controlling? Is there a cost to doing it this way when > you don't > have to? > > As our Michael would say, send me something. > > Stephen > > ----------------------------------------------------------------- > Stephen L. Black, Ph.D. > Professor of Psychology, Emeritus > Bishop's University e-mail: [EMAIL PROTECTED] > 2600 College St. > Sherbrooke QC J1M 1Z7 > Canada > > ------------------------------------------------------------------- > ---- > > --- > ---
