I'm pondering a recent study by Ebberling (2006) on the effect of reducing soft drink consumption on body mass index (BMI) in teen-agers. There are some things that bug me about their paper, and I'm looking for advice and comment from the list, or at least from those of you who get their jollies from contemplating experimental design and analysis. The rest of you will suddenly remember your appointment with the dentist.
(Warning: I may submit this and there will probably be no place to credit anyone for help they may provide here. If that's ok, read on...) Ebberling et al randomly divided their participants (n = 103) into two groups. One got free no-cal drinks in place of their usual soft drinks (ok, pop to you; they called them SSBs---sugar-sweetened beverages) which the kids previously had guzzled with abandon. The other (control) group continued to knock back the pop. Twenty-five weeks later they evaluated change in BMI. They analyzed their results using multiple linear regression, with which I'm not familiar. Nevertheless the result was clear: a small, non- significant decrease in BMI for the experimental group. So did they conclude that they had failed to show that eliminating pop reduced BMI? Not a chance. What they actually concluded was the opposite, namely "Decreasing the consumption of SSBs seems to be a promising strategy for the prevention and treatment of overweight adolescents." They did this by invoking the dreaded post-hoc and sub-group analysis manoeuver. They noted that baseline BMI was significant as an "effect modifier", meaning (I think) that the heavier they started out, the more weight they lost. I think this must have been for the groups combined. They did show separate scatterplots (Fig 2 A and B) of BMI change vs baseline BMI for control and experimental groups with regression lines for each. The points were all over the place, and there was nothing obviously different between the groups, at least to my eyeball. The experimental line did head down with increasing BMI; the control didn't. Then they asserted (just like that) that "the intervention effect was significant for baseline BMI > 30", citing only Figure 2C. Figure 2C shows a "95% confidence band on difference between study groups". I believe this line must be the difference between the two regression lines with 95% limits displayed. The confidence limits stop including zero at a BMI of 30. So they concluded they had a significant effect after a BMI of 30. Finally my question. Is this kosher? By scanning down the difference regression line until you get to a place you like, isn't that the equivalent of doing multiple tests without correction? Of course, if I'm right, I don't know how I'd express this so I'd sound as though I know what I'm talking about. Curiously, this finding isn't mentioned in their abstract. There they only mention another data-dredged comparison. They divided their subjects into thirds based on pre-experimental BMI. They found that that for the heaviest third, lo and behold, there was a significant post-treatment drop in BMI compared with the control group. Now this implies three separate comparisons, and if they Bonferroni- corrected, it wouldn't fly. But that doesn't bother me, because I think it's a reasonable prediction beforehand that only the heaviest would benefit from kicking pop (even if, as they presented it, they hadn't actually planned this). No, what bothers me is why they chose thirds. Why not fourths, or divide their groups in half? Actually, overweight in adolescents is conventionally defined as a BMI above 85% of the population at that age. Elsewhere, they make use of that 85% criterion, but not here. If they had, I would accept their result. But dividing into thirds makes me wonder, especially as the experimental group has one really nice outlier (decrease in BMI) which falls just within the experimental sub-group as they've defined it. Shift the divider slightly back, and it includes nasty data best left out; shift it forward, and that nice point goes to the control. So I asked them politely for the raw data to do my own analysis. They refused. Given that the APA explicitly tells its authors to fill such requests promptly and cooperatively (or something like that), that doesn't seem too welcoming of them. does it? One more thing. At a critical point in their Results, they say this: "Among the subjects in the upper baseline-BMI tertile...BMI change differed markedly between the intervention...and the control...(p= .03), whereas no significant group difference was seen for the subjects in the middle and lower tertiles (p =.04 for interaction). What does this mean? They explained it to me, but I'm still not sure. Stephen Ebberling, C. et al (2006). Effects of decreasing sugar-sweetined beverage consumption on body weight in adolescents: a randomized, controlled pilot study. Pediatrics 117, 673-- ----------------------------------------------------------------- Stephen L. Black, Ph.D. Department of Psychology Bishop's University e-mail: [EMAIL PROTECTED] Lennoxville, QC J1M 1Z7 Canada Dept web page at http://www.ubishops.ca/ccc/div/soc/psy TIPS discussion list for psychology teachers at http://faculty.frostburg.edu/psyc/southerly/tips/index.htm ----------------------------------------------------------------------- --- To make changes to your subscription go to: http://acsun.frostburg.edu/cgi-bin/lyris.pl?enter=tips&text_mode=0&lang=english
