I was put in mind of Jeffrey Cummings work simulating the instability of p values. He found that even for fairly robust experimental designs, replication fails about 50% of the time. I think that what this new study really found is that many people use research designs that are either not robust enough or too complex for the available data. That lowers the failed replication rate from about 50% down to the 36% found here.
Here is a short video where Cummings demonstrates his findings. http://youtu.be/5OL1RqHrZQ8 Best, Chris ----- Christopher D. Green Department of Psychology York University Toronto, ON M6C 1G4 Canada chri...@yorku.ca > On Aug 29, 2015, at 12:28 PM, Wuensch, Karl Louis <wuens...@ecu.edu> wrote: > > > > I don’t think the replication results are very surprising. It > seems to be closely related to the “file drawer problem” and publication > policy. Suzy Q and a bunch of others are, independently, studying the same > effect. The effect happens to be pretty small, and the sample sizes Suzy and > others use not very large. Sampling error leads half of the researchers to > underestimate the magnitude of the effect, half overestimating it. Suzy is > the lucky one who gets data leading to an overestimate sufficiently large to > produce significant results. Now this is not a Type I error, there is an > effect, but she has, unknowingly, overestimated the magnitude of the effect. > It should not be surprising that when others try to replicate it, they get > much smaller estimates of the magnitude of the effect. > > Yes, meta-analysis is called for here, if you can get the data out > of all those file drawers. > > Cheers, > > From: Jim Clark [mailto:j.cl...@uwinnipeg.ca] > Sent: Friday, August 28, 2015 3:41 PM > To: Teaching in the Psychological Sciences (TIPS) > Subject: Re: [tips] PSYC failed the replication test > > > > Hi > > Some notable lapses I think. No mention of meta-analysis that I saw. And the > Science editor's comment that marginal results not be published is exactly > wrong. Every finding should be published so that studies can be aggregated. > Psychology made a big mistake when it turned from small papers consisting > mostly of methods and results for single studies to mega papers with multiple > studies and (inflated?) intros and discussions. Thes problems are exacerbated > by calls for larger and larger Ns. > > Jim > > Sent from my iPhone > > On Aug 28, 2015, at 2:33 PM, "Dennis Goff" <dg...@randolphcollege.edu> wrote: > > > > You all might find an article published on the Five Thirty Eight website > (link http://fivethirtyeight.com) useful when talking about this issue with > students. The article includes a very nice interactive demonstration of > p-hacking that I think will help students to understand that concept. Beyond > that, the article takes a balanced approach to the problems of replicability, > the growth in the number of journal articles being published, fraud and other > related topics. The title is Science Isn’t Broken by , Christie Aschwanden. > Here is the link http://fivethirtyeight.com/features/science-isnt-broken/ > > Dennis > > ---------------------------------------------------------- > Dennis M Goff > Charles A Dana Professor of Psychology > Chair Department of Psychology > Office: (434) 947-8547 > > Randolph College > Founded as Randolph-Macon Woman's College in 1891 > 2500 Rivermont Avenue > Lynchburg, VA 24503 > www.randolphcollege.edu > > > > > From: Rick Froman [mailto:rfro...@jbu.edu] > Sent: Friday, August 28, 2015 2:17 PM > To: Teaching in the Psychological Sciences (TIPS) <tips@fsulist.frostburg.edu> > Subject: RE:[tips] PSYC failed the replication test > > > > It is true that we only got a D+ L but 68% is quite a bit higher J than some > of the estimates of reproducibility I have heard bandied about (which are > close to zero). > > Rick > > Dr. Rick Froman > Professor of Psychology > Box 3519 > John Brown University > 2000 W. University Siloam Springs, AR 72761 > rfro...@jbu.edu > (479) 524-7295 > > > From: Wuensch, Karl Louis [mailto:wuens...@ecu.edu] > Sent: Friday, August 28, 2015 12:59 PM > To: Teaching in the Psychological Sciences (TIPS) <tips@fsulist.frostburg.edu> > Subject: [tips] PSYC failed the replication test > > > > This article is going on the reading list for my grad students in > stats. Once we have covered power and publication biases this article > should, I hope, lead to some lively discussion. > > http://www.sciencemag.org/content/349/6251/aac4716 > > Cheers, > <image001.jpg> > Karl L. Wuensch, Professor and ECU Scholar/Teacher, Dept. of Psychology > East Carolina University, Greenville NC 27858-4353, USA, Earth > Associate Editor, Society & Animals > Voice: 252-328-9420 Fax: 252-328-6283 > http://core.ecu.edu/psyc/wuenschk/klw.htm > > --- > > You are currently subscribed to tips as: rfro...@jbu.edu. > > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13039.37a56d458b5e856d05bcfb3322db5f8a&n=T&l=tips&o=46469 > > (It may be necessary to cut and paste the above URL if the line is broken) > > or send a blank email to > leave-46469-13039.37a56d458b5e856d05bcfb3322db5...@fsulist.frostburg.edu > > --- > > You are currently subscribed to tips as: dg...@randolphcollege.edu. > > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13169.2bfd597f06c032f81efb35e857e2dd91&n=T&l=tips&o=46470 > > (It may be necessary to cut and paste the above URL if the line is broken) > > or send a blank email to > leave-46470-13169.2bfd597f06c032f81efb35e857e2d...@fsulist.frostburg.edu > > --- > > You are currently subscribed to tips as: j.cl...@uwinnipeg.ca. > > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13251.645f86b5cec4da0a56ffea7a891720c9&n=T&l=tips&o=46472 > > (It may be necessary to cut and paste the above URL if the line is broken) > > or send a blank email to > leave-46472-13251.645f86b5cec4da0a56ffea7a89172...@fsulist.frostburg.edu > > --- > > You are currently subscribed to tips as: wuens...@ecu.edu. > > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=13060.c78b93d4d09ef6235e9d494b3534420e&n=T&l=tips&o=46473 > > (It may be necessary to cut and paste the above URL if the line is broken) > > or send a blank email to > leave-46473-13060.c78b93d4d09ef6235e9d494b35344...@fsulist.frostburg.edu > > --- > > You are currently subscribed to tips as: chri...@yorku.ca. > > To unsubscribe click here: > http://fsulist.frostburg.edu/u?id=430248.781165b5ef80a3cd2b14721caf62bd92&n=T&l=tips&o=46481 > > (It may be necessary to cut and paste the above URL if the line is broken) > > or send a blank email to > leave-46481-430248.781165b5ef80a3cd2b14721caf62b...@fsulist.frostburg.edu > --- You are currently subscribed to tips as: arch...@mail-archive.com. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=46483 or send a blank email to leave-46483-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
