Two reactions: It is not so much the failure of peer review to consider the true value of replications, but a failure of long-standing editorial/journal policy, which is then passed down to referees. Also, wrt the blog, an editor may not be able to compel an author to release his/her data, but if data availability is a precondition to publication and the author refuses to release data, that editor has the power to issue an expression of concern or even a retraction of that author's work. These outcomes, which have already occurred in some instances, usually have significant negative consequences for the authors.
Miguel ________________________________________ From: William Scott [wsc...@wooster.edu] Sent: Monday, October 23, 2017 7:45 PM To: Teaching in the Psychological Sciences (TIPS) Subject: Re: [tips] When the Impossible is Shown to be Impossible: A Case Study in Failing to Replicate There are those who would argue that it is the peer review process that has failed us by avoiding the publication of replication studies in favor of flashy "discovery". As well many of the critics who have been accused of cyber-bullying have argued that when the arguments have been made in the abstract without specifically calling out individual researchers, there has been little consequent action, but now they are getting results. Here is an interesting post by James Heathers about what it feels like to be a so-called cyber-bully. https://medium.com/@jamesheathers/the-buck-stops-nowhere-8284a57c88c9 [https://cdn-images-1.medium.com/max/1600/1*NPuM_JhIXSZIr3AcZ_S6mQ.jpeg]<https://medium.com/@jamesheathers/the-buck-stops-nowhere-8284a57c88c9> The Buck Stops Nowhere – James Heathers – Medium<https://medium.com/@jamesheathers/the-buck-stops-nowhere-8284a57c88c9> medium.com I wrote them all down in shorthand, then forgot about them. At least, for a while. The reason why is simple: I have a job and writing, well, writing takes time. The ... ________________________________ From: Helweg-Larsen, Marie <helw...@dickinson.edu> Sent: Monday, October 23, 2017 1:29:30 PM To: Teaching in the Psychological Sciences (TIPS) Subject: RE: [tips] When the Impossible is Shown to be Impossible: A Case Study in Failing to Replicate I think they are bullies (and I am not trying to deflect attention from myself). And I don’t think it is good for science. We already have a process by which we can discuss replication (or problems with studies) namely by publishing in peer reviewed journals focusing on the evidence instead of making personal attacks. Blogs are not peer reviewed and are just a few individual people’s beliefs without any checks and balances. We should focus on making our science better not on destroying people’s careers. The field is full of underpowered studies as well as findings that will probably not hold up under replication (although lots of replication studies are themselves under powered or presented with glee as if one failed replication can provide conclusive evidence). We should fix these problems (in our training of undergraduate and graduate students, journal editorial policies, etc.) and not by attacking individual researchers. Marie Marie Helweg-Larsen, Ph.D. Professor l Department of Psychology Kaufman 168 l Dickinson College Phone 717.245.1040 l Fax 717.245.1971 http://blogs.dickinson.edu/helwegm/ From: Christopher Green [mailto:chri...@yorku.ca] Sent: Sunday, October 22, 2017 2:52 PM To: Teaching in the Psychological Sciences (TIPS) <tips@fsulist.frostburg.edu> Subject: Re: [tips] When the Impossible is Shown to be Impossible: A Case Study in Failing to Replicate So far as I know, Marie, there are no bullies here. They accusation is an attempt by some people whose work has not held up to deflect attention from themselves. True, there are some people who are blunt about what they take to be obvious points that every scientist should readily accept (e.g., phenomena that aren’t replicable are not scientifically valid phenomena). There are some people who have become exasperated when they have spent considerable time and effort showing that some supposed finding has serious flaws, and all they get back for their efforts is dodging and weaving. We have all been taught since we were undergrads that replication is the gold standard of science. So, it is bizarre to have people act as though attempts at replication are some sort of unfair “attack" on them. It is, on the contrary, a central pillar of scientific practice (even if psychologists have long been too lax about it). We should be conducting far more attempts at replication, not trying to shut down the little that is now, finally, started being done. If we are going to feel sorry for someone, it shouldn’t be for the people who did poor work in the first place, then, when it was shown to be poor, played the victim instead of the culprit. We should feel sorry for all those early career researchers — graduate students, post-docs, untenured profs — who assumed that the senior researchers, journal reviewers, and editors knew what they were doing and, so, attempted to use the published literature as a basis for their own research. But they were unable to re-produce the supposedly established results and so, in the end, did not graduate, did not get permanent positions, and did not get tenure. That’s who I feel sorry for. If we had been more vigilant in the first place, these poor people would not have found themselves in so untenable a situation in the first place. Chris ….. Christopher D Green Department of Psychology York University Toronto, ON M3J 1P3 Canada 43.773895°, -79.503670° chri...@yorku.ca<mailto:chri...@yorku.ca> http://www.yorku.ca/christo orcid.org/0000-0002-6027-6709 ………………………………... On Oct 22, 2017, at 1:18 PM, Helweg-Larsen, Marie <helw...@dickinson.edu<mailto:helw...@dickinson.edu>> wrote: The story IS about Amy Cutter and her experiences as a person, a woman, a social psychologist, etc. It is a fascinating story exactly because the article draws on her experiences to make broader points about gender, a new online bullying culture and how that culture can change people (or in this case at least one person). It seems that the bullies themselves don’t understand the idea of replication or the fact the single-replications are often vastly underpowered (see attached article). And of course bullies are righteous in their work to “fix” things. I think Amy Cutter’s experiences are echoed in the field now and it scares me. Marie Marie Helweg-Larsen, Ph.D. Professor l Department of Psychology Kaufman 168 l Dickinson College Phone 717.245.1040 l Fax 717.245.1971 http://blogs.dickinson.edu/helwegm/ From: Christopher Green [mailto:chri...@yorku.ca] Sent: Saturday, October 21, 2017 12:49 PM To: Teaching in the Psychological Sciences (TIPS) <tips@fsulist.frostburg.edu<mailto:tips@fsulist.frostburg.edu>> Subject: Re: [tips] When the Impossible is Shown to be Impossible: A Case Study in Failing to Replicate I don’t think it was a choice. I think it was a journalistic reflex — go for the personal, the emotional, because it draws in readers. The same reason they pepper stories about murder rates with profiles about individual murder victims. But in this particular case, it directed readers away from the real story. They came out feeling sympathetic with the Harvard scientist whose work had been (correctly, let us remember) called into question instead of gasping for air at the gaping hole that was staring them in the face. Cuddy’s reaction might be worth a paragraph, but here we got this lavish, heroic treatment of her hard childhood, her teenage car accident, her difficulties returning to school, and her rise, against all odds, to a Princeton PhD and a Harvard professorship. But that’s not the story. The story is that nearly all of the psychological research of the past 50+ years is now under a cloud of suspicion because we (nearly all) acted badly — sometimes disingenuously — with respect to statistical analysis. Instead of demanding that those who wanted the title of “scientist” really internalize the math, the probability theory, and the critical assumption that underlie data analysis, we turned it into a perfunctory "cookbook” that made it easy for people to think (or rationalize) that there were no real consequences for cutting corners — replacing butter with margarine, leaving out the dash of salt, trying to replace sugar with some sort of artificial sweetener — and what we ended up with was a hot mess that no one who is serious about these things can stomach anymore. And it’s not as though any of this is new (though everyone who gets caught keeps saying they had never heard of it before). The computational facility that allows for the simulation studies of the past few years is new, but Paul Meehl and Jacob Cohen and David Bakan, and Jum Nunnally, and Bill Rozeboom. and Bob Rosenthal have all been telling us this stuff since the 1960s and 1970s (and through the 1980s and 1990s). (Heck, there are even couple of articles by a guy named Berkson from 1938 and 1942). But even most of the psychologists who bothered to read this material (I was lucky that it was assigned to me as an undergrad — Thanks TIPSter Stuart McKelvie!) decided to harumph and go on pretty much as before — a little worse each decade as the designs got more complex. It is a massive s#*t sandwich, and it threatens the credibility of not only psychology, but of a ton of medical research (classic cancer experiments aren’t replicating), not to mention the rest of the social/behavioral sciences (every single one of which — except economics — shows an explicable hump just inside the .05 p-value, when you survey the literature — graph here: https://twitter.com/jtleek/status/890180014733492225). And we’re suppose to focus on poor Amy Cuddy’s feelings? The thing is (for those of you inclined to think that this is “really” a gender issue), Cuddy is way old news now. Brian Wansink’s food lab at Cornell is having to correct and retract dozens of articles — research that has already been (mis-)used to change the practices of school cafeterias and the like. For heaven’s sake, he was so “sloppy" that one of his most famous studies on the eating behaviour of 8-11 year olds turned out to have actually been run on 4-5 year olds! I wouldn't expect him or his lab to last another year. There are half a dozen other prominent labs under scrutiny of this kind as well. Psychology is in deep trouble. Much worse than we know yet, especially once the politicians who hate social science already get ahold of it. Amy Cuddy’s feelings won’t amount to a hill of beans once the full scale of this thing is understood. Chris ….. Christopher D Green Department of Psychology York University Toronto, ON M3J 1P3 Canada 43.773895°, -79.503670° chri...@yorku.ca<mailto:chri...@yorku.ca> http://www.yorku.ca/christo orcid.org/0000-0002-6027-6709<http://orcid.org/0000-0002-6027-6709> ………………………………... On Oct 21, 2017, at 11:51 AM, Michael Palij <m...@nyu.edu<mailto:m...@nyu.edu>> wrote: On Fri, 20 Oct 2017 20:03:43 -0700, Christopher Green wrote: >Interesting article, but I thought it made the usual journalistic error >of personalizing the story too much, making readers come away >feeling for the people instead of understanding the problem. Although I agree that there is too much personalization (I believe this is done so that the reader can (a) see the person described as more as a relatable person), and (b) borrowing some of the writing conventions from fiction to make what would be a dry nonfiction story more interesting. In an article like this, I can accept it. However, after a long hiatus, I am teaching Introduction to Psych and find that the textbook is filled with too many personalized examples or what I would call "cutesy" examples that to simplify the presentation and make it more accessible to undergraduates. I think that this helps students maybe to understand the presentation (or develop the illusion of understanding) which will be challenged when they try to read actual empirical research articles (e.g., "hey, where's the main character? Where's the dramatic action and tension? etc.). That being said, from a 'qualitative research" perspective, I think it is interesting to see what a person who has published a piece of research that cannot be replicated is feeling and thinking. Amy Cuddy appear to be a highly capable and skilled person who though she has given up on academia (at least for now) will make out all right (kinda like John B. Watson, if you know what I mean). The larger issue of the replication crisis, the pressures to publish popular (to the general public not the scientific community) articles, and to get external funding, I think, will be lost on the general reader. Seeing how these factors affect a likable character is perhaps the only way to show what these factors are and can do to a person. Another thing to keep in mind is that this article uses a person who is basically good but was incautious. It might have been more interesting if the person being covered was Diederik Stapel who seems to be a much darker person and who appears to have inentionally done bad things. One could say that bad science arises from good people doing "incompetent" research and "bad" people doing fraudulent research, among other things (e.g., following fads that focus on the weong things). -Mike Palij New York University m...@nyu.edu<mailto:m...@nyu.edu> --- You are currently subscribed to tips as: chri...@yorku.ca<mailto:chri...@yorku.ca>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=430248.781165b5ef80a3cd2b14721caf62bd92&n=T&l=tips&o=51592 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-51592-430248.781165b5ef80a3cd2b14721caf62b...@fsulist.frostburg.edu<mailto:leave-51592-430248.781165b5ef80a3cd2b14721caf62b...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: helw...@dickinson.edu<mailto:helw...@dickinson.edu>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13234.b0e864a6eccfc779c8119f5a4468797f&n=T&l=tips&o=51593 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-51593-13234.b0e864a6eccfc779c8119f5a44687...@fsulist.frostburg.edu<mailto:leave-51593-13234.b0e864a6eccfc779c8119f5a44687...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: chri...@yorku.ca<mailto:chri...@yorku.ca>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=430248.781165b5ef80a3cd2b14721caf62bd92&n=T&l=tips&o=51605 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-51605-430248.781165b5ef80a3cd2b14721caf62b...@fsulist.frostburg.edu<mailto:leave-51605-430248.781165b5ef80a3cd2b14721caf62b...@fsulist.frostburg.edu> <Maxwell et al, 2015.pdf> --- You are currently subscribed to tips as: helw...@dickinson.edu<mailto:helw...@dickinson.edu>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13234.b0e864a6eccfc779c8119f5a4468797f&n=T&l=tips&o=51606 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-51606-13234.b0e864a6eccfc779c8119f5a44687...@fsulist.frostburg.edu<mailto:leave-51606-13234.b0e864a6eccfc779c8119f5a44687...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: wsc...@wooster.edu<mailto:wsc...@wooster.edu>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13058.902daf6855267276c83a639cbb25165c&n=T&l=tips&o=51618 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-51618-13058.902daf6855267276c83a639cbb251...@fsulist.frostburg.edu<mailto:leave-51618-13058.902daf6855267276c83a639cbb251...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: ro...@stjohns.edu<mailto:ro...@stjohns.edu>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=1632838.7e62b84813297f170a6fc240dab8c12d&n=T&l=tips&o=51619 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-51619-1632838.7e62b84813297f170a6fc240dab8c...@fsulist.frostburg.edu<mailto:leave-51619-1632838.7e62b84813297f170a6fc240dab8c...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: arch...@mail-archive.com. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=51624 or send a blank email to leave-51624-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
