Christopher Green wrote: [snip] >but of a ton of medical research (classic cancer >experiments aren’t replicating), not to mention >the rest of the social/behavioral sciences (every >single one of which — except economics — shows an >[in]explicable hump just inside the .05 p-value, when >you survey the literature — graph here: > >https://twitter.com/jtleek/status/890180014733492225 ><https://twitter.com/jtleek/status/890180014733492225>).
I must admit that it is a pretty graph but it really should be called a "ridgeplot" instead of a "joyplot" ("joy" comes from the band "Joy Division" which actually refers to a horrendous situation -- google it if you really want to know). But maybe I'm too skeptic of things but if you actually get the data (its in a R data formatted file which with the right wizardry can be imported into SPSS), all 3+ million cases and just select the sources for psychology and sociology (N=78,6K, that's what the person who put the data together used in the "field" variable) you will see the following if you code p-values into 1 unit bands (i.e., 5.00 to 4.00 -- p-values are expressed as percentages) all the way down to p<.00 (no, the p-value is not zero but really, really, really tiny -- maybe a little too tiny?). Why, all of the p-values are less than .01. How can that be if there is supposed to be a "hump" in the .04-.05 interval? Maybe not in this dataset. What do we see if we include all of the cases (3.6+ million cases)? The top row (1) contains sources with a p-value between 4-5 and there is only one such case. There are no results with a p-value between 3-4. There are 3 values in the intervanl 2-3, and 9 values in the interval 1-2. There are over 3.6 million cases with p-values between 0-1 and 20.8K with a p-value less than .01. This doesn't seem to match the ridgeplot that Jeff Leek has but we have analyzed the data in different ways and I admit that maybe I made an error somewhere but reviewing my work, I don't see any errors. So, no "hump" in the interval .04-05 and as the p-values decrease, the frequencies increase up to the interval .01-.00. For p<= .001, there is a significant drop. So, what pattern are we supposed to see if these are terrible studies? If one looks at the raw p-values what is most striking is how many cases with p< .00000001 there are (are p-value that are all zero up to the 16th decimal value). This is really peculiar because either the effect sizes are Godzilla sized or the sample sizes are Godzilla sized or both (i.e., Godzilla^2). Yes, I think there is a replication crisis but I don't think that the data that Jeff Leek used to make his pretty ridgelplot graph provides any support for it. But what do I know, right? Anyone can download the data from the Github website and do the analyses I report either in R or SPSS or whatever statistical software one uses. I await the coming results. By the way, because the tables above are jpegs, this post probably won't make it into the Mail Archive or the digest but you should get it if you get post via direct email. -MIke Palij New York University m...@nyu.edu --- You are currently subscribed to tips as: arch...@mail-archive.com. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=51597 or send a blank email to leave-51597-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu