Christopher Green wrote:
[snip]
>but of a ton of medical research (classic cancer
>experiments aren’t replicating), not to mention
>the rest of the social/behavioral sciences (every
>single one of which — except economics — shows an
>[in]explicable hump just inside the .05 p-value, when
>you survey the literature — graph here: >
>https://twitter.com/jtleek/status/890180014733492225
><https://twitter.com/jtleek/status/890180014733492225>).
I must admit that it is a pretty graph but it really should be called
a "ridgeplot" instead of a "joyplot" ("joy" comes from the band "Joy
Division" which actually refers to a horrendous situation -- google it
if you really want to know). But maybe I'm too skeptic of things but
if you actually get the data (its in a R data formatted file which with
the right wizardry can be imported into SPSS), all 3+ million cases
and just select the sources for psychology and sociology (N=78,6K, that's
what the person who put the data together used in the "field" variable)
you will see the following if you code p-values into 1 unit bands
(i.e., 5.00 to 4.00 -- p-values are expressed as percentages) all the
way down to p<.00 (no, the p-value is not zero but really, really, really
tiny -- maybe a little too tiny?).
Why, all of the p-values are less than .01. How can that be if there
is supposed
to be a "hump" in the .04-.05 interval? Maybe not in this dataset.
What do we see if we include all of the cases (3.6+ million cases)?
The top row (1) contains sources with a p-value between 4-5 and there is only
one such case. There are no results with a p-value between 3-4. There are
3 values in the intervanl 2-3, and 9 values in the interval 1-2. There are over
3.6 million cases with p-values between 0-1 and 20.8K with a p-value
less than .01.
This doesn't seem to match the ridgeplot that Jeff Leek has but we have analyzed
the data in different ways and I admit that maybe I made an error somewhere but
reviewing my work, I don't see any errors.
So, no "hump" in the interval .04-05 and as the p-values decrease, the
frequencies
increase up to the interval .01-.00. For p<= .001, there is a significant drop.
So, what pattern are we supposed to see if these are terrible studies?
If one looks at the raw p-values what is most striking is how many cases with
p< .00000001 there are (are p-value that are all zero up to the 16th
decimal value).
This is really peculiar because either the effect sizes are Godzilla
sized or the
sample sizes are Godzilla sized or both (i.e., Godzilla^2).
Yes, I think there is a replication crisis but I don't think that the data that
Jeff Leek used to make his pretty ridgelplot graph provides any support for
it. But what do I know, right? Anyone can download the data from the Github
website and do the analyses I report either in R or SPSS or whatever statistical
software one uses.
I await the coming results.
By the way, because the tables above are jpegs, this post probably won't
make it into the Mail Archive or the digest but you should get it if you get
post via direct email.
-MIke Palij
New York University
m...@nyu.edu
---
You are currently subscribed to tips as: arch...@mail-archive.com.
To unsubscribe click here:
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=51597
or send a blank email to
leave-51597-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
