No argument here. Just me not being clear. A' and AUC are valid measures comparing two systems and much more interpretable than other SDT measures given the parameters as Mike explains but they are not direct measures of SDT parameters as typically explains. Pastore, Crawley, Berens and Skelly (2003) present a good discussion of the issues including the advantages and disadvantages of A'. Specifically, A' is not independent from bias and is actually a poorer estimate when performance is nearer to perfect in terns of hits or false alarms. For the 3 of us who care about this issue, estimates of d' aren't much good in those extremes either. Macmillan and Creelman (1991) suggest adjusting hit rates of 100% to 1-(1/2n) an false alarm rates to 1/2n and I don't have any reason to doubt that I just don't see it used very often.
The use of the sensitivity/specificity reporting doesn't capture both the sensitive and response bias as explained in SDT examples (i.e.g, estimate of the distance between the two distributions (I believe this the reason that this entry is clear to distinguish the sensitivity index, called d' as something different from sensitivity as true positives). The two approaches might be considered two sides of the same coin but they are not the same side of the same coin. Macmillan, N.A., & Creelman, C.D. (1991). Detection Theory: A User’s Guide. NY: Cambridge University Press. Pastore, R.E., Crawley, E.J., Berens, M.S., & Skelley, M.A. (2003). "Nonparametirc" A' and other modern misconceptions about signal detection theory. Psychonomic Builletin and Review, 10(3), 556-569. Doug Peterson, PhD Associate Professor of Psychology The University of South Dakota Vermillion SD 57069 605.677.5295 ________________________________________ From: Mike Palij [m...@nyu.edu] Sent: Friday, January 29, 2016 12:05 PM To: Teaching in the Psychological Sciences (TIPS) Cc: Michael Palij Subject: RE: [tips] signal detection and ROC curves On Fri, 29 Jan 2016 08:49:23 -0800, Douglas Peterson wrote: [snip] >... SDT continues to be applicable in a number of settings, >particularly medical tests, many use a the AUC that Mike mentions >and while this isn't technically SDT (no z transforms) the ROC >method is identical (here is a short and good example > http://www.nature.com/nmeth/journal/v12/n9/fig_tab/nmeth.3482_SF9.html > ) A few points: (1) As I mentioned in an earlier post, SDT is based on Wald's statistical theory which serves as the basis for the Neyman-Pearson framework for statistical testing. The decision matrix originally developed is a 2 x 2 table where the rows represent the response ("yes" or "no", "present" or "absent", etc.) and the columns represent the "true state of nature", that is, stimulus was presented or not presented (this is knows with absolute certainty since they are selected by the researcher; given that the "true state" is known, the question that remains is how well do the responses or decision match the true state -- if the Hit rate is 100% and Correct Rejection ate is 100%, then there the False Alarm rate = 0.00 and the Miss rate = 0.00, in other words, performance is perfect which with weak stimuli in psychophysics rarely/never occurs). (2) I am puzzled by Peterson's statement that AUC is not really SDT given that it's equivalent A' was developed by memory researchers as early as the 1960s and has been shown to be part of SDT. In the http://opl.apa.org experiment on the "Self Reference Effect", the dependent variable is a version of A' that represents the area under "curve" created by the single pair of Hit and False Alarm rates. One reference on this point is the following: Macmillan, N. A., & Creelman, C. D. (1996). Triangles in ROC space: History and theory of "nonparametric" measures of sensitivity and response bias. Psychonomic Bulletin & Review, 3(2), 164-170. Given that the ROC/MOC/AOC is presented in a unit square -- the x-axis represents the probability of a false alarm is limited to the range 0.00 to 1.00 and the y-axis represents the prob of a Hit which also ranges from 0.00 to 1.00 -- chance performance is represented by the diagonal line representing P(Hit)=P(FA). In traditional SDT this implies d-prime is zero. It also implies that the area under the performance curve is 0.50 which can be interpreted as a measure of accuracy; in this case, it represents chance performance (hence the term "chance diagonal"). In most Yes-No recognition memory experiments, only one hit rate and one false alarm rate is obtained. For nonrandom performance, this provides a single point above the chance diagonal, forming a triangle with the chance diagonal as the base. The sum of the area of the triangle and the area under the chance diagonal (i.e., 0.50) becomes a measure of accuracy. As the Hit rate increases and the False Alarm rate decreases, the area in the triangle increases -- in the limit when the False Alarm rate is zero, the triangle fills the upper space and A' or AuC is 1.00 or the entire area of the unit-square. Thus, perfect performance is represented by A' = AuC = 1.00. (3) In making a medical diagnosis or interpreting a medical test, the same reasoning above is employed but the terms differ:: Hit rate becomes True Positive Rate = "Sensitivity" Correct Rejection become True Negative Rate = "Specificity" For more on these ideas and how they are used to determine how good your usual medical test is, see the Wikipedia entry: https://en.wikipedia.org/wiki/Sensitivity_and_specificity This entry eventually leads to d-prime but go to the Wikipedia entry on ROC curves for alternative measures; including AuC: https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve This my third post to TiPS today, so no more till the morrow. -Mike Palij New York University m...@nyu.edu --- You are currently subscribed to tips as: doug.peter...@usd.edu. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=12991.6a54289b29ceb58cb7609cc50e0dc1c8&n=T&l=tips&o=48012 or send a blank email to leave-48012-12991.6a54289b29ceb58cb7609cc50e0dc...@fsulist.frostburg.edu --- You are currently subscribed to tips as: arch...@mail-archive.com. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=48013 or send a blank email to leave-48013-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
