Dukka k.c. wrote:
Dear all
I am trying to compare the performances of several methods using the AUC0.1
and
not the whole AUC. (meaning I wanted to compare to AUC's whose x axis only
goes to
0.1 not 1)
I came to know about the Mcneil Hanley test from Bernardo Rangel Tura
and I referred to the original paper for the calculation of "r" which is an
argument of the function
cROC. I can only find the value of "r" for the whole AUC's .
seROC<-function(AUC,na,nn){
a<-AUC
q1<-a/(2-a)
q2<-(2*a^2)/(1+a)
se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
se
}
cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
se1<-seROC(AUC1,na1,nn1)
se2<-seROC(AUC2,na2,nn2)
sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
zad<-(AUC1-AUC2)/sed
p<-dnorm(zad)
a<-list(zad,p)
a
Could somebody kindly suggest me how to calculate the value of "r" or
some ways to calculate the statistical significance measure for the
differences of auc for a part of
the curve like AUC0.1.
Thank You
The ROC area is not a sensitive enough measure for comparing two
competing predictors. Its power is too low. See for example the
following papers. Note that Pencina et al's approach is now in the
Hmisc package (function improveProb; documentation to be coming soon).
Likelihood ratio tests are even more powerful.
@Article{pen08eva,
author = {Pencina, Michael J. and {D'Agostino Sr}, Ralph B. and
{D'Agostino Jr}, Ralph B. and Vasan, Ramachandran S.},
title = {Evaluating the added predictive ability of a new marker:
{From} area under the {ROC} curve to reclassification and beyond},
journal = Stat in Med,
year = 2008,
volume = 27,
pages = {157-172},
annote = {discrimination;model performance;AUC;C-index;risk
prediction;biomarker;small differences in ROC area can still be very
meaningful;example of insignificant test for difference in ROC areas
with very significant results from new method;Yates' discrimination
slope;reclassification table;limiting version of this based on whether
and amount by which probabilities rise for events and lower for
non-events when compare new model to old;comparing two models}
}
@Article{coo07use,
author = {Cook, Nancy R.},
title = {Use and misues of the receiver operating characteristic
curve in risk prediction},
journal = {Circulation},
year = 2007,
volume = 115,
pages = {928-935},
annote = {reclassification table;problems with c index;problems
with ROC area;example of large change in predicted risk in
cardiovascular disease with tiny change in ROC area;possible limits to c
index when calibration is perfect;importance of calibration accuracy and
changes in predicted risk when new variables are added}
}
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
