On Fri, 18 Jul 2003 15:21:55 +0200 "Koen Vermeer" <[EMAIL PROTECTED]> wrote:
> Hi, > > I am having conceptual problems with cross-validating an ROC. The thing is > that for me the only reason to draw an ROC is to show the individual > fpr/tpr pairs, so one can choose the optimal setting for a specific > application (depending on prevalence, cost of FP/FN, etc). So, in fact, > the ROC just shows various algorithms, and you choose one that suits you > best. The thing with validation is that it is supposed to be done on the > final algorithm, not on some intermediate result. > > More detailed: > Consider algorithm A. It tests a number of algorithms (1..N) and chooses > the best one (say number i). Even if algorithm A uses cross-validation to > train and test all N algorithms, we cannot say that the error rate of > algorithm A is the same as the estimated error rate of algorithm i. So, we > cross-validate algorithm A: We use a data set to train it (and thus to > select i) and an independent set to test its performance. > > Now, if we compare this to the ROC, the ROC is like the outcome of all N > algorithms. Based on the application, one would choose the best algorithm. > Cross-validation is therefore not possible before this selection has been > made. > > On the other hand, one could ofcourse 'cross-validate' the ROC. For > example, the ROCs of the several folds could be averaged in some way, or > the individual tpr/fpr pairs could be cross-validated. > > I would appreciate any comments on this! > > Regards, > Koen Vermeer > The choice of a single cutpoint presents a host of statistical and subject matter deficiencies, but if you really need one (which implies that your internal utility function is the same as the consumers') you are right that you need to make the cross-validation take the cutpoint search into account. The bootstrap is probably the best approach. Have an algorithm for choosing the "best" cutpoint and repeat that algorithm 200 times by replacing the original dataset with samples with replacement from the original (using the same total number of observations). You can get a confidence interval for the cutpoint this way. --- Frank E Harrell Jr Prof. of Biostatistics & Statistics Div. of Biostatistics & Epidem. Dept. of Health Evaluation Sciences U. Virginia School of Medicine http://hesweb1.med.virginia.edu/biostat . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
