Do you know that there are (pseudo-)randomness involved in CV? Even if you fix the parameters and run multiple times, you're going to get different answers, let alone changing parameters each time. Hoping to narrow the optimal parameters down to that fine a resolution is generally not realistic. Also, there may well be multiple optima in the CV error `surface'. Take your pick. Andy
-----Original Message----- From: Michael [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 01, 2006 3:59 AM To: Liaw, Andy Cc: R-help@stat.math.ethz.ch Subject: Re: [R] does svm have a CV to obtain the best "cost" parameter? Thanks a lot Andy. I read that paper and followed the instructions, but met with a lot peculiarities: 1. In using "tune" function for "svm", the best "cost" value turns out to be multi-peaks, and not with a single global peak. So I don't know which peak to follow in order to refine my search grid and do more detailed search in a smaller/focused range. Please see below. - Detailed performance results: cost error 1 0.0004882813 0.05065909 2 0.0005608879 0.05122727 3 0.0006442910 0.04895130 4 0.0007400960 0.04725000 5 0.0008501470 0.04497078 6 0.0009765625 0.04497078 7 0.0011217757 0.04497078 8 0.0012885819 0.04440260 9 0.0014801920 0.04155844 10 0.0017002941 0.03985065 11 0.0019531250 0.04099675 12 0.0022435515 0.04327273 13 0.0025771639 0.04099675 14 0.0029603839 0.03929221 15 0.0034005881 0.03986039 16 0.0039062500 0.04157143 17 0.0044871029 0.04099675 18 0.0051543278 0.04042857 19 0.0059207678 0.03871753 20 0.0068011763 0.03871429 21 0.0078125000 0.03985065 22 0.0089742059 0.04042532 23 0.0103086556 0.04042532 24 0.0118415357 0.04099675 25 0.0136023526 0.04042532 26 0.0156250000 0.04440260 2. I first tried 2^(-15:15), and found the best "cost" to be around 2^(-8), then I reduce the range, run "tune" on cost values 2^(-11:-6), and it returned a best "cost" value to be 2^(-9), which is different from 2^(-8), then I run it on seq(-11, -6, by = 0.2), the best "cost" value was found to be 2^(-7.2), and with the above multi-peaks... each time the best "cost" is at a different value. And with the above multi-peaks, a lot of local optima, I don't know what range should I focus on for the next step... The code I've used is as below: obj <- tune(svm, x, y, ranges = list(cost = 2^seq(-11, -6, by=0.2)), tunecontrol = tune.control(sampling = "cross") , kernel='linear' ) ------------------------ What can I do now? Thanks a lot! On 2/28/06, Liaw, Andy < [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote: You might find http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf <http://www.csie.ntu.edu.tw/%7Ecjlin/papers/guide/guide.pdf> helpful. Parameter tuning is essential for avoiding overfitting. Andy -----Original Message----- From: Michael [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> ] Sent: Tuesday, February 28, 2006 3:30 PM To: Liaw, Andy Cc: R-help@stat.math.ethz.ch <mailto:R-help@stat.math.ethz.ch> Subject: Re: [R] does svm have a CV to obtain the best "cost" parameter? Hi Andy, Thanks a lot for your answer! So what do I do if the model overfits? Thanks a lot! On 2/28/06, Liaw, Andy < <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]> wrote: From: Michael > > Hi all, > > I am using the "svm" command in the e1071 package. > > Does it have an automatic way of setting the "cost" parameter? See ?best.svm in that package. > I changed a few values for the "cost" parameter but I hope there is a > systematic way of obtaining the best "cost" value. > > I noticed that there is a "cross" (Cross validation) > parameter in the "svm" > function. > > But I did not see how it can be used to optimize the "cost" parameter. > > By the way, what does a 0 training error and a high testing > error mean? > Varying "cross=5", or "cross=10", etc. does not change the > training error > and testing error at all. How to improve? Overfitting, which varying different validation method will not solve. Andy > Thanks a lot! > > M. > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help@stat.math.ethz.ch <mailto:R-help@stat.math.ethz.ch> mailing list > https://stat.ethz.ch/mailman/listinfo/r-help <https://stat.ethz.ch/mailman/listinfo/r-help> > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html <http://www.R-project.org/posting-guide.html> > > ---------------------------------------------------------------------------- -- Notice: This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New Jersey, USA 08889), and/or its affiliates (which may be known outside the United States as Merck Frosst, Merck Sharp & Dohme or MSD and in Japan, as Banyu) that may be confidential, proprietary copyrighted and/or legally privileged. It is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please notify us immediately by reply e-mail and then delete it from your system. ---------------------------------------------------------------------------- -- ---------------------------------------------------------------------------- -- Notice: This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station, New Jersey, USA 08889), and/or its affiliates (which may be known outside the United States as Merck Frosst, Merck Sharp & Dohme or MSD and in Japan, as Banyu) that may be confidential, proprietary copyrighted and/or legally privileged. It is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please notify us immediately by reply e-mail and then delete it from your system. ---------------------------------------------------------------------------- -- ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ [[alternative HTML version deleted]] ______________________________________________ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
