On Wed, May 28, 2008 at 03:47:49PM -0700, Xiaohui Chen wrote: > Frank E Harrell Jr ??????: > >Xiaohui Chen wrote: > >>step or stepAIC functions do the job. You can opt to use BIC by > >>changing the mulplication of penalty. > >> > >>I think AIC and BIC are not only limited to compare two pre-defined > >>models, they can be used as model search criteria. You could > >>enumerate the information criteria for all possible models if the > >>size of full model is relatively small. But this is not generally > >>scaled to practical high-dimensional applications. Hence, it is often > >>only possible to find a 'best' model of a local optimum, e.g. > >>measured by AIC/BIC. > > > >Sure you can use them that way, and they may perform better than other > >measures, but the resulting model will be highly biased (regression > >coefficients biased away from zero). AIC and BIC were not designed to > >be used in this fashion originally. Optimizing AIC or BIC will not > >produce well-calibrated models as does penalizing a large model. > > > Sure, I agree with this point. AIC is used to correct the bias from the > estimations which minimize the KL distance of true model, provided the > assumed model family contains the true model. BIC is designed for > approximating the model marginal likelihood. Those are all > post-selection estimating methods. For simutaneous variable selection > and estimation, there are better penalizations like L1 penalty, which is > much better than AIC/BIC in terms of consistency.
Xiaohui, Tibshirani (1996) suggests that the quality of the L1 penalty depends on the structure of the dataset. As I recall, subset selection was preferred for finding a small number of large effects, lasso (L1) for finding a small to moderate number of moderate-sized effects, and ridge (L2) for many small effects. Can you provide any references to more up-to-date simulations that you would recommend? Cheers, Andrew -- Andrew Robinson Department of Mathematics and Statistics Tel: +61-3-8344-6410 University of Melbourne, VIC 3010 Australia Fax: +61-3-8344-4599 http://www.ms.unimelb.edu.au/~andrewpr http://blogs.mbs.edu/fishing-in-the-bay/ ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.