Re: [R] [ADMB Users] an alternative to R for nonlinear stat models
De: Chris Gast [mailto:cmg...@gmail.com] Enviado el: jueves, 17 de junio de 2010 22:32 Para: Rubén Roa CC: r-help@r-project.org; us...@admb-project.org Asunto: Re: [ADMB Users] an alternative to R for nonlinear stat models I spoke with my colleague who did most of the testing, and he has informed me that much of the hessian sensitivity actually came from a separate program (based on Numerical Recipes in C++ code) that did not use optim(), after having stopped using optim() due to speed issues. In my experience with optim, the reltol argument has improved important in this regard. Very small changes in the parameter estimates at the converged solution (influenced by reltol) can lead to different standard error estimates by inverting the hessian, especially for parameter estimates close to zero (as vulnerability coefficients can be in many models with such a feature). It is a limitation of the finite difference method for computing the hessian based on optimal parameter estimates. Chris If the problem originates in estimates being close to zero, a simple transformation (log or exp) might help. However, what I would really like to know is the performance of different methods of the optim function. I guess the reltol parameter and other control parameters would act different depending on the method, whether Nelder-Mead, CG, etc. I am currently experimenting with CG and although it is slow for my model, it has always produced hessian matrices that were invertible and with all diagonals positive after inversion (unlike Nelder-Mead, the default). Rubén Dr. Rubén Roa-Ureta AZTI - Tecnalia / Marine Research Unit Txatxarramendi Ugartea z/g 48395 Sukarrieta (Bizkaia) SPAIN __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] [ADMB Users] an alternative to R for nonlinear stat models
De: users-boun...@admb-project.org [mailto:users-boun...@admb-project.org] En nombre de Chris Gast Enviado el: miércoles, 16 de junio de 2010 21:11 Para: Arni Magnusson CC: r-help@r-project.org; us...@admb-project.org Asunto: Re: [ADMB Users] an alternative to R for nonlinear stat models Hi Arni (and others), My dissertation work involves use (and extension) of models of the same ilk (sometimes exactly the same) as those described by Nancy Gove and John Skalski in their 2002 article. I began with R, and moved to my own home-brewed C/C++ programs for the sake of of speed when fitting models and real and simulated data. In addition, we found that the estimated standard errors (based on the inverse hessian output from optim()) were very sensitive to tolerance criteria--often changing orders of magnitude. Hi, Regarding the last bit, optim() has several methods (Nelder-Mead, simulated annealing, conjugate gradient, etc). It is interesting to me which method produced what result with the standard errors from the inverse Hessian. Can you briefly ellaborate? Thanks Rubén Dr. Rubén Roa-Ureta AZTI - Tecnalia / Marine Research Unit Txatxarramendi Ugartea z/g 48395 Sukarrieta (Bizkaia) SPAIN __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] [ADMB Users] an alternative to R for nonlinear stat models
I spoke with my colleague who did most of the testing, and he has informed me that much of the hessian sensitivity actually came from a separate program (based on Numerical Recipes in C++ code) that did not use optim(), after having stopped using optim() due to speed issues. In my experience with optim, the reltol argument has improved important in this regard. Very small changes in the parameter estimates at the converged solution (influenced by reltol) can lead to different standard error estimates by inverting the hessian, especially for parameter estimates close to zero (as vulnerability coefficients can be in many models with such a feature). It is a limitation of the finite difference method for computing the hessian based on optimal parameter estimates. Chris - Chris Gast cmg...@gmail.com On Wed, Jun 16, 2010 at 11:05 PM, Rubén Roa r...@azti.es wrote: De: users-boun...@admb-project.org [mailto:users-boun...@admb-project.org] En nombre de Chris Gast Enviado el: miércoles, 16 de junio de 2010 21:11 Para: Arni Magnusson CC: r-help@r-project.org; us...@admb-project.org Asunto: Re: [ADMB Users] an alternative to R for nonlinear stat models Hi Arni (and others), My dissertation work involves use (and extension) of models of the same ilk (sometimes exactly the same) as those described by Nancy Gove and John Skalski in their 2002 article. I began with R, and moved to my own home-brewed C/C++ programs for the sake of of speed when fitting models and real and simulated data. In addition, we found that the estimated standard errors (based on the inverse hessian output from optim()) were very sensitive to tolerance criteria--often changing orders of magnitude. Hi, Regarding the last bit, optim() has several methods (Nelder-Mead, simulated annealing, conjugate gradient, etc). It is interesting to me which method produced what result with the standard errors from the inverse Hessian. Can you briefly ellaborate? Thanks Rubén Dr. Rubén Roa-Ureta AZTI - Tecnalia / Marine Research Unit Txatxarramendi Ugartea z/g 48395 Sukarrieta (Bizkaia) SPAIN [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] [ADMB Users] an alternative to R for nonlinear stat models
As an additional note, the parscale argument can be useful to improve stability in convergence results in optim(). Chris - Chris Gast cmg...@gmail.com On Thu, Jun 17, 2010 at 1:32 PM, Chris Gast cmg...@gmail.com wrote: I spoke with my colleague who did most of the testing, and he has informed me that much of the hessian sensitivity actually came from a separate program (based on Numerical Recipes in C++ code) that did not use optim(), after having stopped using optim() due to speed issues. In my experience with optim, the reltol argument has improved important in this regard. Very small changes in the parameter estimates at the converged solution (influenced by reltol) can lead to different standard error estimates by inverting the hessian, especially for parameter estimates close to zero (as vulnerability coefficients can be in many models with such a feature). It is a limitation of the finite difference method for computing the hessian based on optimal parameter estimates. Chris - Chris Gast cmg...@gmail.com On Wed, Jun 16, 2010 at 11:05 PM, Rubén Roa r...@azti.es wrote: De: users-boun...@admb-project.org [mailto:users-boun...@admb-project.org] En nombre de Chris Gast Enviado el: miércoles, 16 de junio de 2010 21:11 Para: Arni Magnusson CC: r-help@r-project.org; us...@admb-project.org Asunto: Re: [ADMB Users] an alternative to R for nonlinear stat models Hi Arni (and others), My dissertation work involves use (and extension) of models of the same ilk (sometimes exactly the same) as those described by Nancy Gove and John Skalski in their 2002 article. I began with R, and moved to my own home-brewed C/C++ programs for the sake of of speed when fitting models and real and simulated data. In addition, we found that the estimated standard errors (based on the inverse hessian output from optim()) were very sensitive to tolerance criteria--often changing orders of magnitude. Hi, Regarding the last bit, optim() has several methods (Nelder-Mead, simulated annealing, conjugate gradient, etc). It is interesting to me which method produced what result with the standard errors from the inverse Hessian. Can you briefly ellaborate? Thanks Rubén Dr. Rubén Roa-Ureta AZTI - Tecnalia / Marine Research Unit Txatxarramendi Ugartea z/g 48395 Sukarrieta (Bizkaia) SPAIN [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] [ADMB Users] an alternative to R for nonlinear stat models
Hi Arni (and others), My dissertation work involves use (and extension) of models of the same ilk (sometimes exactly the same) as those described by Nancy Gove and John Skalski in their 2002 article. I began with R, and moved to my own home-brewed C/C++ programs for the sake of of speed when fitting models and real and simulated data. In addition, we found that the estimated standard errors (based on the inverse hessian output from optim()) were very sensitive to tolerance criteria--often changing orders of magnitude. I have recently made the move to ADMB to make use of the capabilities for mixed effect models. One of my current tasks involves simulating data (in R), fitting models in ADMB via the shell() command in R, and importing .rep files into R for processing results of multiple simulations. In this respect, I too describe my working environment as R with ADMB, and understand the relative benefits of this setup. For these specific types of models, I've found ADMB to be irreplaceable. Similarly, I could not easily summarize simulation results without R; both are vital components to the work I'm doing. With respect to your benchmarking suggestion: I have many implementations of models for real and simulated data in both R and ADMB (past and current iterations of my work on the models described above), and may be able to complete some of this work alongside my own work, although I don't yet claim that either my R or ADMB implementations are completely optimized for speed. Chris - Chris Gast University of Washington Quantitative Ecology and Resource Management cmg...@gmail.com On Wed, Jun 16, 2010 at 11:39 AM, Arni Magnusson arn...@hafro.is wrote: As far as I can tell, Gove et al. (2002) might be a good example for benchmarking the optimization performance of R vs. ADMB. It would be great if expert R users/developers could tweak Beni's model so that the performance comparison is valid. The main purpose is not to see which is faster or more reliable, but to quantify how much performance is gained by moving from R to ADMB. I have just typed and uploaded an excerpt from an old but thorough benchmark, where ADMB outperformed Gauss, Matlab, and S-Plus ( http://admb-project.org/community/benchmarks/optimization). The benchmark was performed in 1997 and an update would be very valuable, as software and hardware have improved since then. R is an interpreted language that can do just about anything. ADMB is a compiled language (thin layer on top of C++) that does optimization and nothing else. I don't think it is a realistic goal for R to match the optimizing performance of ADMB. I use R for most of my work, but crunch numbers with ADMB when computational speed and flexibility become an issue. As an analogy, data frames in R are great, along with functions like aggregate(), apply(), merge(), and xtabs(). But when the tables are too large and too many, it's time to delegate the problem to a relational database. In the case of model X, which is already implemented in R, it may take half an hour or half a month to convert it to ADMB, and the payoff depends on how often the model needs to be run. In the case of model Y, which is not yet implemented, some users may be quicker to implement it in R than in ADMB, so again it would be good to have an idea about the relative performance gain. Model Z may not run at all in R, due to its size and complexity. Although I started this email with R vs. ADMB, my daily working environment is better described as R and ADMB. I'm a regular contributor to both R (4 packages and a couple of functions in the base packages) and ADMB (dev core team). Others have contributed R packages to interface with ADMB ( http://admb-project.org/community/admb-meeting-march-29-31/InterfacingADMBwithR.pdf/at_download/file), but for many the interface is just reading and writing text files. I fully appreciate the comfort and efficiency of the R working environment, and the benefits of performing most tasks inside the same environment. But the R community has no need to be on the defense against ADMB, any more than against relational databases. If you work with computationally intensive models, I encourage you to try out ADMB (admb-project.org, free software) and hopefully end up contributing ideas and/or code. Best regards, Arni P.S. Phew. Future emails mentioning ADMB on r-help can be more brief and to the point, citing this message for details. Windows ADMB-IDE installer http://code.google.com/p/admb-project/downloads/list?q=ide*exe Windows, Linux, and Mac OS standalone http://code.google.com/p/admb-project/downloads/list?q=windows http://code.google.com/p/admb-project/downloads/list?q=linux http://code.google.com/p/admb-project/downloads/list?q=macos ADMB modes for various editors, including Emacs and Vim http://admb-project.org/community/editing-tools
