Re: [Rd] Competing with one's own work
Ben raised an interesting point: A better question might be how packages get added to the *recommended* package list (rather than how code gets added to base R). As maintainer of survival one of the surprising things is the number of packages that depend on mine. This has caused me to change my opinion over the last few years about how much expansion of the package should occur. I used to feel bad that the package doesn't keep up with everything, now I tend to vote for putting new ideas elsewhere. Consider competing risks for instance. The cleanest way to code this is to extend the Surv(time, status) construct to allow more than a 0/1 status variable. I thought about this seriously, and realized that such a change would have ripple effects on some of the other routines -- an extra if statement here and there. This is doable, but what about the effect on the 153 dependent packages! The stability of survival is now more important than its feature set. My first extention to random effects (a very simplistic one) was incorporated into the main survival package. I've pushed the later developments into thier own package. It was definitely the right choice. Terry T __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Competing with one's own work (Prof. John C Nash)
John, thanks for starting (or restarting) this thread. I'd like to add to the discussion with another concrete example, about as simple as it gets, which may help focus at least part of this discussion. I have worked with Taylor Arnold to implement a method developed in Conover (1972) for Kolmogorov-Smirnov goodness-of-fit tests for discrete distributions (one-sample only). We needed this for an applied problem. It seemed to be a natural extension to stats::ks.test(), so we modified that code (commenting every addition very carefully) and modifying ks.test.Rd in parallel (and with commenting). For convenience, we put this in a package ks.test that is on R-Forge but not submitted to CRAN. We've written a short paper about this (and also implemented similar Cramer-von Mises tests in package cvm.test). 1. I think the cvm.test function/package is suitable for CRAN (rather, I can't make a compelling argument it should be added to the base distribution). It doesn't directly extend anything in the base R distribution at the moment (at least, to my knowledge). 2. I think it would be ideal for stats::ks.test() to be updated using the new ks.test.R and ks.test.Rd. I'll spare you the longer argument, but there are simple examples of a bug (quotes intended, because it surrounds non-intended functionality with discrete distributions) in stats::ks.test(). 3. Finally, I note the presence of a FIXME in stats::ks.test() that looks rather straightforward. I'd be happy to do this FIXME as part of this contribution (though perhaps I should read the cited paper and conduct some simple simulations). A simple, that would be great, Jay or don't bother would suffice -- it may be that someone else is working on it. Thoughts welcome, either on these particular issues, or on the manner in which they relate to John's thread. Cheers, Jay -- John W. Emerson (Jay) Associate Professor of Statistics Department of Statistics Yale University http://www.stat.yale.edu/~jay __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Competing with one's own work
No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should be deprecated and users encouraged to try other routines. It is also getting more and more important that novice users be provided with sensible guidance and robust default settings and choices. In many areas, users are faced with more choice than is efficient for the majority of problems. My question is: How should such changes be suggested / assisted? It seems to me that this is beyond a simple feature request. Some discussion on pros and cons would be appropriate, and those like myself who are familiar with particular tools can and should offer help. Alternatively, is there a document available in the style Writing R Extensions that has a title like How the R Base Packages are Updated? A brief search was negative. I'm happy to compete with my own prior work to provide improvements. It would be nice to see some of those improvements become the benchmark for further progress. Best, John Nash __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Competing with one's own work
On 03/12/2010 10:57 AM, Prof. John C Nash wrote: No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should be deprecated and users encouraged to try other routines. It is also getting more and more important that novice users be provided with sensible guidance and robust default settings and choices. In many areas, users are faced with more choice than is efficient for the majority of problems. My question is: How should such changes be suggested / assisted? It seems to me that this is beyond a simple feature request. Some discussion on pros and cons would be appropriate, and those like myself who are familiar with particular tools can and should offer help. Alternatively, is there a document available in the style Writing R Extensions that has a title like How the R Base Packages are Updated? A brief search was negative. I'm happy to compete with my own prior work to provide improvements. It would be nice to see some of those improvements become the benchmark for further progress. There are answers at many different levels to your questions. The simplest is that base packages are part of R, so they get updated when a member of R Core updates them, and the updates get released when a new version of R is released. So if you want a change, you need to convince a member of the core to make it. Pointing out a bug is the easiest way to do this: bugs usually get fixed quickly, if they are clearly demonstrated. If you want a bigger change, you need to make a convincing argument in favour of it. If you pick a topic that is of particular interest to one core member, and you can convince him to make the change, then it will happen. If pick some obscure topic that's not of interest to anyone, you'll need a very strong argument to make it interesting. Part of any of these arguments is explaining why the change needs to be made to the base, why it can't just be published in a contributed package. (That's why bug fixes are easy, and big additions to the base packages are not.) Duncan Murdoch __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Competing with one's own work
Dear Duncan, What constitutes a convincing argument for making significant changes? Taking the example of optimization algorithms (say, for smooth objective functions), how does one make a convincing argument that a particular class of algorithms are better than another class? This can be a difficult task, but quite doable with good benchmarking practices. Supposing for the moment that such a convincing argument has been made, is that sufficient to get the R-core to act upon it? Are there compelling factors other than just algorithm A is better than algorithm B? I'd think that the argument is relatively easy if the need for the change is driven by consumer demand. But, even here I am not sure how to make an argument to the R-core to consider the big changes. For example, there is a reasonable demand for constrained (smooth) optimization algorithms in R (based on R-help queries). Currently, there are only 3 packages that can handle this. However, in the base distribution only `constrOptim' function is provided, which cannot handle anything more than linear, inequality constraints. I think that the base distribution needs to have a package for constrained optimization that can handle linear/nonlinear and equality/inequality constraints. John, thanks for raising an important issue. Thanks Best, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch Sent: Friday, December 03, 2010 11:13 AM To: nas...@uottawa.ca Cc: r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On 03/12/2010 10:57 AM, Prof. John C Nash wrote: No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should be deprecated and users encouraged to try other routines. It is also getting more and more important that novice users be provided with sensible guidance and robust default settings and choices. In many areas, users are faced with more choice than is efficient for the majority of problems. My question is: How should such changes be suggested / assisted? It seems to me that this is beyond a simple feature request. Some discussion on pros and cons would be appropriate, and those like myself who are familiar with particular tools can and should offer help. Alternatively, is there a document available in the style Writing R Extensions that has a title like How the R Base Packages are Updated? A brief search was negative. I'm happy to compete with my own prior work to provide improvements. It would be nice to see some of those improvements become the benchmark for further progress. There are answers at many different levels to your questions. The simplest is that base packages are part of R, so they get updated when a member of R Core updates them, and the updates get released when a new version of R is released. So if you want a change, you need to convince a member of the core to make it. Pointing out a bug is the easiest way to do this: bugs usually get fixed quickly, if they are clearly demonstrated. If you want a bigger change, you need to make a convincing argument in favour of it. If you pick a topic that is of particular interest to one core member, and you can convince him to make the change, then it will happen. If pick some obscure topic that's not of interest to anyone, you'll need a very strong argument to make it interesting. Part of any of these arguments is explaining why the change needs to be made to the base, why it can't just be published in a contributed package. (That's why bug fixes are easy, and big additions to the base packages are not.) Duncan Murdoch __ R-devel@r-project.org mailing
Re: [Rd] Competing with one's own work
On Fri, Dec 3, 2010 at 11:01 AM, Ravi Varadhan rvarad...@jhmi.edu wrote: Dear Duncan, What constitutes a convincing argument for making significant changes? Taking the example of optimization algorithms (say, for smooth objective functions), how does one make a convincing argument that a particular class of algorithms are better than another class? This can be a difficult task, but quite doable with good benchmarking practices. Supposing for the moment that such a convincing argument has been made, is that sufficient to get the R-core to act upon it? Are there compelling factors other than just algorithm A is better than algorithm B? I'd think that the argument is relatively easy if the need for the change is driven by consumer demand. But, even here I am not sure how to make an argument to the R-core to consider the big changes. For example, there is a reasonable demand for constrained (smooth) optimization algorithms in R (based on R-help queries). Currently, there are only 3 packages that can handle this. However, in the base distribution only `constrOptim' function is provided, which cannot handle anything more than linear, inequality constraints. I think that the base distribution needs to have a package for constrained optimization that can handle linear/nonlinear and equality/inequality constraints. constrOptim is in the stats package, not the base package. Functions that are already in the required packages are maintained by R core. If you know of bugs in such functions you should report them. Because there is a heavy burden in maintaining the large corpus of software in R and its required packages, additions are viewed skeptically, Adopting new capabilities and new code in a required package like stats means that some member of R core has to be willing to maintain it. If the capabilities can be incorporated in a contributed package then that is the preferred method of extending R. The burden of maintaining the code, fixing bugs or other infelicities, etc. is on the package maintainer. I don't see anything in what you are proposing that could not be incorporated in a contributed package. John, thanks for raising an important issue. Thanks Best, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch Sent: Friday, December 03, 2010 11:13 AM To: nas...@uottawa.ca Cc: r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On 03/12/2010 10:57 AM, Prof. John C Nash wrote: No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should be deprecated and users encouraged to try other routines. It is also getting more and more important that novice users be provided with sensible guidance and robust default settings and choices. In many areas, users are faced with more choice than is efficient for the majority of problems. My question is: How should such changes be suggested / assisted? It seems to me that this is beyond a simple feature request. Some discussion on pros and cons would be appropriate, and those like myself who are familiar with particular tools can and should offer help. Alternatively, is there a document available in the style Writing R Extensions that has a title like How the R Base Packages are Updated? A brief search was negative. I'm happy to compete with my own prior work to provide improvements. It would be nice to see some of those improvements become the benchmark for further progress. There are answers at many different levels to your questions. The simplest is that base packages are part of R, so they get updated when a member of R Core updates them
Re: [Rd] Competing with one's own work
Dear Doug, Thank you for the response. constrOptim is in the stats package, not the base package. Yes, I know, and I meant to say base *distribution* rather than base package. The burden of maintaining the code, fixing bugs or other infelicities, etc. is on the package maintainer. Of course. I don't see anything in what you are proposing that could not be incorporated in a contributed package. I agree, and it has already been done. What I am really asking is this: what is the rationale behind having a package incorporated into the base distribution? Best, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: dmba...@gmail.com [mailto:dmba...@gmail.com] On Behalf Of Douglas Bates Sent: Friday, December 03, 2010 1:28 PM To: Ravi Varadhan Cc: Duncan Murdoch; nas...@uottawa.ca; r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On Fri, Dec 3, 2010 at 11:01 AM, Ravi Varadhan rvarad...@jhmi.edu wrote: Dear Duncan, What constitutes a convincing argument for making significant changes? Taking the example of optimization algorithms (say, for smooth objective functions), how does one make a convincing argument that a particular class of algorithms are better than another class? This can be a difficult task, but quite doable with good benchmarking practices. Supposing for the moment that such a convincing argument has been made, is that sufficient to get the R-core to act upon it? Are there compelling factors other than just algorithm A is better than algorithm B? I'd think that the argument is relatively easy if the need for the change is driven by consumer demand. But, even here I am not sure how to make an argument to the R-core to consider the big changes. For example, there is a reasonable demand for constrained (smooth) optimization algorithms in R (based on R-help queries). Currently, there are only 3 packages that can handle this. However, in the base distribution only `constrOptim' function is provided, which cannot handle anything more than linear, inequality constraints. I think that the base distribution needs to have a package for constrained optimization that can handle linear/nonlinear and equality/inequality constraints. constrOptim is in the stats package, not the base package. Functions that are already in the required packages are maintained by R core. If you know of bugs in such functions you should report them. Because there is a heavy burden in maintaining the large corpus of software in R and its required packages, additions are viewed skeptically, Adopting new capabilities and new code in a required package like stats means that some member of R core has to be willing to maintain it. If the capabilities can be incorporated in a contributed package then that is the preferred method of extending R. The burden of maintaining the code, fixing bugs or other infelicities, etc. is on the package maintainer. I don't see anything in what you are proposing that could not be incorporated in a contributed package. John, thanks for raising an important issue. Thanks Best, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch Sent: Friday, December 03, 2010 11:13 AM To: nas...@uottawa.ca Cc: r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On 03/12/2010 10:57 AM, Prof. John C Nash wrote: No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should
Re: [Rd] Competing with one's own work
On 03/12/2010 12:01 PM, Ravi Varadhan wrote: Dear Duncan, What constitutes a convincing argument for making significant changes? I don't think there's any answer to that other than an argument that convinces someone to make the changes. What would convince you to work on a problem? Your answer is very different from mine, and mine is different from that of anyone else in the core group. Taking the example of optimization algorithms (say, for smooth objective functions), how does one make a convincing argument that a particular class of algorithms are better than another class? This can be a difficult task, but quite doable with good benchmarking practices. I don't see how that's relevant. That's an argument to make to users, not to the core group. A user wants to use the best optimizer for his/her own problem. The core group wants functions in base R that we will maintain. Supposing for the moment that such a convincing argument has been made, is that sufficient to get the R-core to act upon it? By definition, yes. Are there compelling factors other than just algorithm A is better than algorithm B? Yes. The decision about whether it belongs in a package or in base R is about who should maintain the code. If I think it is fantastic code, but you will do a better job of maintaining it than I will, then there's no way I'd put it in base R. I'd think that the argument is relatively easy if the need for the change is driven by consumer demand. But, even here I am not sure how to make an argument to the R-core to consider the big changes. For example, there is a reasonable demand for constrained (smooth) optimization algorithms in R (based on R-help queries). Currently, there are only 3 packages that can handle this. However, in the base distribution only `constrOptim' function is provided, which cannot handle anything more than linear, inequality constraints. I think that the base distribution needs to have a package for constrained optimization that can handle linear/nonlinear and equality/inequality constraints. As Doug said, I don't see anything in what you are proposing that could not be incorporated in a contributed package. I think I answered your followup question above: the rationale for including it in base R is because someone in the core team is in a better position to maintain the code than an outside package maintainer would be. Duncan Murdoch John, thanks for raising an important issue. Thanks Best, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch Sent: Friday, December 03, 2010 11:13 AM To: nas...@uottawa.ca Cc: r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On 03/12/2010 10:57 AM, Prof. John C Nash wrote: No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should be deprecated and users encouraged to try other routines. It is also getting more and more important that novice users be provided with sensible guidance and robust default settings and choices. In many areas, users are faced with more choice than is efficient for the majority of problems. My question is: How should such changes be suggested / assisted? It seems to me that this is beyond a simple feature request. Some discussion on pros and cons would be appropriate, and those like myself who are familiar with particular tools can and should offer help. Alternatively, is there a document available in the style Writing R Extensions that has a title like How the R Base Packages are Updated? A brief search was negative. I'm happy to compete with my own prior work to provide improvements
Re: [Rd] Competing with one's own work
The decision about whether it belongs in a package or in base R is about who should maintain the code. Ok. I understand it now. Thanks, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: Duncan Murdoch [mailto:murdoch.dun...@gmail.com] Sent: Friday, December 03, 2010 2:19 PM To: Ravi Varadhan Cc: nas...@uottawa.ca; r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On 03/12/2010 12:01 PM, Ravi Varadhan wrote: Dear Duncan, What constitutes a convincing argument for making significant changes? I don't think there's any answer to that other than an argument that convinces someone to make the changes. What would convince you to work on a problem? Your answer is very different from mine, and mine is different from that of anyone else in the core group. Taking the example of optimization algorithms (say, for smooth objective functions), how does one make a convincing argument that a particular class of algorithms are better than another class? This can be a difficult task, but quite doable with good benchmarking practices. I don't see how that's relevant. That's an argument to make to users, not to the core group. A user wants to use the best optimizer for his/her own problem. The core group wants functions in base R that we will maintain. Supposing for the moment that such a convincing argument has been made, is that sufficient to get the R-core to act upon it? By definition, yes. Are there compelling factors other than just algorithm A is better than algorithm B? Yes. The decision about whether it belongs in a package or in base R is about who should maintain the code. If I think it is fantastic code, but you will do a better job of maintaining it than I will, then there's no way I'd put it in base R. I'd think that the argument is relatively easy if the need for the change is driven by consumer demand. But, even here I am not sure how to make an argument to the R-core to consider the big changes. For example, there is a reasonable demand for constrained (smooth) optimization algorithms in R (based on R-help queries). Currently, there are only 3 packages that can handle this. However, in the base distribution only `constrOptim' function is provided, which cannot handle anything more than linear, inequality constraints. I think that the base distribution needs to have a package for constrained optimization that can handle linear/nonlinear and equality/inequality constraints. As Doug said, I don't see anything in what you are proposing that could not be incorporated in a contributed package. I think I answered your followup question above: the rationale for including it in base R is because someone in the core team is in a better position to maintain the code than an outside package maintainer would be. Duncan Murdoch John, thanks for raising an important issue. Thanks Best, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch Sent: Friday, December 03, 2010 11:13 AM To: nas...@uottawa.ca Cc: r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On 03/12/2010 10:57 AM, Prof. John C Nash wrote: No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should be deprecated and users encouraged to try other routines. It is also getting more and more important that novice users be provided with sensible
Re: [Rd] Competing with one's own work
At one time I lobbied for putting something in base or a required package, and it was suggested that the idea at the time was to remove things rather than add them. Generally, I agree that is a good idea, so I did not lobby more. When this question comes up it is always asked, and answered, in terms of putting things in. However, is there a process for moving things out to normal packages rather than keeping them in required packages or base? Paul -Original Message- From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r- project.org] On Behalf Of Douglas Bates Sent: December 3, 2010 1:28 PM To: Ravi Varadhan Cc: r-devel@r-project.org; nas...@uottawa.ca Subject: Re: [Rd] Competing with one's own work On Fri, Dec 3, 2010 at 11:01 AM, Ravi Varadhan rvarad...@jhmi.edu wrote: Dear Duncan, What constitutes a convincing argument for making significant changes? Taking the example of optimization algorithms (say, for smooth objective functions), how does one make a convincing argument that a particular class of algorithms are better than another class? This can be a difficult task, but quite doable with good benchmarking practices. Supposing for the moment that such a convincing argument has been made, is that sufficient to get the R-core to act upon it? Are there compelling factors other than just algorithm A is better than algorithm B? I'd think that the argument is relatively easy if the need for the change is driven by consumer demand. But, even here I am not sure how to make an argument to the R-core to consider the big changes. For example, there is a reasonable demand for constrained (smooth) optimization algorithms in R (based on R-help queries). Currently, there are only 3 packages that can handle this. However, in the base distribution only `constrOptim' function is provided, which cannot handle anything more than linear, inequality constraints. I think that the base distribution needs to have a package for constrained optimization that can handle linear/nonlinear and equality/inequality constraints. constrOptim is in the stats package, not the base package. Functions that are already in the required packages are maintained by R core. If you know of bugs in such functions you should report them. Because there is a heavy burden in maintaining the large corpus of software in R and its required packages, additions are viewed skeptically, Adopting new capabilities and new code in a required package like stats means that some member of R core has to be willing to maintain it. If the capabilities can be incorporated in a contributed package then that is the preferred method of extending R. The burden of maintaining the code, fixing bugs or other infelicities, etc. is on the package maintainer. I don't see anything in what you are proposing that could not be incorporated in a contributed package. John, thanks for raising an important issue. Thanks Best, Ravi. --- Ravi Varadhan, Ph.D. Assistant Professor, Division of Geriatric Medicine and Gerontology School of Medicine Johns Hopkins University Ph. (410) 502-2619 email: rvarad...@jhmi.edu -Original Message- From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r- project.org] On Behalf Of Duncan Murdoch Sent: Friday, December 03, 2010 11:13 AM To: nas...@uottawa.ca Cc: r-devel@r-project.org Subject: Re: [Rd] Competing with one's own work On 03/12/2010 10:57 AM, Prof. John C Nash wrote: No, this is not about Rcpp, but a comment in that overly long discussion raised a question that has been in my mind for a while. This is that one may have work that is used in R in the base functionality and there are improvements that should be incorporated. For me, this concerns the BFGS, Nelder-Mead and CG options of optim(), which are based on the 1990 edition (Pascal codes) of my 1979 book Compact numerical methods..., which were themselves derived from other people's work. By the time Brian Ripley took that work (with permission, even though not strictly required. Thanks!) there were already some improvements to these same algorithms (mainly bounds and masks) in the BASIC codes of the 1987 book by Mary Walker-Smith and I. However, BASIC to R is not something I'd wish on anyone. Now there are some R packages, including some I've been working on, that do offer improvements on the optim() offerings. I would not say mine are yet fully ready for incorporation into the base, but they are pretty close. Equally I think some of the tools in the base should be deprecated and users encouraged to try other routines. It is also getting more and more important that novice users be provided with sensible guidance and robust default settings and choices. In many areas, users are faced with more choice than is efficient for the majority of problems. My question is: How should such changes be suggested
Re: [Rd] Competing with one's own work
Ravi Varadhan rvaradhan at jhmi.edu writes: The decision about whether it belongs in a package or in base R is about who should maintain the code. Ok. I understand it now. Thanks, Ravi. A point that may not have been made (sorry if it was and I missed it): A better question might be how packages get added to the *recommended* package list (rather than how code gets added to base R). Of the 16 recommended packages, 2 are maintained by R-core itself, 12 by various R-core members acting as individuals (I assume), and 2 by non-R-core people. It seems that if a contributed package sticks around long enough and proves itself sufficiently useful and of sufficiently high quality (and well enough maintained), that it could then be suggested as a recommended package. i1 - installed.packages() i2 - i1[!is.na(i1[,Priority]),] ff - function(x) table(sapply(x[,Package],maintainer)) ff(i2[i2[,Priority]==base,]) R Core Team r-c...@r-project.org 12 ff(i2[i2[,Priority]==recommended,]) Brian Ripley rip...@stats.ox.ac.uk 7 Deepayan Sarkar deepayan.sar...@r-project.org 2 Doug and Martin matrix-auth...@r-project.org 1 Luke Tierney l...@stat.uiowa.edu 1 Martin Maechler maech...@stat.math.ethz.ch 1 R-core r-c...@r-project.org 1 R-core r-c...@r-project.org 1 Simon Wood simon.w...@r-project.org 1 Terry Therneau therneau.te...@mayo.edu 1 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
