Re: [Rcpp-devel] Distribution of Rcpp codebase
On Mon, Apr 7, 2014 at 4:12 AM, Romain François rom...@r-enthusiasts.comwrote: [...] However, in terms of wins: - package developers would know for sure which version of the codebase is used with their package. Once they have done testing, they don't have to be hostage of api breakage and things like please recompile your package, etc ... - developers of Rcpp* are less trapped by the compatibility issues, hands are set free to innovate. The only small downsides I see here is that (1) users potentially have to do more work to include Rcpp* in their packages (although you can just write an R function to include/update their Rcpp* versions); and that (2) source packages will be somewhat bigger. Btw. you are essentially simulating versioned package dependencies this way. :) The next thing to consider is that Rcpp is not just Rcpp, there are really nice extensions like RcppArmadillo, etc ... perhaps we could setup some tools (e.g. RcppJam) to combine several header only libraries into the end package, instead of what we do now, which is have some headers in Rcpp, some in RcppArmadillo, some in RcppGSL, ... with every risk of one being outdated or out of sync with the other. Exactly. IMHO this could work well and take the pressure of both Rcpp* developers and users. Gabor ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Distribution of Rcpp codebase
Le 7 avr. 2014 à 15:13, Gábor Csárdi csardi.ga...@gmail.com a écrit : On Mon, Apr 7, 2014 at 4:12 AM, Romain François rom...@r-enthusiasts.com wrote: [...] However, in terms of wins: - package developers would know for sure which version of the codebase is used with their package. Once they have done testing, they don’t have to be hostage of api breakage and things like « please recompile your package, etc … » - developers of Rcpp* are less trapped by the compatibility issues, hands are set free to innovate. The only small downsides I see here is that (1) users potentially have to do more work to include Rcpp* in their packages (although you can just write an R function to include/update their Rcpp* versions); and that (2) source packages will be somewhat bigger. The difference is that if they don’t want a new version, they can stick with the version they have in their package. So when they want newer Rcpp, they can have it, but they are not forced to do anything when the Rcpp* team wants to update the codebase. About 1). We just need to find a good way to propagate newer Rcpp* into a repo. Maybe through git subtree, git sub modules or something. We can come up with some tools. About 2). So What ;) Btw. you are essentially simulating versioned package dependencies this way. :) Yep. The next thing to consider is that Rcpp is not just Rcpp, there are really nice extensions like RcppArmadillo, etc ... perhaps we could setup some tools (e.g. RcppJam) to combine several header only libraries into the end package, instead of what we do now, which is have some headers in Rcpp, some in RcppArmadillo, some in RcppGSL, … with every risk of one being outdated or out of sync with the other. Exactly. IMHO this could work well and take the pressure of both Rcpp* developers and users. Gabor ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
[Rcpp-devel] No no no -- Distribution of Rcpp codebase
Romain, Gabor, Would you considerr taking this discussion elsewhere? Or at least make it very clear that you are talking about the Rcpp11, which as AFAICT has not left Github. Rcpp was created as a CRAN package. It remains a CRAN package. It fits squarely into the dependency mechanism of CRAN. It has proven useful, and stable. As a consequence it is as of today used by almost 200 CRAN packages and almost 20 BioC package. Please do not confuse our users by implying that any of this is about to change. You are more than welcome to experiment with new approaches, new implementation, new distributions, ... but *please* make sure you continue to do so under a new name too. This list is about Rcpp. The CRAN package. Thanks, Dirk -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] No no no -- Distribution of Rcpp codebase
As I understand, it would still be an R package, on CRAN, at least this is certainly a possibility. The idea was about putting the headers into *every* package that uses Rcpp, and then these packages don't have to specify LinkingTo: Rcpp*, and they would be completely independent of subsequent Rcpp* releases. But maybe I misunderstood. Gabor On Mon, Apr 7, 2014 at 9:34 AM, Dirk Eddelbuettel e...@debian.org wrote: Romain, Gabor, Would you considerr taking this discussion elsewhere? Or at least make it very clear that you are talking about the Rcpp11, which as AFAICT has not left Github. Rcpp was created as a CRAN package. It remains a CRAN package. It fits squarely into the dependency mechanism of CRAN. It has proven useful, and stable. As a consequence it is as of today used by almost 200 CRAN packages and almost 20 BioC package. Please do not confuse our users by implying that any of this is about to change. You are more than welcome to experiment with new approaches, new implementation, new distributions, ... but *please* make sure you continue to do so under a new name too. This list is about Rcpp. The CRAN package. Thanks, Dirk -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] No no no -- Distribution of Rcpp codebase
Le 7 avr. 2014 à 15:34, Dirk Eddelbuettel e...@debian.org a écrit : Romain, Gabor, Would you considerr taking this discussion elsewhere? This is a broad discussion that might be relevant to anyone doing R and C++ work. So no. Or at least make it very clear that you are talking about the Rcpp11, which as AFAICT has not left Github. Rcpp11 needs R 3.1.0, so of course it is not on CRAN yet. But this discussion is not about the Rcpp11 package, it is not about the Rcpp package. It is about exchanging ideas about what would be beneficial to the community of people using R and C++. The way the Rcpp codebase is currently distributed might have been the best model for years, but I’m not convinced it is the best solution nowadays, so I wanted to discuss, and this list is the best venue, whether you like it or not. As for Rcpp11, I am indeed going to release it as an R package, but I will also definitely consider alternatives such as snapshotting the distribution as part of the client package. I think there are advantages to doing that. But don’t worry, I will definitely not do anything to change the Rcpp package. You are doing a fine job of keeping it alive. Rcpp was created as a CRAN package. It remains a CRAN package. It fits squarely into the dependency mechanism of CRAN. It has proven useful, and stable. As a consequence it is as of today used by almost 200 CRAN packages and almost 20 BioC package. Please do not confuse our users by implying that any of this is about to change. You are more than welcome to experiment with new approaches, new implementation, new distributions, ... but *please* make sure you continue to do so under a new name too. or what ? This list is about Rcpp. The CRAN package. In mt view, this list is about R and C++, broadly. At least it was the intention when I created it many years ago. There are plenty of threads in this list that are not stricly related to Rcpp. The world of R and C++ is bigger than Rcpp. Romain ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Distribution of Rcpp codebase
On Mon, Apr 7, 2014 at 9:20 AM, Romain François rom...@r-enthusiasts.comwrote: [...] The only small downsides I see here is that (1) users potentially have to do more work to include Rcpp* in their packages (although you can just write an R function to include/update their Rcpp* versions); and that (2) source packages will be somewhat bigger. The difference is that if they don't want a new version, they can stick with the version they have in their package. So when they want newer Rcpp, they can have it, but they are not forced to do anything when the Rcpp* team wants to update the codebase. About 1). We just need to find a good way to propagate newer Rcpp* into a repo. Maybe through git subtree, git sub modules or something. We can come up with some tools. Oh, I see, you mean installing Rcpp* dependent packages from git(hub) directly. I imagine some people would want to put the headers into their tree, as ordinary files. Others might want to have it as submodules. For the latter to work, I imagine you would need to set up some special repo structure. Btw. I obviously don't have to tell you, but I want to point it out that this snapshotting is already possible today. I can just take the Rcpp* headers and put them in my package, and then I don't have to worry about future changes. This is assuming the package is header-only. With compiled code it is trickier, because of the name clashes. So I think, the question is whether to support this with some infrastructure, e.g. to update the embedded headers, have them as a git submodule, etc. About 2). So What ;) Agreed. CRAN-core might be fussy about it, though, if source packages get much bigger because of this. Or might be not. Gabor [...] ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Distribution of Rcpp codebase
Le 7 avr. 2014 à 19:20, Gábor Csárdi csardi.ga...@gmail.com a écrit : On Mon, Apr 7, 2014 at 9:20 AM, Romain François rom...@r-enthusiasts.com wrote: [...] The only small downsides I see here is that (1) users potentially have to do more work to include Rcpp* in their packages (although you can just write an R function to include/update their Rcpp* versions); and that (2) source packages will be somewhat bigger. The difference is that if they don’t want a new version, they can stick with the version they have in their package. So when they want newer Rcpp, they can have it, but they are not forced to do anything when the Rcpp* team wants to update the codebase. About 1). We just need to find a good way to propagate newer Rcpp* into a repo. Maybe through git subtree, git sub modules or something. We can come up with some tools. Oh, I see, you mean installing Rcpp* dependent packages from git(hub) directly. I imagine some people would want to put the headers into their tree, as ordinary files. Having the files into the client package is what I meant. This way, the developer of the package is in control of what version is used. The challenge is how to make it easy. Others might want to have it as submodules. For the latter to work, I imagine you would need to set up some special repo structure. I think git subtree gives something close to what I mean. But obviously someone could just copy the files manually, it would just be a matter of copying the inst/include directory. Btw. I obviously don't have to tell you, but I want to point it out that this snapshotting is already possible today. I can just take the Rcpp* headers and put them in my package, and then I don't have to worry about future changes. This is assuming the package is header-only. With compiled code it is trickier, because of the name clashes. Good luck doing that with current Rcpp. Many functions are compiled into the shared library shipped with the package. It is not linking as it used to do it, which is the big win of 0.11.0, but there is still a library. Many of these functions go through the namespace of the Rcpp package, etc … so you could snapshot Rcpp, but it would definitely not be easy at all. So I think, the question is whether to support this with some infrastructure, e.g. to update the embedded headers, have them as a git submodule, etc. About 2). So What ;) Agreed. CRAN-core might be fussy about it, though, if source packages get much bigger because of this. Or might be not. This is a drop of water in the ocean of CRAN packages. I think the benefit would outweigh this quite easily. In any case, not gonna happen, at least not with Rcpp. Romain ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Distribution of Rcpp codebase
Here's some thoughts, from the perspective of a package developer who might want to use Rcpp11. One option is to have the Rcpp11 distribution live as a git repository within the `inst/include` directory of a developer's package. A package developer could clone the repository (or have it track the inst/include subdirectory of Rcpp11 -- not sure if that's possible), and at their leisure pull the current version, pull a certain tag / release, and so on.). Maybe there's something that needs to be in .Rbuildignore to ensure the .git repository doesn't get added into the built package on R CMD build. Some R functions, e.g. useRcpp11(), could be written that, when run in the package directory, clones a repository in inst/include/Rcpp, and also updates pertinent licensing information (this package uses Rcpp11, which is licensing under so-and-so...). Essentially they would be nice wrappers to git. Maybe package these as part of an Rcpp11 CRAN / GitHub release. I'm not sure how well a mechanism like this would work for syncing across the other Rcpp libraries, e.g. RcppArmadillo, though. One should expect that packages depending on Rcpp11 will still want to be submitted to CRAN, and one potential problem (I know we could say 'not our problem', but we should be good citizens) is that, if R-Core decides to make an R API change that causes these packages to fail R CMD check, it will be more painful for them to accept maintenance releases from all these packages. Finally -- we might consider moving this discussion to a new mailing list, since I think Rcpp and Rcpp11 are divergent enough that we should consider Rcpp-devel for Rcpp-the-package-specific discussion, and a different mailing list for Rcpp11 development. Thoughts, Romain? On Mon, Apr 7, 2014 at 10:20 AM, Gábor Csárdi csardi.ga...@gmail.com wrote: On Mon, Apr 7, 2014 at 9:20 AM, Romain François rom...@r-enthusiasts.com wrote: [...] The only small downsides I see here is that (1) users potentially have to do more work to include Rcpp* in their packages (although you can just write an R function to include/update their Rcpp* versions); and that (2) source packages will be somewhat bigger. The difference is that if they don't want a new version, they can stick with the version they have in their package. So when they want newer Rcpp, they can have it, but they are not forced to do anything when the Rcpp* team wants to update the codebase. About 1). We just need to find a good way to propagate newer Rcpp* into a repo. Maybe through git subtree, git sub modules or something. We can come up with some tools. Oh, I see, you mean installing Rcpp* dependent packages from git(hub) directly. I imagine some people would want to put the headers into their tree, as ordinary files. Others might want to have it as submodules. For the latter to work, I imagine you would need to set up some special repo structure. Btw. I obviously don't have to tell you, but I want to point it out that this snapshotting is already possible today. I can just take the Rcpp* headers and put them in my package, and then I don't have to worry about future changes. This is assuming the package is header-only. With compiled code it is trickier, because of the name clashes. So I think, the question is whether to support this with some infrastructure, e.g. to update the embedded headers, have them as a git submodule, etc. About 2). So What ;) Agreed. CRAN-core might be fussy about it, though, if source packages get much bigger because of this. Or might be not. Gabor [...] ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Distribution of Rcpp codebase
On Mon, Apr 7, 2014 at 2:19 PM, Romain François rom...@r-enthusiasts.comwrote: [...] Oh, I see, you mean installing Rcpp* dependent packages from git(hub) directly. I imagine some people would want to put the headers into their tree, as ordinary files. Having the files into the client package is what I meant. This way, the developer of the package is in control of what version is used. The challenge is how to make it easy. Sure, we are on the same page here. What I meant is that if the client package is on CRAN, then you'll have to include the Rcpp* header files in the package, obviously. Unless you create Rcpp* packages that contain multiple versions of the headers, in different folders. Personally, if I created a client package for Rcpp11 today, I would just copy over the header files, in a subfolder, and keep them unchanged, so that it is easy to update them with a delete/copy. If there is an RcppJam package that can update these files, perhaps add compilation flags, etc., that's also great. Btw. I obviously don't have to tell you, but I want to point it out that this snapshotting is already possible today. I can just take the Rcpp* headers and put them in my package, and then I don't have to worry about future changes. This is assuming the package is header-only. With compiled code it is trickier, because of the name clashes. Good luck doing that with current Rcpp. Many functions are compiled into the shared library shipped with the package. It is not linking as it used to do it, which is the big win of 0.11.0, but there is still a library. Many of these functions go through the namespace of the Rcpp package, etc ... so you could snapshot Rcpp, but it would definitely not be easy at all. I agree. With header-only packages it'll be easy. With compiled code it is trickier, but C++ is actually easier than C, because for the name clashes, you can just put the code in a new namespace. Then you'll need to redefine :: and .Call in your package, and even this might not be enough, especially for Rcpp. Gabor ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Distribution of Rcpp codebase
Thanks to have interest in this discussion. Le 7 avr. 2014 à 20:30, Kevin Ushey kevinus...@gmail.com a écrit : Here's some thoughts, from the perspective of a package developer who might want to use Rcpp11. One option is to have the Rcpp11 distribution live as a git repository within the `inst/include` directory of a developer's package. A package developer could clone the repository (or have it track the inst/include subdirectory of Rcpp11 -- not sure if that's possible), and at their leisure pull the current version, pull a certain tag / release, and so on.). Maybe there's something that needs to be in .Rbuildignore to ensure the .git repository doesn't get added into the built package on R CMD build. Some R functions, e.g. useRcpp11(), could be written that, when run in the package directory, clones a repository in inst/include/Rcpp, and also updates pertinent licensing information (this package uses Rcpp11, which is licensing under so-and-so...). Essentially they would be nice wrappers to git. Maybe package these as part of an Rcpp11 CRAN / GitHub release. I think git subtree is the right tool. I'm not sure how well a mechanism like this would work for syncing across the other Rcpp libraries, e.g. RcppArmadillo, though. As long as these other are header only, that would be pretty easy. We’d have to diverge them as well as they carry various fastLm, etc ... which really don’t belong there (before the Rcpp police gets involves : I know they will stay). One should expect that packages depending on Rcpp11 will still want to be submitted to CRAN, and one potential problem (I know we could say 'not our problem', but we should be good citizens) is that, if R-Core decides to make an R API change that causes these packages to fail R CMD check, it will be more painful for them to accept maintenance releases from all these packages. There are guarantees on R api changes. I don’t see this as an issue. Finally -- we might consider moving this discussion to a new mailing list, since I think Rcpp and Rcpp11 are divergent enough that we should consider Rcpp-devel for Rcpp-the-package-specific discussion, and a different mailing list for Rcpp11 development. Thoughts, Romain? I though this mailing list was about discussing R and C++, but apparently I was wrong. Fair enough if this is just Rcpp related. Perhaps I’ll leave the mailing list as I’m not that interested in Rcpp anymore. Rcpp11 has its own github issues, this I think is enough for now, but sure maybe a new mailing list would be good. On Mon, Apr 7, 2014 at 10:20 AM, Gábor Csárdi csardi.ga...@gmail.com wrote: On Mon, Apr 7, 2014 at 9:20 AM, Romain François rom...@r-enthusiasts.com wrote: [...] The only small downsides I see here is that (1) users potentially have to do more work to include Rcpp* in their packages (although you can just write an R function to include/update their Rcpp* versions); and that (2) source packages will be somewhat bigger. The difference is that if they don't want a new version, they can stick with the version they have in their package. So when they want newer Rcpp, they can have it, but they are not forced to do anything when the Rcpp* team wants to update the codebase. About 1). We just need to find a good way to propagate newer Rcpp* into a repo. Maybe through git subtree, git sub modules or something. We can come up with some tools. Oh, I see, you mean installing Rcpp* dependent packages from git(hub) directly. I imagine some people would want to put the headers into their tree, as ordinary files. Others might want to have it as submodules. For the latter to work, I imagine you would need to set up some special repo structure. Btw. I obviously don't have to tell you, but I want to point it out that this snapshotting is already possible today. I can just take the Rcpp* headers and put them in my package, and then I don't have to worry about future changes. This is assuming the package is header-only. With compiled code it is trickier, because of the name clashes. So I think, the question is whether to support this with some infrastructure, e.g. to update the embedded headers, have them as a git submodule, etc. About 2). So What ;) Agreed. CRAN-core might be fussy about it, though, if source packages get much bigger because of this. Or might be not. Gabor [...] ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Distribution of Rcpp codebase
On Mon, Apr 7, 2014 at 2:59 PM, Romain François rom...@r-enthusiasts.comwrote: [...] Some R functions, e.g. useRcpp11(), could be written that, when run in the package directory, clones a repository in inst/include/Rcpp, and also updates pertinent licensing information (this package uses Rcpp11, which is licensing under so-and-so...). Essentially they would be nice wrappers to git. Maybe package these as part of an Rcpp11 CRAN / GitHub release. I think git subtree is the right tool. Personally, I would not make git compulsory for developers of clients, and I would just add/modify R code in Rcpp11 to download/update the clones and include them in the client. I think it is simpler than setting up git subtree, etc. One should expect that packages depending on Rcpp11 will still want to be submitted to CRAN, and one potential problem (I know we could say 'not our problem', but we should be good citizens) is that, if R-Core decides to make an R API change that causes these packages to fail R CMD check, it will be more painful for them to accept maintenance releases from all these packages. They can just update to a newer embedded Rcpp11. You can keep two Rcpp11 branches alive if you want, so that users can upgrade to a bug-fix version without breaking the Rcpp11 API in their package. Gabor ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
[Rcpp-devel] [ANN] RcppArmadillo 0.4.200.0
Earlier today, Conrad released a new stable release Armadillo 4.200. As in the past, I rolled this up for CRAN, and we now have a fresh RcppArmadillo 0.4.200.0 on CRAN. This release was preceded by two test release which were on Github only (as CRAN prefers to have about one release / month). I tested it against all 61 CRAN packages using RcppArmadillo (but did not install all Depends / Suggests so no claim to being complete) and did not uncover any issues. Special thanks to Conrad for adding the and || operators I suggested. Enjoy, Dirk -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
