Hello all, Following up on this old thread as I have recently observed, rather a bad practice (maintaining order of installation for R packages rather than relying on R for that), for solving a problem that R branch tools4pkgs (mentioned in this email) addresses very well. More details can be found in https://github.com/dewittpe/R-install-dependencies/issues/3
Therefore I extracted functionality from base R branch and put into standalone package, named after R branch: https://github.com/jangorecki/tools4pkgs Sharing for whoever would reach this email thread in future. Best Regards, Jan Gorecki On Sat, Oct 29, 2022 at 6:26 PM Jan Gorecki <j.gore...@wit.edu.pl> wrote: > > Thank you Gabriel, > > Just for future readers. Below is a base R way to address this common > problem, as instructed by you (+stopifnot to suppress print). > > Rscript -e 'stopifnot(file.copy("DESCRIPTION", > file.path(tdir<-tempdir(), "PACKAGES"))); > db<-available.packages(paste0("file://", tdir)); > install.packages(setdiff(tools::package_dependencies(read.dcf("DESCRIPTION", > fields="Package")[[1L]], db, which="most")[[1L]], > installed.packages(priority="high")[,"Package"]))' > > 3 liner, 310 chars long command, far from ideal, but does work. > > Best, > Jan > > > On Fri, Oct 28, 2022 at 10:42 PM Gabriel Becker <gabembec...@gmail.com> wrote: > > > > Hi Jan, > > > > > > On Fri, Oct 28, 2022 at 1:57 PM Jan Gorecki <j.gore...@wit.edu.pl> wrote: > >> > >> Gabriel, > >> > >> It is the most basic CI use case. One wants to install only > >> dependencies only of the package, and run R CMD check on the package. > > > > > > Really what you're looking for though, is to install all the dependencies > > which aren't present right? Excluding base packages is just a particular > > way to do that under certain assumptions about the CI environment. > > > > So > > > > > > needed_pkgs <- setdiff(package_dependencies(...), > > installed.packages()[,"Package"]) > > install.packages(needed_pkgs, repos = fancyrepos) > > > > > > will do what you want without installing the package itself, if that is > > important. This will filter out base and recommended packages (which will > > be already installed in your CI container, since R is). > > > > > > Now this does not take into account versioned dependencies, so it's not > > actually fully correct (whereas installing the package is), but it gets you > > where you're trying to go. And in a clean CI container without cached > > package installation for the deps, its equivalent. > > > > > > Also, as an aside, if you need to get the base packages, you can do > > > > installed.packages(priority="base")[,"Package"] > > > > base compiler datasets graphics grDevices grid > > > > "base" "compiler" "datasets" "graphics" "grDevices" "grid" > > > > methods parallel splines stats stats4 tcltk > > > > "methods" "parallel" "splines" "stats" "stats4" "tcltk" > > > > tools utils > > > > "tools" "utils" > > > > > > (to get base and recommended packages use 'high' instead of 'base') > > > > No need to be reaching down into unexported functions. So if you *really* > > only want to exclude base functions (which likely will give you some > > protection from versioned dep issues), you can change the code above to > > > > needed_pkgs <- setdiff(package_dependencies(...), > > installed.packages(priority = "high")[,"Package"]) > > install.packages(needed_pkgs, repos = fancyrepos) > > > > Best, > > ~G > > > >> > >> On Fri, Oct 28, 2022 at 8:42 PM Gabriel Becker <gabembec...@gmail.com> > >> wrote: > >> > > >> > Hi Jan, > >> > > >> > The reason, I suspect without speaking for R-core, is that by design you > >> > should not be specifying package dependencies as additional packages to > >> > install. install.packages already does this for you, as it did in the > >> > construct of a repository code that I provided previously in the thread. > >> > You should be *only* doing > >> > > >> > install.packages(<pkg in question>, repos = *) > >> > > >> > Then everything happens automatically via extremely well tested very > >> > mature code. > >> > > >> > I (still) don't understand why you'd need to pass install.packages the > >> > vector of dependencies yourself, as that is counter to install.packages' > >> > core design. > >> > > >> > Does that make sense? > >> > > >> > Best, > >> > ~G > >> > > >> > On Fri, Oct 28, 2022 at 12:18 PM Jan Gorecki <j.gore...@wit.edu.pl> > >> > wrote: > >> >> > >> >> Gabriel, > >> >> > >> >> I am trying to design generic solution that could be applied to > >> >> arbitrary package. Therefore I went with the latter solution you > >> >> proposed. > >> >> If we wouldn't have to exclude base packages, then its a 3 liner > >> >> > >> >> file.copy("DESCRIPTION", file.path(tdir<-tempdir(), "PACKAGES")); > >> >> db<-available.packages(paste0("file://", tdir)); > >> >> utils::install.packages(tools::package_dependencies("pkgname", db, > >> >> which="most")[[1L]]) > >> >> > >> >> As you noticed, we still have to filter out base packages. Otherwise > >> >> it won't be a robust utility that can be used in CI. Therefore we have > >> >> to add a call to tools:::.get_standard_package_names() which is an > >> >> internal function (as of now). Not only complicating the call but also > >> >> putting the functionality outside of safe use. > >> >> > >> >> Considering above, don't you agree that the following one liner could > >> >> nicely address the problem? The problem that hundreds/thousands of > >> >> packages are now addressing in their CI scripts by using a third party > >> >> packages. > >> >> > >> >> utils::install.packages(packages.dcf("DESCRIPTION", which="most")) > >> >> > >> >> It is hard to me to understand why R members don't consider this basic > >> >> functionality to be part of base R. Possibly they just don't need it > >> >> themselves. Yet isn't this sufficient that hundreds/thousands of > >> >> packages does need this functionality? > >> >> > >> >> Best regards, > >> >> Jan > >> >> > >> >> On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gore...@wit.edu.pl> > >> >> wrote: > >> >> > > >> >> > Gabriel and Simon > >> >> > > >> >> > I completely agree with what you are saying. > >> >> > The thing is that obtaining recursive deps, all/most whatever, is > >> >> > already well supported in core R. What is missing is just this single > >> >> > functionality I am requesting. > >> >> > > >> >> > If you will look into the branch you can see there is mirror.packages > >> >> > function meant to mirror a slice of CRAN. It is doing exactly what > >> >> > you described: package_dependencies; to obtain recursive deps, then > >> >> > download all, etc. > >> >> > I would love to have this function provided by core R as well, but we > >> >> > need to start somewhere. > >> >> > > >> >> > There are other use cases as well. > >> >> > For example CI, where one wants to install all/most dependencies and > >> >> > then run R CMD check. Then we don't worry about recursive deps are > >> >> > they will be resolved automatically. > >> >> > I don't think it's reasonable to force users to use 3rd party > >> >> > packages to handle such a common and simple use case. Otherwise one > >> >> > has to hard code deps in CI script. Not robust at all. > >> >> > > >> >> > packages.dcf and repos.dcf makes all that way easier, and are solid > >> >> > base for building customized orchestration like mirroring slice of > >> >> > CRAN. > >> >> > > >> >> > Best regards > >> >> > Jan > >> >> > > >> >> > On Sun, Oct 16, 2022, 01:31 Simon Urbanek > >> >> > <simon.urba...@r-project.org> wrote: > >> >> >> > >> >> >> Jan, > >> >> >> > >> >> >> I think using a single DCF as input is not very practical and would > >> >> >> not be useful in the context you describe (creating self contained > >> >> >> repos) since they typically concern a list of packages, but > >> >> >> essentially splitting out the part of install.packages() which > >> >> >> determines which files will be pulled from where would be very > >> >> >> useful as it would be trivial to use it to create repository (what > >> >> >> we always do in corporate environments) instead of installing the > >> >> >> packages. I suspect that install packages is already too complex so > >> >> >> instead of adding a flag to install.packages one could move that > >> >> >> functionality into a separate function - we all do that constantly > >> >> >> for the sites we manage, so it would be certainly something > >> >> >> worthwhile. > >> >> >> > >> >> >> Cheers, > >> >> >> Simon > >> >> >> > >> >> >> > >> >> >> > On Oct 15, 2022, at 7:14 PM, Jan Gorecki <j.gore...@wit.edu.pl> > >> >> >> > wrote: > >> >> >> > > >> >> >> > Hi Gabriel, > >> >> >> > > >> >> >> > It's very nice usage you provided here. Maybe instead of adding new > >> >> >> > function we could extend packages_depenedncies then? To accept > >> >> >> > file path to > >> >> >> > dsc file. > >> >> >> > > >> >> >> > What about repos.dcf? Maybe additional repositories could be an > >> >> >> > attribute > >> >> >> > attached to returned character vector. > >> >> >> > > >> >> >> > The use case is to, for a given package sources, obtain its > >> >> >> > dependencies, > >> >> >> > so one can use that for installing them/mirroring CRAN subset, or > >> >> >> > whatever. > >> >> >> > The later is especially important for a production environment > >> >> >> > where one > >> >> >> > wants to have fixed version of packages, and mirroring relevant > >> >> >> > subset of > >> >> >> > CRAN is the most simple, and IMO reliable, way to manage such > >> >> >> > environment. > >> >> >> > > >> >> >> > Regards > >> >> >> > Jan > >> >> >> > > >> >> >> > On Fri, Oct 14, 2022, 23:34 Gabriel Becker <gabembec...@gmail.com> > >> >> >> > wrote: > >> >> >> > > >> >> >> >> Hi Jan and Jan, > >> >> >> >> > >> >> >> >> Can you explain a little more what exactly you want the > >> >> >> >> non-recursive, > >> >> >> >> non-version aware dependencies from an individual package for? > >> >> >> >> > >> >> >> >> Either way package_dependencies will do this for you* with a > >> >> >> >> little > >> >> >> >> "aggressive convincing". It wants output from available.packages, > >> >> >> >> but who > >> >> >> >> really cares what it wants? It's a function and we are people :) > >> >> >> >> > >> >> >> >>> library(tools) > >> >> >> >>> db <- read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION") > >> >> >> >>> package_dependencies("rtables", db, which = > >> >> >> >>> intersect(c("Depends", > >> >> >> >> "Suggests", "Imports", "LinkingTo"), colnames(db))) > >> >> >> >> $rtables > >> >> >> >> [1] "methods" "magrittr" "formatters" "dplyr" "tibble" > >> >> >> >> [6] "tidyr" "testthat" "xml2" "knitr" > >> >> >> >> "rmarkdown" > >> >> >> >> [11] "flextable" "officer" "stats" "htmltools" "grid" > >> >> >> >> > >> >> >> >> > >> >> >> >> The only gotcha that I see immediately is that "LinkingTo" isn't > >> >> >> >> always > >> >> >> >> there (whereas it is with real output from available.packages). > >> >> >> >> If you > >> >> >> >> know your package doesn't have that (or that it does) at call > >> >> >> >> time , this > >> >> >> >> becomes a one-liner: > >> >> >> >> > >> >> >> >> package_dependencies("rtables", db = > >> >> >> >> read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which = > >> >> >> >> c("Depends", "Suggests", "Imports")) > >> >> >> >> $rtables > >> >> >> >> [1] "methods" "magrittr" "formatters" "dplyr" "tibble" > >> >> >> >> [6] "tidyr" "testthat" "xml2" "knitr" > >> >> >> >> "rmarkdown" > >> >> >> >> [11] "flextable" "officer" "stats" "htmltools" "grid" > >> >> >> >> > >> >> >> >> You can also trick it a slightly different way by giving it what > >> >> >> >> it > >> >> >> >> actually wants > >> >> >> >> > >> >> >> >>> tdir <- tempdir() > >> >> >> >>> file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION", > >> >> >> >>> file.path(tdir, > >> >> >> >> "PACKAGES")) > >> >> >> >> [1] TRUE > >> >> >> >>> avl <- available.packages(paste0("file://", tdir)) > >> >> >> >>> library(tools) > >> >> >> >>> package_dependencies("rtables", avl) > >> >> >> >> $rtables > >> >> >> >> [1] "methods" "magrittr" "formatters" "stats" > >> >> >> >> "htmltools" > >> >> >> >> [6] "grid" > >> >> >> >> > >> >> >> >>> package_dependencies("rtables", avl, which = "all") > >> >> >> >> $rtables > >> >> >> >> [1] "methods" "magrittr" "formatters" "stats" > >> >> >> >> "htmltools" > >> >> >> >> [6] "grid" "dplyr" "tibble" "tidyr" "testthat" > >> >> >> >> [11] "xml2" "knitr" "rmarkdown" "flextable" "officer" > >> >> >> >> > >> >> >> >> So the only real benefits I see that we'd be picking up here is > >> >> >> >> automatic > >> >> >> >> filtering by priority, and automatic extraction of the package > >> >> >> >> name from > >> >> >> >> the DESCRIPTION file. I'm not sure either of those warrant a new > >> >> >> >> exported > >> >> >> >> function that R-core has to maintain forever. > >> >> >> >> > >> >> >> >> Best, > >> >> >> >> ~G > >> >> >> >> > >> >> >> >> * I haven't tested this across all OSes, but I dont' know of any > >> >> >> >> reason it > >> >> >> >> wouldn't work generally. > >> >> >> >> > >> >> >> >> On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki > >> >> >> >> <j.gore...@wit.edu.pl> wrote: > >> >> >> >> > >> >> >> >>> Hello Jan, > >> >> >> >>> > >> >> >> >>> Thanks for confirming about many packages reinventing this > >> >> >> >>> missing > >> >> >> >>> functionality. > >> >> >> >>> packages.dcf was not meant handle versions. It just extracts > >> >> >> >>> names of > >> >> >> >>> dependencies... Yes, such a simple thing, yet missing in base R. > >> >> >> >>> > >> >> >> >>> Versions of packages can be controlled when setting up R pkgs > >> >> >> >>> repo. This > >> >> >> >>> is > >> >> >> >>> how I used to handle it. Making a CRAN subset mirror of fixed > >> >> >> >>> version > >> >> >> >>> pkgs. > >> >> >> >>> BTW. function for that is also included in mentioned branch. I > >> >> >> >>> am just not > >> >> >> >>> proposing it, to increase the chance of having at least this > >> >> >> >>> simple, > >> >> >> >>> missing, functionality merged. > >> >> >> >>> > >> >> >> >>> Best > >> >> >> >>> Jan > >> >> >> >>> > >> >> >> >>> On Fri, Oct 14, 2022, 15:14 Jan Netík <neti...@gmail.com> wrote: > >> >> >> >>> > >> >> >> >>>> Hello Jan, > >> >> >> >>>> > >> >> >> >>>> I have seen many packages that implemented dependencies > >> >> >> >>>> "extraction" on > >> >> >> >>>> their own for internal purposes and today I was doing exactly > >> >> >> >>>> that for > >> >> >> >>>> mine. It's not a big deal using read.dcf on DESCRIPTION. It was > >> >> >> >>> sufficient > >> >> >> >>>> for me, but I had to take care of some \n chars (the overall > >> >> >> >>>> returned > >> >> >> >>> value > >> >> >> >>>> has some rough edges, in my opinion). However, the function > >> >> >> >>>> from the > >> >> >> >>> branch > >> >> >> >>>> seems to not care about version requirements, which are crucial > >> >> >> >>>> for me. > >> >> >> >>>> Maybe that is something to reconsider before merging. > >> >> >> >>>> > >> >> >> >>>> Best, > >> >> >> >>>> Jan > >> >> >> >>>> > >> >> >> >>>> pá 14. 10. 2022 v 2:27 odesílatel Jan Gorecki > >> >> >> >>>> <j.gore...@wit.edu.pl> > >> >> >> >>>> napsal: > >> >> >> >>>> > >> >> >> >>>>> Dear R devs, > >> >> >> >>>>> > >> >> >> >>>>> I would like to raise a request for a simple helper function. > >> >> >> >>>>> Utility function to extract package dependencies from > >> >> >> >>>>> DESCRIPTION file. > >> >> >> >>>>> > >> >> >> >>>>> I do think that tools package is better place, for such a > >> >> >> >>>>> fundamental > >> >> >> >>>>> functionality, than community packages. > >> >> >> >>>>> > >> >> >> >>>>> tools pkg seems perfect fit (having already great function > >> >> >> >>>>> write_PACKAGES). > >> >> >> >>>>> > >> >> >> >>>>> Functionality I am asking for is already in R svn repository > >> >> >> >>>>> since > >> >> >> >>> 2016, > >> >> >> >>>>> in > >> >> >> >>>>> a branch tools4pkgs. Function is called 'packages.dcf'. > >> >> >> >>>>> Another one 'repos.dcf' would be a good functional > >> >> >> >>>>> complementary to it. > >> >> >> >>>>> > >> >> >> >>>>> Those two simple helper functions really makes it easier for > >> >> >> >>> organizations > >> >> >> >>>>> to glue together usage of their own R packages repos and CRAN > >> >> >> >>>>> repo in a > >> >> >> >>>>> smooth way. That could possibly help to offload CRAN from new > >> >> >> >>> submissions. > >> >> >> >>>>> > >> >> >> >>>>> gh mirror link for easy preview: > >> >> >> >>>>> > >> >> >> >>>>> > >> >> >> >>> https://github.com/wch/r-source/blob/tools4pkgs/src/library/tools/R/packages.R#L419 > >> >> >> >>>>> > >> >> >> >>>>> Regards > >> >> >> >>>>> Jan Gorecki > >> >> >> >>>>> > >> >> >> >>>>> [[alternative HTML version deleted]] > >> >> >> >>>>> > >> >> >> >>>>> ______________________________________________ > >> >> >> >>>>> R-devel@r-project.org mailing list > >> >> >> >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel > >> >> >> >>>>> > >> >> >> >>>> > >> >> >> >>> > >> >> >> >>> [[alternative HTML version deleted]] > >> >> >> >>> > >> >> >> >>> ______________________________________________ > >> >> >> >>> R-devel@r-project.org mailing list > >> >> >> >>> https://stat.ethz.ch/mailman/listinfo/r-devel > >> >> >> >>> > >> >> >> >> > >> >> >> > > >> >> >> > [[alternative HTML version deleted]] > >> >> >> > > >> >> >> > ______________________________________________ > >> >> >> > R-devel@r-project.org mailing list > >> >> >> > https://stat.ethz.ch/mailman/listinfo/r-devel > >> >> >> > > >> >> >> ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
