[Accidentally sent the following to r-sig-mac-request@*, so now forwarding ...]
Matrix will commence ABI versioning in 1.6-2. The ABI version will be available
in R as Matrix.Version()[["abi"]] and in C as R_MATRIX_ABI_VERSION (from header
Matrix/Matrix.h). It is numeric_version("1") in Matrix 1.6-2 and _implicitly_
numeric_version("0") in Matrix < 1.6-2.
Packages with "LinkingTo: Matrix" wanting to detect binary incompatibility can
compare the ABI versions available at build and load time and warn if they
differ. The ABI version can be expected to change much more slowly than the
package version, if we are "maintaining" correctly ...
That still does not address issues due to caching of S4 methods and classes not
declared in NAMESPACEs with importMethodsFrom or importClassesFrom, so rebuilds
may still be requested periodically, outside of ABI changes.
Mikael
On 2023-09-26 3:03 am, r-sig-mac-requ...@r-project.org wrote:
On 26.09.2023 04:36, Simon Urbanek wrote:
On 26/09/2023, at 12:28 PM, Kirill Müller<kir...@cynkra.com> wrote:
Thanks, Simon and Uwe, for your inputs.
It's not that TMB requires a particular version of Matrix, it just suggests
using the same version of Matrix as it was built with. We may have seen this
problem before, with innocuous changes in Rcpp breaking downstream packages.
A different example, running in two separate sessions for comparison, is shown
below.
That looks like an issue in TMB - is there really a problem between the two
versions or is TMB just making it up? I strongly suspect the latter.
Same from here.
Windows may be rebuilding more often than strictly necessary, but it seems the
right thing to do to me. We could assess the overhead using historic data to
see how big of an impact that would be.
I think we'd only need to rebuild in rare cases, mostly in case of S4
changes, only in few cases for other reasons, e.g. for stored objects or
linking. The overhead is immense.
finding out how it works is on my ToDo list for a long time, (as sereval
other things), but not simply at all. I have some heuristics i mind,
though. Well, CRAN management eats up too much time anyway.
I hear you're proposing that the TMB maintainers release an updated version
whenever Matrix is updated?
Short answer is no for most people, except TMB's own current behavior requires
it (by its own design) so it depends on what the TMB maintainers really want -
if they really want to insist on very specific Matrix version (for whatever
reason), then yes.
The version generally doesn't matter in vast majority of cases, because
packages typically don't make changes that break their reverse dependencies -
which is why the check like that TMB does is generally not very helpful since
it is way too conservative to the point of being uninformative. Even if you use
LinkingTo, you should not be relying on internal behavior of a very specific
version of a package - that's the whole point of defined API like what packages
present through the LinkingTo-API - you only request a version of the API
regardless of its implementation. If you use a new feature then you obviously
have to bump your version and specify the the new version of your dependency so
that problem doesn't arise. So there is an issue only if your dependency makes
a change which breaks your own package (which is extremely rare - in the case
of Matrix it is unfortunate, mostly due to issues in the S4 implementation) and
then the only reliable solution is to bump your version since the two binaries
are undistinguishable. The problem is that there is no automated way to do that
- pretty much the dependency has to tell you or you have to find out yourself.
As I was explaining this is not about re-building - it doesn't help because
it's not generally available to users as they have no way to know that they
need to re-install exactly the same version of a package like TMB - unless the
package bumps the version. Essentially the users would have to remove and
re-install all packages every time they update a single one which is not
realistic.
Yes, as I said in my former message, update.packages() won't know about
the new TMB binary, the user has to install.packages("TMB") it explicitly.
From my point of view TMB should not generally check the version like
this unless it can really test for the need to rebuild.
If all packages would be doing that half of CRAN would be broken after a
Matrix or MASS update...
Best,
Uwe
Cheers,
Simon
Best regards
Kirill
# Example 1: happy path
dir.create("templib")
.libPaths(normalizePath("templib"))
Sys.setenv(R_LIBS_USER = .libPaths()[[1]])
# Sanity check
rownames(installed.packages())
#> [1] "base" "boot" "class" "cluster" "codetools"
#> [6] "compiler" "datasets" "foreign" "graphics" "grDevices"
#> [11] "grid" "KernSmooth" "lattice" "MASS" "Matrix"
#> [16] "methods" "mgcv" "nlme" "nnet" "parallel"
#> [21] "rpart" "spatial" "splines" "stats" "stats4"
#> [26] "survival" "tcltk" "tools" "utils"
packageVersion("Matrix")
#> [1] '1.5.4.1'
# Install TMB from source, now matching the Matrix version
install.packages(c("Rcpp", "RcppEigen"))
#> Installing packages into
'/private/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T/RtmpSL7Etm/reprex-bc8b35150024-vague-bull/templib'
#> (as 'lib' is unspecified)
#>
#> The downloaded binary packages are in
#>
/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T//RtmpOpHc0P/downloaded_packages
install.packages("TMB", type = "source", repos ="https://cloud.r-project.org/";)
#> Installing package into
'/private/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T/RtmpSL7Etm/reprex-bc8b35150024-vague-bull/templib'
#> (as 'lib' is unspecified)
# Perform implicit consistency check
packageVersion("Matrix")
#> [1] '1.5.4.1'
requireNamespace("TMB")
#> Loading required namespace: TMB
# Example 2: upgrading Matrix after installing TMB
dir.create("templib")
.libPaths(normalizePath("templib"))
Sys.setenv(R_LIBS_USER = .libPaths()[[1]])
# Sanity check
rownames(installed.packages())
#> [1] "base" "boot" "class" "cluster" "codetools"
#> [6] "compiler" "datasets" "foreign" "graphics" "grDevices"
#> [11] "grid" "KernSmooth" "lattice" "MASS" "Matrix"
#> [16] "methods" "mgcv" "nlme" "nnet" "parallel"
#> [21] "rpart" "spatial" "splines" "stats" "stats4"
#> [26] "survival" "tcltk" "tools" "utils"
packageVersion("Matrix")
#> [1] '1.5.4.1'
# Install TMB from source, now matching the Matrix version
install.packages(c("Rcpp", "RcppEigen"))
#> Installing packages into
'/private/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T/RtmpSL7Etm/reprex-bc8b4fff25a6-next-esok/templib'
#> (as 'lib' is unspecified)
#>
#> The downloaded binary packages are in
#>
/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T//RtmpyN9yHv/downloaded_packages
install.packages("TMB", type = "source", repos ="https://cloud.r-project.org/";)
#> Installing package into
'/private/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T/RtmpSL7Etm/reprex-bc8b4fff25a6-next-esok/templib'
#> (as 'lib' is unspecified)
# Upgrading Matrix
install.packages("Matrix")
#> Installing package into
'/private/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T/RtmpSL7Etm/reprex-bc8b4fff25a6-next-esok/templib'
#> (as 'lib' is unspecified)
#>
#> The downloaded binary packages are in
#>
/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T//RtmpyN9yHv/downloaded_packages
# Perform implicit consistency check
packageVersion("Matrix")
#> [1] '1.6.1.1'
requireNamespace("TMB")
#> Loading required namespace: TMB
#> Warning in checkMatrixPackageVersion(): Package version inconsistency
detected.
#> TMB was built with Matrix version 1.5.4.1
#> Current Matrix version is 1.6.1.1
#> Please re-install 'TMB' from source using install.packages('TMB', type =
'source') or ask CRAN for a binary version of 'TMB' matching CRAN's 'Matrix'
package
On 25.09.23 15:22, Simon Urbanek wrote:
Kirill,
we have no way of knowing when a packages introduced a breaking change (which
it really shouldn't), so the maintainers would have to inform as (Matrix
authors did inform us that 1.6.0 introduces breaking change so that's why we
did a manual re-bulid on that update). Also checking exact version (like TMB
seems to do) doesn't work for the same reason and is not a good way to deal
with the problem - we cannot re-build all packages with each change, because of
the amount of time that would take (by now it takes more that a day to re-build
CRAN just for one R target) and more fundamental problem with re-builds:
The real problem here is deeper than just triggering rebuilds: the users have
no way of knowing if a package has been re-built and updated, because the
version does not change. If let's say TMB is release 1.0 used Matrix 1.6.0 and
then Matrix is updated to 1.6.1 then TMB cannot require it because it did not
bump its version, i.e. the binary for TMB 1.0 with Matrix 1.6.0 is
indistinguishable from a binary of TMB 1.0 with Matrix 1.6.1 on CRAN so if the
user used update.packages() it would not count as a new TMB version. Therefore
TMB really has to update its version from 1.0 to let's say 1.0-1 if it requests
a rebuild against changes in Matrix 1.6.1. Clearly, this is not ideal, but
currently there is other way that would guarantee consistency of binaries. We
are aware of the problem and we are we are in fact thinking about possible
solutions (special tags on binary versions) for future version of R, but for
now the package authors have to be aware of potential breakage upstream.
Cheers,
Simon
On 26/09/2023, at 7:46 AM, Kirill Müller via R-SIG-Mac<r-sig-mac@r-project.org>
wrote:
Hi
The TMB package has Matrix in its LinkingTo dependencies. In a clean package
library, I see behavior posted below.
Should the TMB package have been rebuilt when a new version of the Matrix
package was pushed? More general, should we rebuild all reverse LinkingTo
dependencies of a package that gets an update on CRAN? This will mean a lot of
package rebuilds with Rcpp, but also a lot less frustration down the road.
Thanks!
Best regards
Kirill
dir.create("templib")
.libPaths(normalizePath("templib"))
# Sanity check
rownames(installed.packages())
#> [1] "base" "boot" "class" "cluster" "codetools"
#> [6] "compiler" "datasets" "foreign" "graphics" "grDevices"
#> [11] "grid" "KernSmooth" "lattice" "MASS" "Matrix"
#> [16] "methods" "mgcv" "nlme" "nnet" "parallel"
#> [21] "rpart" "spatial" "splines" "stats" "stats4"
#> [26] "survival" "tcltk" "tools" "utils"
packageVersion("Matrix")
#> [1] '1.5.4.1'
# Install fresh binary packages
install.packages(c("Matrix", "TMB"))
#> Installing packages into
'/private/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T/RtmpXIjIPy/reprex-87aa25cc1fa4-sugar-cob/templib'
#> (as 'lib' is unspecified)
#> also installing the dependencies 'Rcpp', 'RcppEigen'
#>
#> The downloaded binary packages are in
#>
/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T//Rtmp4GSL7m/downloaded_packages
packageVersion("Matrix")
#> [1] '1.6.1.1'
# Perform implicit consistency check
requireNamespace("TMB")
#> Loading required namespace: TMB
#> Warning in checkMatrixPackageVersion(): Package version inconsistency
detected.
#> TMB was built with Matrix version 1.6.0
#> Current Matrix version is 1.6.1.1
#> Please re-install 'TMB' from source using install.packages('TMB', type =
'source') or ask CRAN for a binary version of 'TMB' matching CRAN's 'Matrix'
package
_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac
_______________________________________________
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac
