Bug#980809: Matrix package default tolerance on s390x (Re: Bug#980809: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x)
Graham, Thanks for the bug tracker follow-up which made me aware of the ongoing discussion in #665 at glmmTMB. It's frustrating to have the run around but it really looks like as I argued all along: not an issue in Matrix. Now, TMB is of course a complex package too.. Appreciate you chasing this so thoroughly. Dirk PS What is bts-link? Is that a new service we have to tie our BTS to GH and others? -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
Bug#980809: Matrix package default tolerance on s390x (Re: Bug#980809: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x)
On 17 February 2021 at 16:58, Graham Inggs wrote: | Hi Dirk | | On Wed, 17 Feb 2021 at 15:10, Dirk Eddelbuettel wrote: | > Graham: What does that bigendian box say for Sys.info()[["machine"]] ? | | The other big-endian box I tested was perotto.debian.net [*], it says: | | > Sys.info()[["machine"]] | [1] "ppc64" | | > Should we test for endianness instead? And then maybe try to roll up to the | > cause of the difference? | | I've just tested on 32-bit big-endian and the glmmTMB problem does not | occur there, so at this stage I would say test for big-endian **and** | 64 bits. | | Just to be clear, is the fix you are proposing in Matrix going to fix | the glmmTMB error? No. That may well be something else. | If a bug shows up on 64-bit big-endian, but not on little-endian or | 32-bit big-endian, then it's usually a sign that somewhere a 64-bit | variable is incorrectly being read or written to as a 32-bit variable. CRAN uses a Solaris machine for some endianness variability but I believe that machine runs only 32bit. Dirk | Regards | Graham | | | [*] https://db.debian.org/machines.cgi?sortby=purpose=dcs -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
Bug#980809: Matrix package default tolerance on s390x (Re: Bug#980809: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x)
Hi Dirk On Wed, 17 Feb 2021 at 15:10, Dirk Eddelbuettel wrote: > Graham: What does that bigendian box say for Sys.info()[["machine"]] ? The other big-endian box I tested was perotto.debian.net [*], it says: > Sys.info()[["machine"]] [1] "ppc64" > Should we test for endianness instead? And then maybe try to roll up to the > cause of the difference? I've just tested on 32-bit big-endian and the glmmTMB problem does not occur there, so at this stage I would say test for big-endian **and** 64 bits. Just to be clear, is the fix you are proposing in Matrix going to fix the glmmTMB error? If a bug shows up on 64-bit big-endian, but not on little-endian or 32-bit big-endian, then it's usually a sign that somewhere a 64-bit variable is incorrectly being read or written to as a 32-bit variable. Regards Graham [*] https://db.debian.org/machines.cgi?sortby=purpose=dsc
Bug#980809: Matrix package default tolerance on s390x (Re: Bug#980809: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x)
Hi Martin,
On 17 February 2021 at 10:43, Martin Maechler wrote:
| Dear Dirk,
| I'm in vacations right now ...
Ah, nice.
| 1) I think I might simply disable the check *on* that platform .. or
| platforms with similar characteristics.
Sounds good to me.
| Can you send me the outputs of so I can take some of the entries to
| determine I'm on the platform?
|
| sessionInfo()
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: s390x-ibm-linux-gnu (64-bit)
Running under: Debian GNU/Linux bullseye/sid
Matrix products: default
BLAS: /usr/lib/s390x-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/s390x-linux-gnu/lapack/liblapack.so.3.9.0
locale:
[1] C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_4.0.3
>
| str(.Machine, digits=10)
> str(.Machine, digits=10)
List of 28
$ double.eps : num 2.220446049e-16
$ double.neg.eps : num 1.110223025e-16
$ double.xmin : num 2.225073859e-308
$ double.xmax : num 1.797693135e+308
$ double.base : int 2
$ double.digits: int 53
$ double.rounding : int 5
$ double.guard : int 0
$ double.ulp.digits: int -52
$ double.neg.ulp.digits: int -53
$ double.exponent : int 11
$ double.min.exp : int -1022
$ double.max.exp : int 1024
$ integer.max : int 2147483647
$ sizeof.long : int 8
$ sizeof.longlong : int 8
$ sizeof.longdouble: int 16
$ sizeof.pointer : int 8
$ longdouble.eps : num 1.925929944e-34
$ longdouble.neg.eps : num 9.629649722e-35
$ longdouble.digits: int 113
$ longdouble.rounding : int 5
$ longdouble.guard : int 0
$ longdouble.ulp.digits: int -112
$ longdouble.neg.ulp.digits: int -113
$ longdouble.exponent : int 15
$ longdouble.min.exp : int -16382
$ longdouble.max.exp : int 16384
>
| capabilities()
> capabilities()
jpeg pngtiff tcltk X11aqua
TRUETRUETRUETRUE FALSE FALSE
http/ftp sockets libxmlfifo cledit iconv
TRUETRUETRUETRUETRUETRUE
NLS profmem cairo ICU long.double libcurl
TRUETRUETRUETRUETRUETRUE
>
| .Platform
> str(.Platform)
List of 8
$ OS.type : chr "unix"
$ file.sep : chr "/"
$ dynlib.ext: chr ".so"
$ GUI : chr "X11"
$ endian: chr "big"
$ pkgType : chr "source"
$ path.sep : chr ":"
$ r_arch: chr ""
>
| 2) Can you send me the result of rankMatrix(Z, method="qr")
| i.e. `rnkZ.` from the above example ?
> library(Matrix)
> Z. <- readRDS(system.file("external", "Z_NA_rnk.rds", package="Matrix"))
> rnkZ. <- rankMatrix(Z., method = "qr")
> rnkZ.
[1] 724
attr(,"method")
[1] "qr.R"
attr(,"useGrad")
[1] FALSE
attr(,"tol")
[1] 3.126388e-13
>
Graham (in subsequent message) makes a good point about big endian. Maybe we
need a slightly wider net.
It could, as always, of course be something else but related. I could replace
the default BLAS with OpenBLAS etc but ex ante the default BLAS should work :-/
But as you hinted, if we skip that one test on this one architecture it
passes. I used a very minimal diff (below) and quickly rebuilt (by skipping
vignettes as I currently do not have texlive installed on the s390x (but
could add it)).
(sid_s390x-dchroot)edd@zelenka:/tmp/downloaded_packages$ diff -u
Matrix{,.orig}/tests/factorizing.R
--- Matrix/tests/factorizing.R 2021-02-17 13:01:06.511757335 +
+++ Matrix.orig/tests/factorizing.R 2021-01-04 20:32:35.0 +
@@ -472,7 +472,6 @@
})
options(oo)
-if (Sys.info()[["machine"]] != "s390x") {
## problematic rank deficient rankMatrix() case -- only seen in large cases ??
Z. <- readRDS(system.file("external", "Z_NA_rnk.rds", package="Matrix"))
tools::assertWarning(rnkZ. <- rankMatrix(Z., method = "qr")) # gave errors
@@ -482,7 +481,6 @@
oo <- options(warn=2)# no warnings allowed from here
di <- diag(qrZ.@R)
stopifnot(is.na(rnkZ.), is(qrZ, "sparseQR"), is.na(rnk2), anyNA(di))
-}
## The above bug fix was partly wrongly extended to dense matrices for "qr.R":
x <- cbind(1, rep(0:9, 18))
(sid_s390x-dchroot)edd@zelenka:/tmp/downloaded_packages$
(Obviously the if above could be indentented and all that.)
Graham: What does that bigendian box say for Sys.info()[["machine"]] ?
Should we test for endianness instead? And then maybe try to roll up to the
cause of the difference?
Dirk
| Best, Martin
|
| On Mon, Feb 15, 2021 at 5:12 PM Dirk Eddelbuettel wrote:
| >
| >
| > Martin,
| >
| > I have this long-running bug report against Matrix, triggered by glmmTMB on
| > s390x. Graham has been chasing this patiently and we
Bug#980809: Matrix package default tolerance on s390x (Re: Bug#980809: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x)
Hi Martin On Wed, 17 Feb 2021 at 11:45, Martin Maechler wrote: > 1) I think I might simply disable the check *on* that platform .. or > platforms with similar characteristics. I was able to reproduce the issue on our big-endian PowerPC architecture as well, so maybe you can use the following in your check: > .Platform$endian [1] "big" Regards Graham
Bug#980809: Matrix package default tolerance on s390x (Re: Bug#980809: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x)
Dear Dirk,
I'm in vacations right now ...
1) I think I might simply disable the check *on* that platform .. or
platforms with similar characteristics.
Can you send me the outputs of so I can take some of the entries to
determine I'm on the platform?
sessionInfo()
str(.Machine, digits=10)
capabilities()
.Platform
2) Can you send me the result of rankMatrix(Z, method="qr")
i.e. `rnkZ.` from the above example ?
Best, Martin
On Mon, Feb 15, 2021 at 5:12 PM Dirk Eddelbuettel wrote:
>
>
> Martin,
>
> I have this long-running bug report against Matrix, triggered by glmmTMB on
> s390x. Graham has been chasing this patiently and we are now at the level of
> checking on a shell account on the appropriate hardware.
>
> We validated that everything does in fact "still break" with CRAN-current
> packages of everything (and R 4.0.3, I guess by tomorrow we'd have 4.0.4).
>
> glmmTMB does indeed fail in an example -- but so does Matrix 1.3-2 when I run
>
> _R_CHECK_FORCE_SUGGESTS_=false R CMD check \
> --no-manual --no-vignettes Matrix_1.3-2.tar.gz
>
> and I see
>
> [...]
> Running 'factorizing.R'
> ERROR
> Running the tests in 'tests/factorizing.R' failed.
> Last 13 lines of output:
> + inherits(qrZ, "sparseQR")
> + inherits(Rz, "sparseMatrix")
> + isTriangular(Rz)
> + isDiagonal(Rz) # even though formally a "dtCMatrix"
> + qr2rankMatrix(qrZ, do.warn=FALSE) == 6
> + })
> > options(oo)
> >
> > ## problematic rank deficient rankMatrix() case -- only seen in large
> cases ??
> > Z. <- readRDS(system.file("external", "Z_NA_rnk.rds", package="Matrix"))
> > tools::assertWarning(rnkZ. <- rankMatrix(Z., method = "qr")) # gave errors
> Error in assertCondition(expr, classes, .exprString = d.expr) :
> Failed to get warning in evaluating rnkZ. <- rankMatrix(Z., method ...
> Calls: -> assertCondition
> Execution halted
> * checking for unstated dependencies in vignettes ... OK
>
> Can you advise if I should dive into a particular routine or comparison to
> see what is up here?
>
> Best, Dirk
>
> --
> https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
>
--
Martin Maechler,
Seminar fuer Statistik, ETH Zurich
Bug#980809: Matrix package default tolerance on s390x (Re: Bug#980809: rmatrix: breaks autopkgtest of r-cran-glmmtmb on s390x)
Martin,
I have this long-running bug report against Matrix, triggered by glmmTMB on
s390x. Graham has been chasing this patiently and we are now at the level of
checking on a shell account on the appropriate hardware.
We validated that everything does in fact "still break" with CRAN-current
packages of everything (and R 4.0.3, I guess by tomorrow we'd have 4.0.4).
glmmTMB does indeed fail in an example -- but so does Matrix 1.3-2 when I run
_R_CHECK_FORCE_SUGGESTS_=false R CMD check \
--no-manual --no-vignettes Matrix_1.3-2.tar.gz
and I see
[...]
Running 'factorizing.R'
ERROR
Running the tests in 'tests/factorizing.R' failed.
Last 13 lines of output:
+ inherits(qrZ, "sparseQR")
+ inherits(Rz, "sparseMatrix")
+ isTriangular(Rz)
+ isDiagonal(Rz) # even though formally a "dtCMatrix"
+ qr2rankMatrix(qrZ, do.warn=FALSE) == 6
+ })
> options(oo)
>
> ## problematic rank deficient rankMatrix() case -- only seen in large cases
??
> Z. <- readRDS(system.file("external", "Z_NA_rnk.rds", package="Matrix"))
> tools::assertWarning(rnkZ. <- rankMatrix(Z., method = "qr")) # gave errors
Error in assertCondition(expr, classes, .exprString = d.expr) :
Failed to get warning in evaluating rnkZ. <- rankMatrix(Z., method ...
Calls: -> assertCondition
Execution halted
* checking for unstated dependencies in vignettes ... OK
Can you advise if I should dive into a particular routine or comparison to
see what is up here?
Best, Dirk
--
https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
