Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[Duncan Murdoch]

On 10/01/2023 4:07 p.m., Sebastian Meyer wrote:

Am 10.01.23 um 21:28 schrieb Duncan Murdoch:

On 10/01/2023 2:05 p.m., Ivan Krylov wrote:

On Tue, 10 Jan 2023 16:27:53 +
RICHET Yann  wrote:


In facts, 10 threads are asked by armadillo for some LinAlg, which
backs to two threads as warned.


I think you're right about your tests de-facto using two threads, but
it might be a good idea to _default_ to up to two threads in tests and
examples. This is especially valuable for third-party developers who
have to mass-test packages (one of which could be rlibkriging) in
parallel.


- is there any reason that could explain that fedora-*-devel is so
slow for this package or compilation of Rcpp/testthat ?


Compilation time is definitely not the reason. Something in tests/*
actually runs for 30 minutes by itself.


- is there any chance that I can get a deeper log of what happened ?


If you split your tests into separate files under tests/*.R instead of
using a single tests/testthat.R calling the rest of the tests, R will
be able to show you the individual test file that hung and maybe the
line where it happened. (Also, you'll get per-file timing.) But that
is potentially a huge investment: you may have to rewrite the tests to
work outside the testthat harness, and you'd also have to prepare
another CRAN submission just to have those tests run. It's also against
CRAN policy to knowingly submit a package with unfixed ERRORs.

(Currently, R can only tell you that the tests hung in the
test_check('rlibkriging') call in the tests/testthat.R, which isn't
precise enough.)


You can specify a different "reporter" in the test_check() call, and it
will print more useful information.  I don't think there's a perfect
one, but

    test_check('rlibkriging', reporter = "progress")

should at least show you the tests that finished running before the
timeout.


I had similar problems with testthat and timeouts when mass-checking
packages on patched R versions. My notes say


## testthat's 'LocationReporter' does cat() after each expectation
## so we can see results even if timeout is reached
options(testthat.default_check_reporter = c("Location", "Check"))


The help("LocationReporter") says: "This reporter simply prints the
location of every expectation and error. This is useful if you're trying
to figure out the source of a segfault, or you want to figure out which
code triggers a C/C++ breakpoint"

HTH!


Yes, that looks like it would pin down the location of the problem. 
Here's some sample output from it:


  Running ‘testthat.R’ [41s/42s]
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  Start test: can use constructed calls in verify_output() (#945)
'test-verify-output.R:55' [success]
  End test: can use constructed calls in verify_output() (#945)

  Start test: verify_output() doesn't use cli unicode by default
'test-verify-output.R:65' [success]
'test-verify-output.R:73' [success]
  End test: verify_output() doesn't use cli unicode by default

  Start test: verify_output() handles carriage return
'test-verify-output.R:82' [success]
  End test: verify_output() handles carriage return

  Error: Test failures
  Execution halted

One other thing:  you enabled this by using

  options(testthat.default_check_reporter = c("Location", "Check"))

before running the tests; the package writer could do the same thing by 
using


  test_check('rlibkriging', reporter = c("Location", "Check"))

Duncan Murdoch

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[Sebastian Meyer]

Am 10.01.23 um 21:28 schrieb Duncan Murdoch:

On 10/01/2023 2:05 p.m., Ivan Krylov wrote:

On Tue, 10 Jan 2023 16:27:53 +
RICHET Yann  wrote:


In facts, 10 threads are asked by armadillo for some LinAlg, which
backs to two threads as warned.


I think you're right about your tests de-facto using two threads, but
it might be a good idea to _default_ to up to two threads in tests and
examples. This is especially valuable for third-party developers who
have to mass-test packages (one of which could be rlibkriging) in
parallel.


- is there any reason that could explain that fedora-*-devel is so
slow for this package or compilation of Rcpp/testthat ?


Compilation time is definitely not the reason. Something in tests/*
actually runs for 30 minutes by itself.


- is there any chance that I can get a deeper log of what happened ?


If you split your tests into separate files under tests/*.R instead of
using a single tests/testthat.R calling the rest of the tests, R will
be able to show you the individual test file that hung and maybe the
line where it happened. (Also, you'll get per-file timing.) But that
is potentially a huge investment: you may have to rewrite the tests to
work outside the testthat harness, and you'd also have to prepare
another CRAN submission just to have those tests run. It's also against
CRAN policy to knowingly submit a package with unfixed ERRORs.

(Currently, R can only tell you that the tests hung in the
test_check('rlibkriging') call in the tests/testthat.R, which isn't
precise enough.)


You can specify a different "reporter" in the test_check() call, and it 
will print more useful information.  I don't think there's a perfect 
one, but


   test_check('rlibkriging', reporter = "progress")

should at least show you the tests that finished running before the 
timeout.  


I had similar problems with testthat and timeouts when mass-checking 
packages on patched R versions. My notes say



## testthat's 'LocationReporter' does cat() after each expectation
## so we can see results even if timeout is reached
options(testthat.default_check_reporter = c("Location", "Check"))


The help("LocationReporter") says: "This reporter simply prints the 
location of every expectation and error. This is useful if you're trying 
to figure out the source of a segfault, or you want to figure out which 
code triggers a C/C++ breakpoint"


HTH!

Sebastian Meyer

You could possibly also write your own custom reporter that 
could give timings for each of the tests as they run, but the documents 
for how to do that don't seem to exist.  Maybe someone else has done it?


Duncan Murdoch

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel


__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[Duncan Murdoch]

On 10/01/2023 2:05 p.m., Ivan Krylov wrote:

On Tue, 10 Jan 2023 16:27:53 +
RICHET Yann  wrote:


In facts, 10 threads are asked by armadillo for some LinAlg, which
backs to two threads as warned.


I think you're right about your tests de-facto using two threads, but
it might be a good idea to _default_ to up to two threads in tests and
examples. This is especially valuable for third-party developers who
have to mass-test packages (one of which could be rlibkriging) in
parallel.


- is there any reason that could explain that fedora-*-devel is so
slow for this package or compilation of Rcpp/testthat ?


Compilation time is definitely not the reason. Something in tests/*
actually runs for 30 minutes by itself.


- is there any chance that I can get a deeper log of what happened ?


If you split your tests into separate files under tests/*.R instead of
using a single tests/testthat.R calling the rest of the tests, R will
be able to show you the individual test file that hung and maybe the
line where it happened. (Also, you'll get per-file timing.) But that
is potentially a huge investment: you may have to rewrite the tests to
work outside the testthat harness, and you'd also have to prepare
another CRAN submission just to have those tests run. It's also against
CRAN policy to knowingly submit a package with unfixed ERRORs.

(Currently, R can only tell you that the tests hung in the
test_check('rlibkriging') call in the tests/testthat.R, which isn't
precise enough.)


You can specify a different "reporter" in the test_check() call, and it 
will print more useful information.  I don't think there's a perfect 
one, but


  test_check('rlibkriging', reporter = "progress")

should at least show you the tests that finished running before the 
timeout.  You could possibly also write your own custom reporter that 
could give timings for each of the tests as they run, but the documents 
for how to do that don't seem to exist.  Maybe someone else has done it?


Duncan Murdoch

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[Ivan Krylov]

On Tue, 10 Jan 2023 16:27:53 +
RICHET Yann  wrote:

> In facts, 10 threads are asked by armadillo for some LinAlg, which
> backs to two threads as warned. 

I think you're right about your tests de-facto using two threads, but
it might be a good idea to _default_ to up to two threads in tests and
examples. This is especially valuable for third-party developers who
have to mass-test packages (one of which could be rlibkriging) in
parallel.

> - is there any reason that could explain that fedora-*-devel is so
> slow for this package or compilation of Rcpp/testthat ?

Compilation time is definitely not the reason. Something in tests/*
actually runs for 30 minutes by itself.

> - is there any chance that I can get a deeper log of what happened ?

If you split your tests into separate files under tests/*.R instead of
using a single tests/testthat.R calling the rest of the tests, R will
be able to show you the individual test file that hung and maybe the
line where it happened. (Also, you'll get per-file timing.) But that
is potentially a huge investment: you may have to rewrite the tests to
work outside the testthat harness, and you'd also have to prepare
another CRAN submission just to have those tests run. It's also against
CRAN policy to knowingly submit a package with unfixed ERRORs.

(Currently, R can only tell you that the tests hung in the
test_check('rlibkriging') call in the tests/testthat.R, which isn't
precise enough.)

These lines look a bit scary:

tests/testthat/bench-KrigingFit.R:

pack=list.files(file.path("bindings","R"),pattern = ".tar.gz",full.names = T) 
install.packages(pack,repos=NULL)

tests/testthat/notest-LinearRegression.R:

install.packages(pkgs="rlibkriging_0.7-2.tgz", type="source", repos=NULL)
library(rlibkriging)

I don't think that install.packages does anything besides raising a
warning and letting the test continue (there doesn't seem to be any
.tar.gz files inside the package on CRAN, so installation fails), but
it's probably not a good idea to install packages during testing [*].

This is also not quite right but harmless, because rlibkriging is
already loaded:

library(rlibkriging, lib.loc="bindings/R/Rlibs")

I'm afraid I don't know how to reproduce the timeouts you're seeing on
the CRAN Fedora machine.

-- 
Best regards,
Ivan

[*] I once wrote a test that launched multiple child R processes,
created a temporary library in each of them and installed the package
there. The test was designed to only run on the developer's machine in
order to test file format compatibility between different versions of
R. I ended up moving the test away from tests/*.R in order to make sure
it won't run by accident.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[Serguei Sokol via R-devel]

Looks like there is a kind of misunderstanding...

Le 10/01/2023 à 17:27, RICHET Yann a écrit :
> Thank you for your answer.
> In facts, 10 threads are asked by armadillo for some LinAlg, which backs to 
> two threads as warned. But I cannot imagine this costs so much time just for 
> that...
Excessive thread number is a problem per se. I did not say that it was 
responsible for excessive test time.
It's up to you to configure armadillo to use no more then 2 threads when 
run on CRAN. May be, setting environment variable OPENBLAS_NUM_THREADS=2 
could help.

>
> A deeper analysis of time spent seems to point that a large time was mainly 
> spent on testthat and Rcpp dependencies compilation...
Normally, compilation time is not accounted in the quota of 15 min 
dedicated to tests.
I would just focus on reducing this time, e.g. by running tests on 
smaller cases or skipping time-consuming tests conditionally when on CRAN.

Serguei.
>   But other recent packages depending on these also are not spending so much 
> time.
>
> CRAN team, as I failed to reproduce the issue with rhub dockers,
> - is there any reason that could explain that fedora-*-devel is so slow for 
> this package or compilation of Rcpp/testthat ?
> - is our slapack a possible source of... anything ?
> - are we the only package which embeds "standard armadillo" and tries to deal 
> with simple precision lapack issues on fedora ?
> - is there any chance that I can get a deeper log of what happened ?
>
> Best regards,
>
> Yann
>
> -Message d'origine-
> De : Serguei Sokol  
> Envoyé : mardi 10 janvier 2023 11:41
> À : RICHET Yann;R-devel@r-project.org
> Cc : Pascal Havé
> Objet : Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack
>
> Le 10/01/2023 à 11:37, Serguei Sokol a écrit :
>> Le 10/01/2023 à 10:44, RICHET Yann a écrit :
>>> Dear R-devel people,
>>>
>>> We are working to submit a package which is mainly a binding over a
>>> C++ lib (https://github.com/libKriging) using armadillo.
>>> It is _not_ a standard RcppArmadillo package, because we also had to
>>> provide a python binding... so there is a huge layer of cmake &
>>> scripting to make it work with a standard armadillo (but using same
>>> version that RcppArmadillo).
>>> It seems now working with almost all CRAN targets, but a problem
>>> remains with fedora (clang & gcc) devel.
>>>
>>> Our issue is that the rhub docker is not well sync with the CRAN
>>> fedora-clang-devel config:
>>> - failing on CRAN (without clear reason):
>>> https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-fedora-c
>>> lang/rlibkriging-00check.html
>> I see  2 candidates for  reasons of failing on CRAN:
>>   1. test time is 30 min while 15 min are allowed;
>>   2. your code try to get 30 threads
> Oops, it was 10 threads not 30 but it does not change the nature of a 
> possible issue.
>
>> while CRAN limits them to 2;
>>
>> Try to make your tests shorter ( < 15 min) on 2 threads. May be it
>> will help.
>>
>> Best,
>> Serguei.
>>
>>> - passing on rhub:
>>> https://builder.r-hub.io/status/rlibkriging_0.7-3.tar.gz-20f7dc756272
>>> 6497af7c678ab41f4dea
>>>
>>> So we cannot investigate and fix the problem.
>>>
>>> Note that we did quite strange things with the fedora platforms:
>>> include explicitely slapack to provide simple precision for our
>>> (vanilla) armadillo...
>>>
>>> Do you have any idea, or even known problem in mind, that could be
>>> related to this ?
>>>
>>> Best regards,
>>>
>>> --
>>> Dr. Yann Richet
>>> Institute for Radiological Protection and Nuclear Safety
>>> (https://www.irsn.fr),
>>>     Department of Characterization of Natural Unexpected Events and
>>> Sites Office : +33 1 58 35 88 84
>>>
>>> __
>>> R-devel@r-project.org  mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[RICHET Yann]

Thank you for your answer.
In facts, 10 threads are asked by armadillo for some LinAlg, which backs to two 
threads as warned. But I cannot imagine this costs so much time just for that...

A deeper analysis of time spent seems to point that a large time was mainly 
spent on testthat and Rcpp dependencies compilation... But other recent 
packages depending on these also are not spending so much time.

CRAN team, as I failed to reproduce the issue with rhub dockers, 
- is there any reason that could explain that fedora-*-devel is so slow for 
this package or compilation of Rcpp/testthat ?
- is our slapack a possible source of... anything ?
- are we the only package which embeds "standard armadillo" and tries to deal 
with simple precision lapack issues on fedora ?
- is there any chance that I can get a deeper log of what happened ?

Best regards,

Yann

-Message d'origine-
De : Serguei Sokol  
Envoyé : mardi 10 janvier 2023 11:41
À : RICHET Yann ; R-devel@r-project.org
Cc : Pascal Havé 
Objet : Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

Le 10/01/2023 à 11:37, Serguei Sokol a écrit :
> Le 10/01/2023 à 10:44, RICHET Yann a écrit :
>> Dear R-devel people,
>>
>> We are working to submit a package which is mainly a binding over a
>> C++ lib (https://github.com/libKriging) using armadillo.
>> It is _not_ a standard RcppArmadillo package, because we also had to 
>> provide a python binding... so there is a huge layer of cmake & 
>> scripting to make it work with a standard armadillo (but using same 
>> version that RcppArmadillo).
>> It seems now working with almost all CRAN targets, but a problem 
>> remains with fedora (clang & gcc) devel.
>>
>> Our issue is that the rhub docker is not well sync with the CRAN 
>> fedora-clang-devel config:
>> - failing on CRAN (without clear reason): 
>> https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-fedora-c
>> lang/rlibkriging-00check.html
> I see  2 candidates for  reasons of failing on CRAN:
>  1. test time is 30 min while 15 min are allowed;
>  2. your code try to get 30 threads
Oops, it was 10 threads not 30 but it does not change the nature of a possible 
issue.

> while CRAN limits them to 2;
>
> Try to make your tests shorter ( < 15 min) on 2 threads. May be it 
> will help.
>
> Best,
> Serguei.
>
>> - passing on rhub: 
>> https://builder.r-hub.io/status/rlibkriging_0.7-3.tar.gz-20f7dc756272
>> 6497af7c678ab41f4dea
>>
>> So we cannot investigate and fix the problem.
>>
>> Note that we did quite strange things with the fedora platforms: 
>> include explicitely slapack to provide simple precision for our
>> (vanilla) armadillo...
>>
>> Do you have any idea, or even known problem in mind, that could be 
>> related to this ?
>>
>> Best regards,
>>
>> --
>> Dr. Yann Richet
>> Institute for Radiological Protection and Nuclear Safety 
>> (https://www.irsn.fr),
>>    Department of Characterization of Natural Unexpected Events and 
>> Sites Office : +33 1 58 35 88 84
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>


--
Serguei Sokol
Ingenieur de recherche INRAE

Cellule Mathématiques
TBI, INSA/INRAE UMR 792, INSA/CNRS UMR 5504
135 Avenue de Rangueil
31077 Toulouse Cedex 04

tel: +33 5 61 55 98 49
email: so...@insa-toulouse.fr
http://www.toulouse-biotechnology-institute.fr/en/technology_platforms/mathematics-cell.html

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Bioc-devel] ExperimentHub: bad restore file magic number (file may be corrupted)

[Kern, Lori]

It looks like the download got corrupted.  I reset the cache for that EH number 
and it should clear up on the next build.




Lori Shepherd - Kern

Bioconductor Core Team

Roswell Park Comprehensive Cancer Center

Department of Biostatistics & Bioinformatics

Elm & Carlton Streets

Buffalo, New York 14263


From: Bioc-devel  on behalf of Michael Love 

Sent: Tuesday, January 10, 2023 9:40 AM
To: bioc-devel 
Subject: [Bioc-devel] ExperimentHub: bad restore file magic number (file may be 
corrupted)

In the latest release, one of my files in ExperimentHub is giving the
following error on Bioc machines. I haven't touched these files since
they were submitted many years ago to EHub.

http://secure-web.cisco.com/12y5s79xt-pmTPP6fzsYkMo0aALvDKgR-cUjxZcIL7QocYem0ml4eV-pkh4oqQlVq8ZteIbXcu42Ee2A76ClYhjiLHejdzRpRUWm2tCUEwNc6KBS1AiYDSfsOBC3dQ5mK3b3HgrhfvfXRVwW9TNr64PoANtPMDKwcwCG6aYF-RqHQHmvHDx5W2NluvaWlSV8u3wLL4iE0yfBocZDajrHg02fNfdqKkIbJF_iMnVNrLL950twq0FdVn9plKReI2-Eb1eI4a4OxMYO82sikXLqQmnE3w6f5KD--82FtxHaPjkXqyQSvQbRXLuH3ZBS2ussl/http%3A%2F%2Fbioconductor.org%2FcheckResults%2Frelease%2Fdata-experiment-LATEST%2FalpineData%2Fnebbiolo2-buildsrc.html

...
--- re-building �alpineData.Rmd� using knitr
Quitting from lines 18-22 (alpineData.Rmd)
Error: processing vignette 'alpineData.Rmd' failed with diagnostics:
failed to load resource
  name: EH167
  title: ERR188088
  reason: error in evaluating the argument 'x' in selecting a method
for function 'get': bad restore file magic number (file may be
corrupted) -- no data loaded
--- failed re-building �alpineData.Rmd�

On my machine I can load this resource:

 > library(alpineData)
!> ERR188088()
 snapshotDate(): 2022-10-31
 see ?alpineData and browseVignettes('alpineData') for documentation
 loading from cache
 GAlignmentPairs object with 28576 pairs, strandMode=1, and 0 metadata columns:
   seqnames strand   :  ranges  --  ranges
   : --   
   [1]1  -   : 108565979-108566053  -- 108565846-108565920
   [2]1  -   : 108573341-108573415  -- 108573234-108573308
   [3]1  +   : 108581087-108581161  -- 108581239-108581313
   [4]1  +   : 108601105-108601179  -- 108601196-108601270
   [5]1  -   : 108603628-108603701  -- 108603540-108603614
   ...  ...... ... ... ... ...
   [28572]X  -   : 119791266-119791340  -- 119791130-119791204
   [28573]X  -   : 119791431-119791505  -- 119791358-119791432
   [28574]X  -   : 119791593-119791667  -- 119786691-119789940
   [28575]X  -   : 119791629-119791703  -- 119789951-119791587
   [28576]X  -   : 119791637-119791711  -- 119789976-119791612
   ---
   seqinfo: 194 sequences from an unspecified genome

> eh <- ExperimentHub()

!> eh[["EH167"]]
 see ?alpineData and browseVignettes('alpineData') for documentation
 loading from cache
 GAlignmentPairs object with 28576 pairs, strandMode=1, and 0 metadata columns:
   seqnames strand   :  ranges  --  ranges
   : --   
   [1]1  -   : 108565979-108566053  -- 108565846-108565920
   [2]1  -   : 108573341-108573415  -- 108573234-108573308
   [3]1  +   : 108581087-108581161  -- 108581239-108581313
   [4]1  +   : 108601105-108601179  -- 108601196-108601270
   [5]1  -   : 108603628-108603701  -- 108603540-108603614
   ...  ...... ... ... ... ...
   [28572]X  -   : 119791266-119791340  -- 119791130-119791204
   [28573]X  -   : 119791431-119791505  -- 119791358-119791432
   [28574]X  -   : 119791593-119791667  -- 119786691-119789940
   [28575]X  -   : 119791629-119791703  -- 119789951-119791587
   [28576]X  -   : 119791637-119791711  -- 119789976-119791612
   ---
   seqinfo: 194 sequences from an unspecified genome

!> sessionInfo()
 R version 4.2.2 (2022-10-31)
 Platform: aarch64-apple-darwin20 (64-bit)
 Running under: macOS Ventura 13.0.1

 Matrix products: default
 BLAS:   
/Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRblas.0.dylib
 LAPACK: 
/Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

 locale:
 [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

 attached base packages:
 [1] stats4stats graphics  grDevices datasets  utils methods   base

 other attached packages:
  [1] rtracklayer_1.58.0  alpineData_1.24.0
GenomicAlignments_1.34.0
  [4] Rsamtools_2.14.0Biostrings_2.66.0   XVector_0.38.0
  [7] SummarizedExperiment_1.28.0 Biobase_2.58.0
MatrixGenerics_1.10.0
 [10] matrixStats_0.63.0  GenomicRanges_1.50.1
GenomeInfoDb_1.34.4
 [13] IRanges_2.32.0

[Bioc-devel] ExperimentHub: bad restore file magic number (file may be corrupted)

[Michael Love]

In the latest release, one of my files in ExperimentHub is giving the
following error on Bioc machines. I haven't touched these files since
they were submitted many years ago to EHub.

http://bioconductor.org/checkResults/release/data-experiment-LATEST/alpineData/nebbiolo2-buildsrc.html

...
--- re-building ‘alpineData.Rmd’ using knitr
Quitting from lines 18-22 (alpineData.Rmd)
Error: processing vignette 'alpineData.Rmd' failed with diagnostics:
failed to load resource
  name: EH167
  title: ERR188088
  reason: error in evaluating the argument 'x' in selecting a method
for function 'get': bad restore file magic number (file may be
corrupted) -- no data loaded
--- failed re-building ‘alpineData.Rmd’

On my machine I can load this resource:

 > library(alpineData)
!> ERR188088()
 snapshotDate(): 2022-10-31
 see ?alpineData and browseVignettes('alpineData') for documentation
 loading from cache
 GAlignmentPairs object with 28576 pairs, strandMode=1, and 0 metadata columns:
   seqnames strand   :  ranges  --  ranges
   : --   
   [1]1  -   : 108565979-108566053  -- 108565846-108565920
   [2]1  -   : 108573341-108573415  -- 108573234-108573308
   [3]1  +   : 108581087-108581161  -- 108581239-108581313
   [4]1  +   : 108601105-108601179  -- 108601196-108601270
   [5]1  -   : 108603628-108603701  -- 108603540-108603614
   ...  ...... ... ... ... ...
   [28572]X  -   : 119791266-119791340  -- 119791130-119791204
   [28573]X  -   : 119791431-119791505  -- 119791358-119791432
   [28574]X  -   : 119791593-119791667  -- 119786691-119789940
   [28575]X  -   : 119791629-119791703  -- 119789951-119791587
   [28576]X  -   : 119791637-119791711  -- 119789976-119791612
   ---
   seqinfo: 194 sequences from an unspecified genome

> eh <- ExperimentHub()

!> eh[["EH167"]]
 see ?alpineData and browseVignettes('alpineData') for documentation
 loading from cache
 GAlignmentPairs object with 28576 pairs, strandMode=1, and 0 metadata columns:
   seqnames strand   :  ranges  --  ranges
   : --   
   [1]1  -   : 108565979-108566053  -- 108565846-108565920
   [2]1  -   : 108573341-108573415  -- 108573234-108573308
   [3]1  +   : 108581087-108581161  -- 108581239-108581313
   [4]1  +   : 108601105-108601179  -- 108601196-108601270
   [5]1  -   : 108603628-108603701  -- 108603540-108603614
   ...  ...... ... ... ... ...
   [28572]X  -   : 119791266-119791340  -- 119791130-119791204
   [28573]X  -   : 119791431-119791505  -- 119791358-119791432
   [28574]X  -   : 119791593-119791667  -- 119786691-119789940
   [28575]X  -   : 119791629-119791703  -- 119789951-119791587
   [28576]X  -   : 119791637-119791711  -- 119789976-119791612
   ---
   seqinfo: 194 sequences from an unspecified genome

!> sessionInfo()
 R version 4.2.2 (2022-10-31)
 Platform: aarch64-apple-darwin20 (64-bit)
 Running under: macOS Ventura 13.0.1

 Matrix products: default
 BLAS:   
/Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRblas.0.dylib
 LAPACK: 
/Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

 locale:
 [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

 attached base packages:
 [1] stats4stats graphics  grDevices datasets  utils methods   base

 other attached packages:
  [1] rtracklayer_1.58.0  alpineData_1.24.0
GenomicAlignments_1.34.0
  [4] Rsamtools_2.14.0Biostrings_2.66.0   XVector_0.38.0
  [7] SummarizedExperiment_1.28.0 Biobase_2.58.0
MatrixGenerics_1.10.0
 [10] matrixStats_0.63.0  GenomicRanges_1.50.1
GenomeInfoDb_1.34.4
 [13] IRanges_2.32.0  S4Vectors_0.36.0
ExperimentHub_2.6.0
 [16] AnnotationHub_3.6.0 BiocFileCache_2.6.0 dbplyr_2.2.1
 [19] BiocGenerics_0.44.0 pkgdown_2.0.6   testthat_3.1.5
 [22] rmarkdown_2.18  devtools_2.4.5  usethis_2.1.6

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel

Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[Serguei Sokol via R-devel]

Le 10/01/2023 à 11:37, Serguei Sokol a écrit :

Le 10/01/2023 à 10:44, RICHET Yann a écrit :

Dear R-devel people,

We are working to submit a package which is mainly a binding over a 
C++ lib (https://github.com/libKriging) using armadillo.
It is _not_ a standard RcppArmadillo package, because we also had to 
provide a python binding... so there is a huge layer of cmake & 
scripting to make it work with a standard armadillo (but using same 
version that RcppArmadillo).
It seems now working with almost all CRAN targets, but a problem 
remains with fedora (clang & gcc) devel.


Our issue is that the rhub docker is not well sync with the CRAN 
fedora-clang-devel config:
- failing on CRAN (without clear reason): 
https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-fedora-clang/rlibkriging-00check.html

I see  2 candidates for  reasons of failing on CRAN:
 1. test time is 30 min while 15 min are allowed;
 2. your code try to get 30 threads
Oops, it was 10 threads not 30 but it does not change the nature of a 
possible issue.



while CRAN limits them to 2;

Try to make your tests shorter ( < 15 min) on 2 threads. May be it 
will help.


Best,
Serguei.

- passing on rhub: 
https://builder.r-hub.io/status/rlibkriging_0.7-3.tar.gz-20f7dc7562726497af7c678ab41f4dea


So we cannot investigate and fix the problem.

Note that we did quite strange things with the fedora platforms: 
include explicitely slapack to provide simple precision for our 
(vanilla) armadillo...


Do you have any idea, or even known problem in mind, that could be 
related to this ?


Best regards,

--
Dr. Yann Richet
Institute for Radiological Protection and Nuclear Safety 
(https://www.irsn.fr),

   Department of Characterization of Natural Unexpected Events and Sites
Office : +33 1 58 35 88 84

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel






--
Serguei Sokol
Ingenieur de recherche INRAE

Cellule Mathématiques
TBI, INSA/INRAE UMR 792, INSA/CNRS UMR 5504
135 Avenue de Rangueil
31077 Toulouse Cedex 04

tel: +33 5 61 55 98 49
email: so...@insa-toulouse.fr
http://www.toulouse-biotechnology-institute.fr/en/technology_platforms/mathematics-cell.html

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[Serguei Sokol via R-devel]

Le 10/01/2023 à 10:44, RICHET Yann a écrit :

Dear R-devel people,

We are working to submit a package which is mainly a binding over a C++ lib 
(https://github.com/libKriging) using armadillo.
It is _not_ a standard RcppArmadillo package, because we also had to provide a 
python binding... so there is a huge layer of cmake & scripting to make it work 
with a standard armadillo (but using same version that RcppArmadillo).
It seems now working with almost all CRAN targets, but a problem remains with 
fedora (clang & gcc) devel.

Our issue is that the rhub docker is not well sync with the CRAN 
fedora-clang-devel config:
- failing on CRAN (without clear reason): 
https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-fedora-clang/rlibkriging-00check.html

I see  2 candidates for  reasons of failing on CRAN:
 1. test time is 30 min while 15 min are allowed;
 2. your code try to get 30 threads while CRAN limits them to 2;

Try to make your tests shorter ( < 15 min) on 2 threads. May be it will 
help.


Best,
Serguei.


- passing on rhub: 
https://builder.r-hub.io/status/rlibkriging_0.7-3.tar.gz-20f7dc7562726497af7c678ab41f4dea

So we cannot investigate and fix the problem.

Note that we did quite strange things with the fedora platforms: include 
explicitely slapack to provide simple precision for our (vanilla) armadillo...

Do you have any idea, or even known problem in mind, that could be related to 
this ?

Best regards,

--
Dr. Yann Richet
Institute for Radiological Protection and Nuclear Safety (https://www.irsn.fr),
   Department of Characterization of Natural Unexpected Events and Sites
Office : +33 1 58 35 88 84

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Serguei Sokol
Ingenieur de recherche INRAE

Cellule Mathématiques
TBI, INSA/INRAE UMR 792, INSA/CNRS UMR 5504
135 Avenue de Rangueil
31077 Toulouse Cedex 04

tel: +33 5 61 55 98 49
email: so...@insa-toulouse.fr
http://www.toulouse-biotechnology-institute.fr/en/technology_platforms/mathematics-cell.html

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Shouldn't "Loading" be "Attaching" when referring to library() calls?

[Martin Maechler]

> Martin Maechler 
> on Tue, 10 Jan 2023 10:26:53 +0100 writes:

> Michael Chirico via R-devel 
> on Mon, 9 Jan 2023 12:25:46 -0800 writes:

>> require() and library() both emit this message immediately before
>> running library():

>> "Loading required package: %s"

>> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L967-L968

>> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L684-L685

>> Shouldn't "Loading" be "Attaching" instead?

> Good question... I had asked myself as well a couple of times.

> The vagueness is mostly "historical" in the sense that
> originally R (nor S) had namespaces.

Oops!  If you skip the "( .. )" above it's 100% wrong.
Correct (I hope even according Oxford Dictionary) is

" Originally, neither R nor S had namespaces. "



> OTOH,  attach(.)  has always worked for data frames (already in pre-R S).

> Still, I agree (when teaching and otherwise), we should
> clearly distinguish the two notably in the context of packages
> and their namespaces.


>> My understanding is "a package is loaded" is equivalent to
>> "isNamespaceLoaded()", i.e., loadNamespace() was run. And that "a
>> package is attached" is equivalent to "pkg %in% .packages()", i.e.,
>> library(pkg) was run.

> or (easier for teaching)  that  "package:" %in% search()
> -- which is what .packages() does {by default}.

> I agree with the above.


>> Is the terminology not so precise?

>> There's certainly ambiguity in the internal variable names referenced
>> above -- in require, we see

>> loaded <- paste0("package:", package) %in% search()
>> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L680

> again, historical:  require() exists for considerably longer
> than namespaces.

> I think we should at least change the message printed by
> require() and library() you mentioned above.

> Martin


>> Whereas in library() [via .getRequiredPackages2()] we see

>> attached <- paste0("package:", pkg) %in% search()
>> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L931

>> Mike C

> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Bioc-devel] Support for Linux ARM64

[Martin Grigorov]

On Sun, Jan 8, 2023 at 12:24 AM Hervé Pagès 
wrote:

> On 05/01/2023 18:52, Vincent Carey wrote:
>
>
>
> On Thu, Jan 5, 2023 at 7:08 PM Vincent Carey 
> wrote:
>
>>
>>
>> On Thu, Jan 5, 2023 at 1:44 PM Hervé Pagès 
>> wrote:
>>
>>> Hi Martin,
>>>
>>> Linux runs on many architectures, ARM64 is just one of them.
>>>
>>> Our daily builds have traditionally focused on 3 platforms: Intel-based
>>> Linux (Ubuntu 22.04), Windows, and Intel-based Mac. Note that we
>>> recently added ARM64-based Mac to our daily builds.
>>>
>>> One big difference between Linux and the other platforms is that we only
>>> produce binary packages for the latter. More precisely:
>>>
>>> - on the Linux builders: the daily builds only run 'R CMD INSTALL', 'R
>>> CMD build', and 'R CMD check', on each Bioconductor package,
>>>
>>> - on the Windows and Mac builders: the daily builds run all the above
>>> plus an additional step that we call the BUILD BIN step that produces a
>>> binary for each Bioconductor package.
>>>
>>> This means that on Linux, as well as on any other Unix-like OS that is
>>> not macOS (e.g. FreeBSD, OpenBSD, Solaris, HP-UX, etc...), users will
>>> install all their packages (Bioconductor and CRAN) **from source**. This
>>> should work as long as they are on a platform where R is supported and
>>> have the required compilers (C, C++, and Fortran).
>>>
>>> Note that if officially supporting a given platform means running the
>>> daily builds on that particular platform, then there's no way for us to
>>> do that because platform == OS + architecture, and the list of
>>> combinations of Unix-like OS's (Linux, FreeBSD, Solaris, etc...) +
>>> architectures (Intel, ARM64, Sparc, powerpc) is endless. Even if we
>>> narrow this list to Intel-based Linux, there are hundreds of Linux
>>> distributions around that use different kernel, compilers, package
>>> managers, etc...
>>>
>>> All this to say that, as far as the daily builds are concerned, we had
>>> to make choices, and those choices are based on the most commonly used
>>> platforms. Since all Bioconductor packages are tested daily on
>>> Intel-based Linux (Ubuntu 22.04), Windows, Intel-based Mac, and
>>> ARM64-based Mac, we have some reasonable confidence that they will work
>>> properly on these 4 platforms (still not a 100% guarantee of course,
>>> there's nothing like that).
>>>
>>> My understanding is that ARM64-based Linux is still a marginally used
>>> platform so probably not worth for us to allocate resources on adding it
>>> to our daily builds at the moment. If it ever becomes more mainstream in
>>> the future, then we will certainly reconsider. That does not mean that
>>> you can't use Bioconductor on a ARM64-based Linux machine **now**. I see
>>> no reason a priori why you couldn't install (from source) Bioconductor
>>> packages on this platform, and use them, as long as:
>>>
>>>
>> Thanks Hervé for a good overview of the issues.  I think there are a
>> couple
>> of reasons to keep this dialogue going (and there is now a community
>> slack channel
>> for further discussion: #arm-linux at community-bioc.slack.com.)
>>
>> The first reason is Martin's offer of resources to accomplish the support
>> aim.  What
>> exactly that support aim is remains to be made precise.  As you note, a
>> properly
>> configured system with R can use BiocManager::install to build from
>> source, but
>> there are a few additional things that can be done to produce binaries,
>> and perhaps
>> some of our software in BBS or some of the binary repo generation tools
>> could be
>> useful for Martin's group to make a relevant binary repo.  The
>> package-management
>> oriented process of Dirk Eddelbuettel's r2u
>>  also seems potentially relevant.
>> We also
>> have tooling to build all the CRAN dependencies that Bioc packages
>> declare.  This
>> is all in the open and it would be interesting to see how much work is
>> needed to
>> get solutions for ARM64 linux.  It could lead to some robustification of
>> the existing
>> build machinery.  I am not offering to do it, but the fact that all the
>> tooling is out in
>> the open may not be fully clear and I am just mentioning this.
>>
>> The second reason to stay engaged is the nature of the ARM platform,
>> which is
>> said to require lower power consumption for equivalent throughput.  It
>> may be
>> environmentally beneficial to be ahead of the curve in being able to work
>> with
>> this platform.  Earlier I linked to a github issue indicating that rocker
>> now has a dual
>> platform container image including arm64 support but I don't know if that
>> really
>> addresses the issue at hand. Maybe I need to go onto a graviton machine
>> to find out.
>>
>
> So I did this, and here are some notes:
>
> 1) it is easy to get such a machine in AWS, a1.2xlarge
> Linux 10a568f32a1c 4.14.296-222.539.amzn2.aarch64 #1 SMP Wed Oct 26
> 20:36:51 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
> 2) using the 

[Rd] rhub vs. CRAN fedora-*-devel, using armadillo & slapack

[RICHET Yann]

Dear R-devel people,

We are working to submit a package which is mainly a binding over a C++ lib 
(https://github.com/libKriging) using armadillo.
It is _not_ a standard RcppArmadillo package, because we also had to provide a 
python binding... so there is a huge layer of cmake & scripting to make it work 
with a standard armadillo (but using same version that RcppArmadillo).
It seems now working with almost all CRAN targets, but a problem remains with 
fedora (clang & gcc) devel.

Our issue is that the rhub docker is not well sync with the CRAN 
fedora-clang-devel config:
- failing on CRAN (without clear reason): 
https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-fedora-clang/rlibkriging-00check.html
- passing on rhub: 
https://builder.r-hub.io/status/rlibkriging_0.7-3.tar.gz-20f7dc7562726497af7c678ab41f4dea

So we cannot investigate and fix the problem.

Note that we did quite strange things with the fedora platforms: include 
explicitely slapack to provide simple precision for our (vanilla) armadillo... 

Do you have any idea, or even known problem in mind, that could be related to 
this ?

Best regards,

--
Dr. Yann Richet
Institute for Radiological Protection and Nuclear Safety (https://www.irsn.fr), 
  Department of Characterization of Natural Unexpected Events and Sites
Office : +33 1 58 35 88 84

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Shouldn't "Loading" be "Attaching" when referring to library() calls?

[Martin Maechler]

> Michael Chirico via R-devel 
> on Mon, 9 Jan 2023 12:25:46 -0800 writes:

> require() and library() both emit this message immediately before
> running library():

> "Loading required package: %s"

> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L967-L968

> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L684-L685

> Shouldn't "Loading" be "Attaching" instead?

Good question... I had asked myself as well a couple of times.

The vagueness is mostly "historical" in the sense that
originally R (nor S) had namespaces.
OTOH,  attach(.)  has always worked for data frames (already in pre-R S).

Still, I agree (when teaching and otherwise), we should
clearly distinguish the two notably in the context of packages
and their namespaces.


> My understanding is "a package is loaded" is equivalent to
> "isNamespaceLoaded()", i.e., loadNamespace() was run. And that "a
> package is attached" is equivalent to "pkg %in% .packages()", i.e.,
> library(pkg) was run.

or (easier for teaching)  that  "package:" %in% search()
-- which is what .packages() does {by default}.

I agree with the above.


> Is the terminology not so precise?

> There's certainly ambiguity in the internal variable names referenced
> above -- in require, we see

> loaded <- paste0("package:", package) %in% search()
> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L680

again, historical:  require() exists for considerably longer
than namespaces.

I think we should at least change the message printed by
require() and library() you mentioned above.

Martin


> Whereas in library() [via .getRequiredPackages2()] we see

> attached <- paste0("package:", pkg) %in% search()
> 
https://github.com/r-devel/r-svn/blob/4198a2941b702d965bb2374c2b908f48c369f40a/src/library/base/R/library.R#L931

> Mike C

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel