Hi,
On 03/21/2018 07:32 AM, Martin Morgan wrote:
On 03/21/2018 10:14 AM, Henrik Bengtsson wrote:
A few quick comments:
* I don't think the limit was hardened in R 3.4.0; maybe you started to
experience it sound then because more dependent packages started to
rely on
more DLLs?
* The limit is OS/platform specific so it's not that R core is
unwilling to
fix this - I think they havejust tried to find a safe limit that work
everywhere. Having said this...
* It'll be less restrictive by default in R 3.5.0: from it's NEWS "The
maximum number of DLLs that can be loaded into R e.g. via dyn.load() has
been increased up to 614 when the OS limit on the number of open files
allows"
* R.utils::gcDLLs() will attempt to unregister possibly stray DLLs
remaining after unloading packages. So unloading packages and gcDLLs()
may
provide you with a workaround until R 3.5.0 is in place (possibly also
even
after).
See
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HenrikBengtsson_Wishlist-2Dfor-2DR_issues_29&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=7UNVEDPfEm1j53ksGc7naLgnieltNzOnbv4lPUQh4Jk&e=
for
"random" notes and references to this problem.
On Wed, Mar 21, 2018, 04:10 Yuan Tian <tianyuan1991...@gmail.com> wrote:
Hello Guys:
I encountered an error like "maximal number of dlls reached...". I am
maintaining ChAMP package now, which integrated many other packages
in my
research field. I have not updated this package in past 2 months but
suddenly this error happens.
Currently, I think I know the reason is since R 3.4.X, numbers of
DLLs in
default R session was set 100. I have tested that using a newly
started R
3.4.4 to install and load ChAMP package, *it works smoothly*, after
loading, I checked DLL loaded with function getLoadedDLLs(), then I see
ChAMP used 95 Dlls. I know it's a lot, but some of them are loaded by
ChAMP's relying packages but itself. *However, ChAMP cannot pass
Bioconductor check, thus I suspect Bioconductor checking system does NOT
start a new R session for each package right? Which means it's not
100 DLLs
allocated for each package?*
ChAMP is passing BiocCheck in release, where R is 3.4.x ?
https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_checkResults_release_bioc-2DLATEST_ChAMP_malbec1-2Dchecksrc.html&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=_B73-e5Y2XkRdvs7ssHGb6DAdSnMmG30eE_Ywu1W7iQ&e=
mmmh, this is a stale page. Not sure what it's doing here. Will
investigate.
Today's "true" results are here:
http://bioconductor.org/checkResults/release/bioc-LATEST/ChAMP/
Note that you can't get to the stale page by following links from
the report's main page:
http://bioconductor.org/checkResults/release/bioc-LATEST/
Each package is built and tested in a separate process. Vignettes are
actually built in a single process (by R, not Bioconductor) so multiple
vignettes could load a higher number of DLLs.
Also just to clarify: the build machines don't use any special settings
to limit the nb of DLLs. They use whatever R uses by default.
Currently, I have told my users to modify R environment to allow their R
session with more DLLs. It works but only on users computer, not
Bioconductor checking system.
Note that, in release (with R 3.4.4), the max DLL error happens at
the end of an install:
> biocLite("ChAMP")
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.6 (BiocInstaller 1.28.0), R 3.4.3 (2017-11-30).
Installing package(s) ‘ChAMP’
trying URL
'https://bioconductor.org/packages/3.6/bioc/src/contrib/ChAMP_2.9.10.tar.gz'
Content type 'application/x-gzip' length 15513863 bytes (14.8 MB)
==================================================
downloaded 14.8 MB
* installing *source* package ‘ChAMP’ ...
** R
** inst
** preparing package for lazy loading
Warning: replacing previous import ‘igraph::edges’ by ‘graph::edges’
when loading ‘FEM’
Warning: replacing previous import ‘igraph::intersection’ by
‘graph::intersection’ when loading ‘FEM’
Warning: replacing previous import ‘igraph::degree’ by ‘graph::degree’
when loading ‘FEM’
Warning: replacing previous import ‘igraph::union’ by ‘graph::union’
when loading ‘FEM’
Warning: replacing previous import ‘limma::plotMA’ by
‘BiocGenerics::plotMA’ when loading ‘FEM’
Warning: replacing previous import ‘Matrix::colSums’ by
‘BiocGenerics::colSums’ when loading ‘FEM’
Warning: replacing previous import ‘Matrix::colMeans’ by
‘BiocGenerics::colMeans’ when loading ‘FEM’
Warning: replacing previous import ‘Matrix::rowMeans’ by
‘BiocGenerics::rowMeans’ when loading ‘FEM’
Warning: replacing previous import ‘Matrix::rowSums’ by
‘BiocGenerics::rowSums’ when loading ‘FEM’
Warning: replacing previous import ‘Matrix::which’ by
‘BiocGenerics::which’ when loading ‘FEM’
Warning: replacing previous import ‘igraph::normalize’ by
‘BiocGenerics::normalize’ when loading ‘FEM’
Warning: replacing previous import 'plyr::summarise' by
'plotly::summarise' when loading 'ChAMP'
Warning: replacing previous import 'plyr::rename' by 'plotly::rename'
when loading 'ChAMP'
Warning: replacing previous import 'plyr::arrange' by 'plotly::arrange'
when loading 'ChAMP'
Warning: replacing previous import 'plyr::mutate' by 'plotly::mutate'
when loading 'ChAMP'
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object
'/home/hpages/R/R-3.4.3/library/robustbase/libs/robustbase.so':
`maximal number of DLLs reached...
ERROR: lazy loading failed for package 'ChAMP'
* removing '/home/hpages/R/R-3.4.3/library/ChAMP'
The downloaded source packages are in
‘/tmp/Rtmp87p9oq/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
So even if ChAMP's documentation says something about increasing the
max number of DLLs, most users are probably going to find this situation
frustrating and complain about it.
Thus I think I have to reduce dlls used by
current package right? Like removing some relying packages or
function. *My
another question is how many DLLs is allowed by Bioconductor check? I
think
it's less than 100. But I don't know I should cut it into 80 or 60 or
even
50 dlls used.*
It's really disappointing that I need to modify quite a lot of code, and
even could hurt some key functionality of the package. Thus here I am
seeking your help and suggestions here.
Frankly, I think a package with so many dependencies cannot be
maintained -- a change in any one of those packages could break your
package (e.g., by changing their own dependencies to include additional
DLLs!), and it must be virtually impossible to get sufficient test
coverage to be confident that these problems will be detected before the
package is made available to the user. It is time to consider a more
modular design focusing on essential features.
The first thing you could try to do is move some deps from Depends or
Imports to Suggests. Right now, after doing library(ChAMP),
sessionInfo() reports 36 packages on the search path and another 160
packages loaded via a namespace! Couldn't some of those packages be
moved to Suggests without hurting the usability of ChAMP?
One last thing: ChAMP is at version 2.9.10 in release and 2.9.11 in
devel. This is not good. The y part of x.y.z should always be even
in release. Generally speaking, you should only bump the z part of the
version when you make changes to your package. We take care of bumping
the y part for you at release time. See:
https://bioconductor.org/developers/how-to/version-numbering/
Too late to fix in release though. Just be aware that when we release
BioC 3.7, we'll bump ChAMP version to 2.10.0 in the new release and
to 2.11.0 in the new devel (i.e. BioC 3.8).
Cheers,
H.
Martin
Best
Yuan Tian
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel@r-project.org mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e=
[[alternative HTML version deleted]]
_______________________________________________
Bioc-devel@r-project.org mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e=
This email message may contain legally privileged and/or...{{dropped:2}}
_______________________________________________
Bioc-devel@r-project.org mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e=
--
Hervé Pagès
Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024
E-mail: hpa...@fredhutch.org
Phone: (206) 667-5791
Fax: (206) 667-1319
_______________________________________________
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
