Re: [Bioc-devel] Help for Error "Maximal Number of DLLs reached..."
Thanks guys for your kind help! I would follow your tips and try to fix these issues. It's indeed quite hard to maintain it, luckily in past two years I made it. It's indeed the right time to cut down some unnecessary packages and make it easier to be maintained. Thanks for your help! ^_^ Best Yuan Tian On Thu, Mar 22, 2018 at 12:14 AM Hervé Pagèswrote: > Hi, > > On 03/21/2018 07:32 AM, Martin Morgan wrote: > > > > > > On 03/21/2018 10:14 AM, Henrik Bengtsson wrote: > >> A few quick comments: > >> > >> * I don't think the limit was hardened in R 3.4.0; maybe you started to > >> experience it sound then because more dependent packages started to > >> rely on > >> more DLLs? > >> > >> * The limit is OS/platform specific so it's not that R core is > >> unwilling to > >> fix this - I think they havejust tried to find a safe limit that work > >> everywhere. Having said this... > >> > >> * It'll be less restrictive by default in R 3.5.0: from it's NEWS "The > >> maximum number of DLLs that can be loaded into R e.g. via dyn.load() has > >> been increased up to 614 when the OS limit on the number of open files > >> allows" > >> > >> * R.utils::gcDLLs() will attempt to unregister possibly stray DLLs > >> remaining after unloading packages. So unloading packages and gcDLLs() > >> may > >> provide you with a workaround until R 3.5.0 is in place (possibly also > >> even > >> after). > >> > >> See > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HenrikBengtsson_Wishlist-2Dfor-2DR_issues_29=DwICAg=eRAMFD45gAfqt84VtBcfhQ=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8=7UNVEDPfEm1j53ksGc7naLgnieltNzOnbv4lPUQh4Jk= > >> for > >> "random" notes and references to this problem. > >> > >> > >> On Wed, Mar 21, 2018, 04:10 Yuan Tian > wrote: > >> > >>> Hello Guys: > >>> > >>> I encountered an error like "maximal number of dlls reached...". I am > >>> maintaining ChAMP package now, which integrated many other packages > >>> in my > >>> research field. I have not updated this package in past 2 months but > >>> suddenly this error happens. > >>> > >>> Currently, I think I know the reason is since R 3.4.X, numbers of > >>> DLLs in > >>> default R session was set 100. I have tested that using a newly > >>> started R > >>> 3.4.4 to install and load ChAMP package, *it works smoothly*, after > >>> loading, I checked DLL loaded with function getLoadedDLLs(), then I see > >>> ChAMP used 95 Dlls. I know it's a lot, but some of them are loaded by > >>> ChAMP's relying packages but itself. *However, ChAMP cannot pass > >>> Bioconductor check, thus I suspect Bioconductor checking system does > NOT > >>> start a new R session for each package right? Which means it's not > >>> 100 DLLs > >>> allocated for each package?* > > > > ChAMP is passing BiocCheck in release, where R is 3.4.x ? > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_checkResults_release_bioc-2DLATEST_ChAMP_malbec1-2Dchecksrc.html=DwICAg=eRAMFD45gAfqt84VtBcfhQ=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8=_B73-e5Y2XkRdvs7ssHGb6DAdSnMmG30eE_Ywu1W7iQ= > > mmmh, this is a stale page. Not sure what it's doing here. Will > investigate. > > Today's "true" results are here: > >http://bioconductor.org/checkResults/release/bioc-LATEST/ChAMP/ > > Note that you can't get to the stale page by following links from > the report's main page: > >http://bioconductor.org/checkResults/release/bioc-LATEST/ > > > > > > > Each package is built and tested in a separate process. Vignettes are > > actually built in a single process (by R, not Bioconductor) so multiple > > vignettes could load a higher number of DLLs. > > Also just to clarify: the build machines don't use any special settings > to limit the nb of DLLs. They use whatever R uses by default. > > > > >>> > >>> Currently, I have told my users to modify R environment to allow their > R > >>> session with more DLLs. It works but only on users computer, not > >>> Bioconductor checking system. > > Note that, in release (with R 3.4.4), the max DLL error happens at > the end of an install: > > > biocLite("ChAMP") > BioC_mirror: https://bioconductor.org > Using Bioconductor 3.6 (BiocInstaller 1.28.0), R 3.4.3 (2017-11-30). > Installing package(s) ‘ChAMP’ > trying URL > ' > https://bioconductor.org/packages/3.6/bioc/src/contrib/ChAMP_2.9.10.tar.gz > ' > Content type 'application/x-gzip' length 15513863 bytes (14.8 MB) > == > downloaded 14.8 MB > > * installing *source* package ‘ChAMP’ ... > ** R > ** inst > ** preparing package for lazy loading > Warning: replacing previous import ‘igraph::edges’ by ‘graph::edges’ > when loading ‘FEM’ > Warning: replacing previous import ‘igraph::intersection’ by > ‘graph::intersection’ when loading ‘FEM’ > Warning: replacing previous import ‘igraph::degree’ by
Re: [Bioc-devel] Help for Error "Maximal Number of DLLs reached..."
Hi, On 03/21/2018 07:32 AM, Martin Morgan wrote: On 03/21/2018 10:14 AM, Henrik Bengtsson wrote: A few quick comments: * I don't think the limit was hardened in R 3.4.0; maybe you started to experience it sound then because more dependent packages started to rely on more DLLs? * The limit is OS/platform specific so it's not that R core is unwilling to fix this - I think they havejust tried to find a safe limit that work everywhere. Having said this... * It'll be less restrictive by default in R 3.5.0: from it's NEWS "The maximum number of DLLs that can be loaded into R e.g. via dyn.load() has been increased up to 614 when the OS limit on the number of open files allows" * R.utils::gcDLLs() will attempt to unregister possibly stray DLLs remaining after unloading packages. So unloading packages and gcDLLs() may provide you with a workaround until R 3.5.0 is in place (possibly also even after). See https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HenrikBengtsson_Wishlist-2Dfor-2DR_issues_29=DwICAg=eRAMFD45gAfqt84VtBcfhQ=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8=7UNVEDPfEm1j53ksGc7naLgnieltNzOnbv4lPUQh4Jk= for "random" notes and references to this problem. On Wed, Mar 21, 2018, 04:10 Yuan Tianwrote: Hello Guys: I encountered an error like "maximal number of dlls reached...". I am maintaining ChAMP package now, which integrated many other packages in my research field. I have not updated this package in past 2 months but suddenly this error happens. Currently, I think I know the reason is since R 3.4.X, numbers of DLLs in default R session was set 100. I have tested that using a newly started R 3.4.4 to install and load ChAMP package, *it works smoothly*, after loading, I checked DLL loaded with function getLoadedDLLs(), then I see ChAMP used 95 Dlls. I know it's a lot, but some of them are loaded by ChAMP's relying packages but itself. *However, ChAMP cannot pass Bioconductor check, thus I suspect Bioconductor checking system does NOT start a new R session for each package right? Which means it's not 100 DLLs allocated for each package?* ChAMP is passing BiocCheck in release, where R is 3.4.x ? https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_checkResults_release_bioc-2DLATEST_ChAMP_malbec1-2Dchecksrc.html=DwICAg=eRAMFD45gAfqt84VtBcfhQ=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8=_B73-e5Y2XkRdvs7ssHGb6DAdSnMmG30eE_Ywu1W7iQ= mmmh, this is a stale page. Not sure what it's doing here. Will investigate. Today's "true" results are here: http://bioconductor.org/checkResults/release/bioc-LATEST/ChAMP/ Note that you can't get to the stale page by following links from the report's main page: http://bioconductor.org/checkResults/release/bioc-LATEST/ Each package is built and tested in a separate process. Vignettes are actually built in a single process (by R, not Bioconductor) so multiple vignettes could load a higher number of DLLs. Also just to clarify: the build machines don't use any special settings to limit the nb of DLLs. They use whatever R uses by default. Currently, I have told my users to modify R environment to allow their R session with more DLLs. It works but only on users computer, not Bioconductor checking system. Note that, in release (with R 3.4.4), the max DLL error happens at the end of an install: > biocLite("ChAMP") BioC_mirror: https://bioconductor.org Using Bioconductor 3.6 (BiocInstaller 1.28.0), R 3.4.3 (2017-11-30). Installing package(s) ‘ChAMP’ trying URL 'https://bioconductor.org/packages/3.6/bioc/src/contrib/ChAMP_2.9.10.tar.gz' Content type 'application/x-gzip' length 15513863 bytes (14.8 MB) == downloaded 14.8 MB * installing *source* package ‘ChAMP’ ... ** R ** inst ** preparing package for lazy loading Warning: replacing previous import ‘igraph::edges’ by ‘graph::edges’ when loading ‘FEM’ Warning: replacing previous import ‘igraph::intersection’ by ‘graph::intersection’ when loading ‘FEM’ Warning: replacing previous import ‘igraph::degree’ by ‘graph::degree’ when loading ‘FEM’ Warning: replacing previous import ‘igraph::union’ by ‘graph::union’ when loading ‘FEM’ Warning: replacing previous import ‘limma::plotMA’ by ‘BiocGenerics::plotMA’ when loading ‘FEM’ Warning: replacing previous import ‘Matrix::colSums’ by ‘BiocGenerics::colSums’ when loading ‘FEM’ Warning: replacing previous import ‘Matrix::colMeans’ by ‘BiocGenerics::colMeans’ when loading ‘FEM’ Warning: replacing previous import ‘Matrix::rowMeans’ by ‘BiocGenerics::rowMeans’ when loading ‘FEM’ Warning: replacing previous import ‘Matrix::rowSums’ by ‘BiocGenerics::rowSums’ when loading ‘FEM’ Warning: replacing previous import ‘Matrix::which’ by ‘BiocGenerics::which’ when loading ‘FEM’ Warning: replacing previous import ‘igraph::normalize’ by
Re: [Bioc-devel] Help for Error "Maximal Number of DLLs reached..."
On 03/21/2018 10:14 AM, Henrik Bengtsson wrote: A few quick comments: * I don't think the limit was hardened in R 3.4.0; maybe you started to experience it sound then because more dependent packages started to rely on more DLLs? * The limit is OS/platform specific so it's not that R core is unwilling to fix this - I think they havejust tried to find a safe limit that work everywhere. Having said this... * It'll be less restrictive by default in R 3.5.0: from it's NEWS "The maximum number of DLLs that can be loaded into R e.g. via dyn.load() has been increased up to 614 when the OS limit on the number of open files allows" * R.utils::gcDLLs() will attempt to unregister possibly stray DLLs remaining after unloading packages. So unloading packages and gcDLLs() may provide you with a workaround until R 3.5.0 is in place (possibly also even after). See https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29 for "random" notes and references to this problem. On Wed, Mar 21, 2018, 04:10 Yuan Tianwrote: Hello Guys: I encountered an error like "maximal number of dlls reached...". I am maintaining ChAMP package now, which integrated many other packages in my research field. I have not updated this package in past 2 months but suddenly this error happens. Currently, I think I know the reason is since R 3.4.X, numbers of DLLs in default R session was set 100. I have tested that using a newly started R 3.4.4 to install and load ChAMP package, *it works smoothly*, after loading, I checked DLL loaded with function getLoadedDLLs(), then I see ChAMP used 95 Dlls. I know it's a lot, but some of them are loaded by ChAMP's relying packages but itself. *However, ChAMP cannot pass Bioconductor check, thus I suspect Bioconductor checking system does NOT start a new R session for each package right? Which means it's not 100 DLLs allocated for each package?* ChAMP is passing BiocCheck in release, where R is 3.4.x ? http://bioconductor.org/checkResults/release/bioc-LATEST/ChAMP/malbec1-checksrc.html Each package is built and tested in a separate process. Vignettes are actually built in a single process (by R, not Bioconductor) so multiple vignettes could load a higher number of DLLs. Currently, I have told my users to modify R environment to allow their R session with more DLLs. It works but only on users computer, not Bioconductor checking system. Thus I think I have to reduce dlls used by current package right? Like removing some relying packages or function. *My another question is how many DLLs is allowed by Bioconductor check? I think it's less than 100. But I don't know I should cut it into 80 or 60 or even 50 dlls used.* It's really disappointing that I need to modify quite a lot of code, and even could hurt some key functionality of the package. Thus here I am seeking your help and suggestions here. Frankly, I think a package with so many dependencies cannot be maintained -- a change in any one of those packages could break your package (e.g., by changing their own dependencies to include additional DLLs!), and it must be virtually impossible to get sufficient test coverage to be confident that these problems will be detected before the package is made available to the user. It is time to consider a more modular design focusing on essential features. Martin Best Yuan Tian [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel This email message may contain legally privileged and/or...{{dropped:2}} ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
[Bioc-devel] Help for Error "Maximal Number of DLLs reached..."
Hello Guys: I encountered an error like "maximal number of dlls reached...". I am maintaining ChAMP package now, which integrated many other packages in my research field. I have not updated this package in past 2 months but suddenly this error happens. Currently, I think I know the reason is since R 3.4.X, numbers of DLLs in default R session was set 100. I have tested that using a newly started R 3.4.4 to install and load ChAMP package, *it works smoothly*, after loading, I checked DLL loaded with function getLoadedDLLs(), then I see ChAMP used 95 Dlls. I know it's a lot, but some of them are loaded by ChAMP's relying packages but itself. *However, ChAMP cannot pass Bioconductor check, thus I suspect Bioconductor checking system does NOT start a new R session for each package right? Which means it's not 100 DLLs allocated for each package?* Currently, I have told my users to modify R environment to allow their R session with more DLLs. It works but only on users computer, not Bioconductor checking system. Thus I think I have to reduce dlls used by current package right? Like removing some relying packages or function. *My another question is how many DLLs is allowed by Bioconductor check? I think it's less than 100. But I don't know I should cut it into 80 or 60 or even 50 dlls used.* It's really disappointing that I need to modify quite a lot of code, and even could hurt some key functionality of the package. Thus here I am seeking your help and suggestions here. Best Yuan Tian [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel