Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-21 Thread Dirk Eddelbuettel
On 21 December 2016 at 09:42, Karl Millar via R-devel wrote: | Currently what I do is to never unload DLLs. If I need to replace | one, then I just restart R. It's less convenient, but it's always | correct. Same here. Ever since we built littler in 2006 (!!) I have been doing tests at the comm

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-21 Thread Karl Millar via R-devel
It does, but you'd still be relying on the R code ensuring that all of these objects are dead prior to unloading the DLL, otherwise they'll survive the GC. Maybe if the package counted how many such objects exist, it could work out when it's safe to remove the DLL. I'm not sure that it can be don

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-21 Thread Henrik Bengtsson
On Tue, Dec 20, 2016 at 7:39 AM, Karl Millar wrote: > It's not always clear when it's safe to remove the DLL. > > The main problem that I'm aware of is that native objects with > finalizers might still exist (created by R_RegisterCFinalizer etc). > Even if there are no live references to such obje

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-20 Thread Steve Bronder
See inlin ā€‹eā€‹ On Tue, Dec 20, 2016 at 12:14 PM, Spencer Graves < spencer.gra...@prodsyse.com> wrote: > Hi, Dirk: > > > > On 12/20/2016 10:56 AM, Dirk Eddelbuettel wrote: > >> On 20 December 2016 at 17:40, Martin Maechler wrote: >> | > Steve Bronder >> | > on Tue, 20 Dec 2016 01:34:3

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-20 Thread Spencer Graves
Hi, Dirk: On 12/20/2016 10:56 AM, Dirk Eddelbuettel wrote: On 20 December 2016 at 17:40, Martin Maechler wrote: | > Steve Bronder | > on Tue, 20 Dec 2016 01:34:31 -0500 writes: | | > Thanks Henrik this is very helpful! I will try this out on our tests and | > see if gcDLLs(

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-20 Thread Dirk Eddelbuettel
On 20 December 2016 at 17:40, Martin Maechler wrote: | > Steve Bronder | > on Tue, 20 Dec 2016 01:34:31 -0500 writes: | | > Thanks Henrik this is very helpful! I will try this out on our tests and | > see if gcDLLs() has a positive effect. | | > mlr currently has tests b

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-20 Thread Martin Maechler
> Steve Bronder > on Tue, 20 Dec 2016 01:34:31 -0500 writes: > Thanks Henrik this is very helpful! I will try this out on our tests and > see if gcDLLs() has a positive effect. > mlr currently has tests broken down by learner type such as classification, > regression

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-20 Thread Karl Millar via R-devel
It's not always clear when it's safe to remove the DLL. The main problem that I'm aware of is that native objects with finalizers might still exist (created by R_RegisterCFinalizer etc). Even if there are no live references to such objects (which would be hard to verify), it still wouldn't be safe

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-20 Thread Jeroen Ooms
On Tue, Dec 20, 2016 at 7:04 AM, Henrik Bengtsson wrote: > On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some > packages don't unload their DLLs when they being unloaded themselves. I am surprised by this. Why does R not do this automatically? What is the case for keeping the DL

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-19 Thread Steve Bronder
Thanks Henrik this is very helpful! I will try this out on our tests and see if gcDLLs() has a positive effect. mlr currently has tests broken down by learner type such as classification, regression, forecasting, clustering, etc.. There are 83 classifiers alone so even when loading and unloading a

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-19 Thread Henrik Bengtsson
On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some packages don't unload their DLLs when they being unloaded themselves. In other words, there may be left-over DLLs just sitting there doing nothing but occupying space. You can remove these, using: R.utils::gcDLLs() Maybe tha

[Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

2016-12-19 Thread Steve Bronder
This is a request to increase MAX_NUM_DLLS in Rdynload.c in from 100 to 500. On line 131 of Rdynload.c, changing #define MAX_NUM_DLLS 100 to #define MAX_NUM_DLLS 500 In development of the mlr package, there have been several episodes in the past where we have had to break up unit tests becau