Re: [Scikit-learn-general] Save trained classifier

2014-01-06 Thread Adolfo Martinez
Hello, I have a trained ExtraTreesRegressor saved using joblib.dump (without compress). This creates more than ten thousand files, each weighing less than 100KB. When trying to load using joblib.load with mmap_mode="r" I get a [Errno 24] Too many open files. Is there a way to save my regressor wit

Re: [Scikit-learn-general] Save trained classifier

2013-12-20 Thread Olivier Grisel
2013/12/20 Fred Mailhot : > On 19 December 2013 15:16, Olivier Grisel wrote: >> >> [...] >> But on the other hand that makes it possible to [...] to memory map the >> large parameter >> >> arrays by passing mmap_mode='r' to joblib.load for instance. >> >> Memory mapping can be useful to share the

Re: [Scikit-learn-general] Save trained classifier

2013-12-20 Thread Mathieu Blondel
See also this long-standing issue: https://github.com/scikit-learn/scikit-learn/issues/1332 On Fri, Dec 20, 2013 at 4:45 PM, Fred Mailhot wrote: > On 19 December 2013 15:16, Olivier Grisel wrote: > >> [...] >> But on the other hand that makes it possible to [...] to memory map the >> large param

Re: [Scikit-learn-general] Save trained classifier

2013-12-19 Thread Fred Mailhot
On 19 December 2013 15:16, Olivier Grisel wrote: > [...] > But on the other hand that makes it possible to [...] to memory map the > large parameter > arrays by passing mmap_mode='r' to joblib.load for instance. > > Memory mapping can be useful to share the memory of models loaded in > several py

Re: [Scikit-learn-general] Save trained classifier

2013-12-19 Thread Juan Nunez-Iglesias
On Fri, Dec 20, 2013 at 9:15 AM, Su, Jian, Ph.D. wrote: > As Ryan pointed out, joblib is the solution. One bad thing is it creates > multiple files. > If I remember correctly, I fixed the multiple files issue by passing "compress=3" as a keyword argument to joblib.dump. That does prevent the mem

Re: [Scikit-learn-general] Save trained classifier

2013-12-19 Thread Olivier Grisel
2013/12/19 Su, Jian, Ph.D. : > Zach, > > You can save a classifier with pickle, but you need to train the > classifier after you load it, right? No all the internal state of the classifier is preserved as well. > As Ryan pointed out, joblib is the solution. One bad thing is it creates > multiple

Re: [Scikit-learn-general] Save trained classifier

2013-12-19 Thread Su, Jian, Ph.D.
Zach, You can save a classifier with pickle, but you need to train the classifier after you load it, right? As Ryan pointed out, joblib is the solution. One bad thing is it creates multiple files. On 12/19/13 4:06 PM, "Zach Dwiel" wrote: >which classifier are you using? I pickle trained Deci

Re: [Scikit-learn-general] Save trained classifier

2013-12-19 Thread Ryan Rosario
sday, December 19, 2013 2:05 PM To: "scikit-learn-general@lists.sourceforge.net<mailto:scikit-learn-general@lists.sourceforge.net>" mailto:scikit-learn-general@lists.sourceforge.net>> Subject: Re: [Scikit-learn-general] Save trained classifier Yes, use job lib in skle

Re: [Scikit-learn-general] Save trained classifier

2013-12-19 Thread Zach Dwiel
which classifier are you using? I pickle trained DecisionTree and RandomForest classifiers, save them to disk and then load them up in production without a problem all the time. zach On Thu, Dec 19, 2013 at 5:00 PM, Su, Jian, Ph.D. wrote: > Is it possible to save the trained classifier and pass

Re: [Scikit-learn-general] Save trained classifier

2013-12-19 Thread Ryan Rosario
ral@lists.sourceforge.net>> Date: Thursday, December 19, 2013 2:00 PM To: "Scikit-learn-general@lists.sourceforge.net<mailto:Scikit-learn-general@lists.sourceforge.net>" mailto:Scikit-learn-general@lists.sourceforge.net>> Subject: [Scikit-learn-general] Save trained c

[Scikit-learn-general] Save trained classifier

2013-12-19 Thread Su, Jian, Ph.D.
Is it possible to save the trained classifier and pass it to other machine for direct prediction? I tried pickle which only saves the classifier parameters before training. -- Rapidly troubleshoot problems before they aff