Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-10 Thread Fernando Perez
On Mon, Oct 10, 2011 at 3:06 AM, Fernando Perez wrote: >> I hope that this exchange will have been useful for you to understand >> better some aspects of cluster computing in some environment. I cannot >> rely go much further, as it is beyond my expertise. > > In this form, it truly has, and for t

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-10 Thread Fernando Perez
On Sun, Oct 9, 2011 at 5:06 PM, Gael Varoquaux wrote: > On Sun, Oct 09, 2011 at 03:23:51PM -0700, Fernando Perez wrote: >> Well, if it integrates 'quite poorly', it would be nice to hear this >> as a bug report or a question on the ipython lists, because PBS/SGE >> support is precisely one of the

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Gael Varoquaux
On Sun, Oct 09, 2011 at 03:23:51PM -0700, Fernando Perez wrote: > Well, if it integrates 'quite poorly', it would be nice to hear this > as a bug report or a question on the ipython lists, because PBS/SGE > support is precisely one of the current goals... Stefan just pointed > me to this thread, o

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Fernando Perez
On Sun, Oct 9, 2011 at 10:57 AM, Gael Varoquaux wrote: > The problem is that the IPython parallel utilities integrate quite poorly > with batch-driven engines, like SGE. I haven't tried it out myself, but > this is really what I hear around me. Well, if it integrates 'quite poorly', it would be n

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Satrajit Ghosh
hi olivier, As far as know, nope. It might be easy to rewrite this snippet using > the IPython parallel API in your code though: > > http://ipython.org/ipython-doc/dev/parallel/index.html i have done a fair bit of ipython parallel programming. but the use cases for machine learning vary so much

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Gael Varoquaux
On Sun, Oct 09, 2011 at 07:59:08PM +0200, Olivier Grisel wrote: > Good to know. For my use case though I just plan to use the ssh > ipython cluster without any job scheduler on the cluster itself. Yes, this is the way the IPython guys use it, and it works very well. But many of us do not have the

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Olivier Grisel
2011/10/9 Gael Varoquaux : > On Sun, Oct 09, 2011 at 07:54:56PM +0200, Olivier Grisel wrote: >> As far as know, nope. It might be easy to rewrite this snippet using >> the IPython parallel API in your code though: > >>   http://ipython.org/ipython-doc/dev/parallel/index.html > > The problem is that

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Gael Varoquaux
On Sun, Oct 09, 2011 at 07:54:56PM +0200, Olivier Grisel wrote: > As far as know, nope. It might be easy to rewrite this snippet using > the IPython parallel API in your code though: > http://ipython.org/ipython-doc/dev/parallel/index.html The problem is that the IPython parallel utilities inte

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Olivier Grisel
2011/10/9 Satrajit Ghosh : > hi all, > has anybody run the permutation_test_score on a distributed engine like SGE > or PBS? does joblib support such distribution? our estimator actually does > quite a bit of work, so we are trying to optimize the execution. > cheers, As far as know, nope. It migh

Re: [Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Gael Varoquaux
On Sun, Oct 09, 2011 at 01:33:58PM -0400, Satrajit Ghosh wrote: >has anybody run the permutation_test_score on a distributed engine like >SGE or PBS? No. >does joblib support such distribution? No. One the one hand I'd like to say: 'patch welcomed', on the other hand, I know of no li

[Scikit-learn-general] permutation test and distributed computing

2011-10-09 Thread Satrajit Ghosh
hi all, has anybody run the permutation_test_score on a distributed engine like SGE or PBS? does joblib support such distribution? our estimator actually does quite a bit of work, so we are trying to optimize the execution. cheers, satra --