On Sun, Oct 09, 2011 at 01:33:58PM -0400, Satrajit Ghosh wrote: > has anybody run the permutation_test_score on a distributed engine like > SGE or PBS?
No. > does joblib support such distribution? No. One the one hand I'd like to say: 'patch welcomed', on the other hand, I know of no library implementing the distribution on such clusters and stable, so we would have to integrate the low level SGE logic to joblib. Looking at Nipype [1] it does not seems too bad. The hardest work would be probably to create executable jobs that do the work. I would be interested by such a feature in joblib, but only if it comes with a small mocked implementation of the submission API to be able to have a full test coverage. Indeed, I suspect that the code that will create Python jobs on the fly will not be tiny. In the same alley, I think that it might be interesting if the 'n_jobs' can actually accept not only a number, but also optionaly a dispatcher object implementing pretty much a parallel mapper interface (to be defined). This would enable people to write their own dispatcher without having to patch joblib. Indeed, joblib cannot grow to account for all the possible dispatching mechanism, as I am worried that it will be code that will be little tested and eventually bitrot. Cheers, Gaël [1] https://github.com/nipy/nipype/blob/master/nipype/pipeline/plugins/pbs.py https://github.com/nipy/nipype/blob/master/nipype/pipeline/plugins/sge.py ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2 _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
