If you set n_jobs to XXX, it will spawn XXX threads or processes. Thus, you will
need to ask for XXX cores. Note that it’s often possible to retrieve XXX in
your script using os.environ.
If you use less than the XXX cores, then you won’t
use all the available cpu. If you ask for more than XXX cores, you will start
draining computing ressources from the others jobs (slurm jobs / age jobs /
...) that shares the same node
as your job.
Setting an appropriate XXX is dependant on the policy of the cluster.
> Does this mean if I (reserve 64 cores and) specify n_jobs=64 the job will
> take (Approximately) half of the time then n_jobs=32 ?
With joblib, you often have a linear scaling. But it depends upon the
algorithm.
Best regards,
Arnaud
On 21 Aug 2014, at 14:32, Sheila the angel <[email protected]> wrote:
> I still have following doubt:
>
> I understand that n_jobs "should be depending on the number of cpu cores
> available on your machine". But I am running code on Grid computing
> environment where I have to specify the number of CPU cores in advance.
>
> Does this mean if I (reserve 64 cores and) specify n_jobs=64 the job will
> take (Approximately) half of the time then n_jobs=32 ?
>
> And again just to be sure :
> What if I (reserve and) specify n_jobs= 35 , will all the 35 cores be used
> (while the maximum possible job combination is 32)?
>
> I want to avoid the situation where the CPU core is reserved but not used.
>
>
>
> On 21 August 2014 13:34, Sheila the angel <[email protected]> wrote:
> First Thanks for reply.
>
> @Hames : I understand that n_jobs "should be depending on the number of cpu
> cores available on your machine". But I am running code on Grid computing
> environment where I have to specify the number of CPUs in advance.
>
> Does this mean if I (reserve 64 cores and) specify n_jobs=64 the job will
> take (Approximately) half of the time then n_jobs=32 ?
>
> And again just to be sure :
> What if I (reserve and) specify n_jobs= 35 , will all the 35 cores be used
> (while the maximum possible job combination is 32)?
>
> Off course, I want to avoid the situation where the core is reserved but not
> used.
>
>
>
>
> On 21 August 2014 13:03, Lars Buitinck <[email protected]> wrote:
> 2014-08-21 12:32 GMT+02:00 Sheila the angel <[email protected]>:
> > 1. What should be the n_jobs value, 8 or (8*4=) 32 ?
>
> n_jobs is the number of CPUs you want to use, not the amount of work.
> (It's a misnomer because the number of jobs/work items is variable;
> the parameter determines the number of workers performing the jobs.)
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds. Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
> ------------------------------------------------------------------------------
> Slashdot TV.
> Video for Nerds. Stuff that matters.
> http://tv.slashdot.org/_______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Slashdot TV.
Video for Nerds. Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
