Dear All,
Alex Lovell-Troy heads up innovation/cloud supercomputing at Cray (cc'd)
and he is a great resource for all things. I thought he might find this
thread useful.
Best, Alex
On Fri, Jun 28, 2019 at 11:45 PM Olivier Grisel
wrote:
> You have to use a dedicated framework to distribute the
You have to use a dedicated framework to distribute the computation on a
cluster like you cray system.
You can use mpi, or dask with dask-jobqueue but the also need to run
parallel algorithms that are efficient when running in a distributed with a
high cost for communication between distributed wo
Sorry, but just now I reread your answer more closely.
It seems that the "n_jobs" parameter of the DBScan routine brings no
benefit to performance. If I want to improve the performance of the
DBScan routine I will have to redesign the solution to use MPI
resources.
Is it correct?
---
Ats.,
My laptop has Intel I7 processor with 4 cores. When I run the program on
Windows 10, the "joblib.cpu_count()" routine returns "4". In these
cases, the same test I did on the Cray computer caused a 10% increase in
the processing time of the DBScan routine when I used the "n_jobs = 4"
parameter c
>
> where you can see "ncpus = 1" (I still do not know why 4 lines were
> printed -
>
> (total of 40 nodes) and each node has 1 CPU and 1 GPU!
>
> #PBS -l select=1:ncpus=8:mpiprocs=8
> aprun -n 4 p.sh ./ncpus.py
>
You can request 8 CPUs from a job scheduler, but if each node the script
runs on c
Finally I was able to access the Cray computer and run the
routine.
I am sending below the files and commands I used and the result found,
where you can see "ncpus = 1" (I still do not know why 4 lines were
printed - I only know that this amount depends on the value of the
"aprun" command us
2019年6月20日(木) 8:16 Mauricio Reis :
> But documentation (provided by a teacher in charge of the Cray computer)
> shows:
> - each node: 1 CPU, 1 GPU, 32 GBytes
>
If that's true, then it appears to me that running on any individual
compute host (node) has 1-core / 2-threads, and that would be why yo
I can not access the Cray computer at this moment to run the suggested
code. Once you have access, I'll let you know.
But documentation (provided by a teacher in charge of the Cray computer)
shows:
- 10 blades
- 4 nodes per blade = 40 nodes
- each node: 1 CPU, 1 GPU, 32 GBytes
---
Ats.,
Mauri
How many cores du you have on this machine?
joblib.cpu_count()
___
scikit-learn mailing list
scikit-learn@python.org
https://mail.python.org/mailman/listinfo/scikit-learn
I'd like to understand how parallelism works in the DBScan routine in
SciKit Learn running on the Cray computer and what should I do to
improve the results I'm looking at.
I have adapted the existing example in
[https://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html#sphx-glr-au
10 matches
Mail list logo
