I remember that a good practice is using 1 worker per node, there are several emails recommending this. It's the best way to use the maximum RAM available in the cluster i believe.
Bye -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java y .NET en LIFIA 2017-02-22 7:34 GMT-03:00 Ramesh Krishnan <[email protected]>: > HI Ganesh, > > Recommendation is to increase the number of nodes with lesser ram size. > Your number of executors depend on the CPU core hence, i would recommend > using 60 GB RAM cpu's with 2 executors each for your use case. > > Thanks > Ramesh > > On Wed, Feb 22, 2017 at 3:27 PM, Sai Ganesh Muthuraman < > [email protected]> wrote: > >> Hi, >> >> I am running a giraph application in the XSEDE comet cluster for graphs >> of different sizes. >> For a graph with 10,000 edges, I used about 8 workers on 2 nodes, each >> node having 128GB RAM. My input file itself is just about 200KB. >> But when I tried to increase the number of workers to 20 or more and the >> number of nodes, the application takes infinite time and does not finish at >> all. >> >> I have another graph data of size 50MB or so that has millions of edges. >> If the number of workers is 2 or 3, I get this error >> * java.lang.OutOfMemoryError: Java heap space* >> If the number of workers is more, then the application doesn't end at all. >> What is the best way to arrive at the number of workers and the number of >> nodes, given the problem size? Is trial and error the only way? >> >> >> Sai Ganesh >> >> >> >
