HI Ganesh, Recommendation is to increase the number of nodes with lesser ram size. Your number of executors depend on the CPU core hence, i would recommend using 60 GB RAM cpu's with 2 executors each for your use case.
Thanks Ramesh On Wed, Feb 22, 2017 at 3:27 PM, Sai Ganesh Muthuraman < saiganesh...@gmail.com> wrote: > Hi, > > I am running a giraph application in the XSEDE comet cluster for graphs of > different sizes. > For a graph with 10,000 edges, I used about 8 workers on 2 nodes, each > node having 128GB RAM. My input file itself is just about 200KB. > But when I tried to increase the number of workers to 20 or more and the > number of nodes, the application takes infinite time and does not finish at > all. > > I have another graph data of size 50MB or so that has millions of edges. > If the number of workers is 2 or 3, I get this error > * java.lang.OutOfMemoryError: Java heap space* > If the number of workers is more, then the application doesn't end at all. > What is the best way to arrive at the number of workers and the number of > nodes, given the problem size? Is trial and error the only way? > > > Sai Ganesh > > >
