memory usage in giraph

2013-10-16 Thread Jyotirmoy Sundi
Hi. I am having data of around 4B Vertices and 13548192791 edges, say with avg outgoing edge of 4, so total 4B*40 bytes= 160G , When I load this into memory across 165 mappers each with 6G mem, all memory get occupied, I am confused here as we have around 1TB space in memory. What is its thats

Resources or advice on minimising memory usage in Giraph/Hadoop code ?

2012-06-05 Thread Benjamin Heitmann
Hello, can somebody recommend a web page, article or book on minimising the memory usage of Giraph/Hadoop code ? I am looking for non-obvious advice on what *not* to do, and for best practices on what to do inside of Hadoop... E.g. is it preferable to use Java Strings or Hadoop Text