Yes, it does help a LOT Avery; thanks for your patience with a newbie. I have been able to get my job to run ok on the full graph by upping the VM on the Hadoop map tasks. This problem has also forced me to learn how to make AWS EMR parameterize Hadoop, so it's a good exercise anyway. One last question, particularly in view of the byte arrays. Suppose I DID screw up my own classes somehow so they gobbled up too much memory. I have no problem debugging and profiling my stuff because I can use LOG statements and that plus our old friend System.nanoTime() is enough to get me through. But how would I debug memory problems in this environment, even in principle, beyond what jmap gives me in a heap snapshot? Does Giraph or Hadoop have any debugging facilities I could turn on. What do you do? Thanks again.
On Fri, Aug 30, 2013 at 4:54 PM, Avery Ching <ach...@apache.org> wrote: > Ah, the new caches. =) These make things a lot faster (bulk data > sending), but do take up some additional memory. if you look at > GiraphConstants, you can find ways to change the cache sizes (this will > reduce that memory usage). > For example, MAX_EDGE_REQUEST_SIZE will affect the size of the edge > cache. MAX_MSG_REQUEST_SIZE will affect the size of the message cache. > The caches are per worker, so 100 workers would require 50 MB per worker > by default. Feel free to trim it if you like. > > The byte arrays for the edges are the most efficient storage possible > (although not as performance as the native edge stores). > > Hope that helps, > > Avery > > > On 8/29/13 4:53 PM, Jeff Peters wrote: > > Avery, it would seem that optimizations to Giraph have, unfortunately, > turned the majority of the heap into "dark matter". The two snapshots are > at unknown points in a superstep but I waited for several supersteps so > that the activity had more or less stabilized. About the only thing > comparable between the two snapshots are the vertexes, 192561 X > "RecsVertex" in the new version and 191995 X "Coloring" in the old system. > But with the new Giraph 672710176 out of 824886184 bytes are stored as > primitive byte arrays. That's probably indicative of some very fine > performance optimization work, but it makes it extremely difficult to know > what's really out there, and why. I did notice that a number of caches have > appeared that did not exist before, > namely SendEdgeCache, SendPartitionCache, SendMessageCache > and SendMutationsCache. > > Could any of those account for a larger per-worker footprint in a modern > Giraph? Should I simply assume that I need to force AWS to configure its > EMR Hadoop so that each instance has fewer map tasks but with a somewhat > larger VM max, say 3GB instead of 2GB? > > > On Wed, Aug 28, 2013 at 4:57 PM, Avery Ching <ach...@apache.org> wrote: > >> Try dumping a histogram of memory usage from a running JVM and see where >> the memory is going. I can't think of anything in particular that >> changed... >> >> >> On 8/28/13 4:39 PM, Jeff Peters wrote: >> >>> >>> I am tasked with updating our ancient (circa 7/10/2012) Giraph to >>> giraph-release-1.0.0-RC3. Most jobs run fine but our largest job now runs >>> out of memory using the same AWS elastic-mapreduce configuration we have >>> always used. I have never tried to configure either Giraph or the AWS >>> Hadoop. We build for Hadoop 1.0.2 because that's closest to the 1.0.3 AWS >>> provides us. The 8 X m2.4xlarge cluster we use seems to provide 8*14=112 >>> map tasks fitted out with 2GB heap each. Our code is completely unchanged >>> except as required to adapt to the new Giraph APIs. Our vertex, edge, and >>> message data are completely unchanged. On smaller jobs, that work, the >>> aggregate heap usage high-water mark seems about the same as before, but >>> the "committed heap" seems to run higher. I can't even make it work on a >>> cluster of 12. In that case I get one map task that seems to end up with >>> nearly twice as many messages as most of the others so it runs out of >>> memory anyway. It only takes one to fail the job. Am I missing something >>> here? Should I be configuring my new Giraph in some way I didn't used to >>> need to with the old one? >>> >>> >> > >
