Re: Giraph : newbie questions
Giraph partitions the vertices using a hashing function that's basically the equivalent of (hash(vertexID) mod #ofComputeNodes). You can mitigate memory issues by starting the job with a minimum of vertices in your file and then add them dynamically as your job progresses (assuming that your job doesn't require all of the vertices). -David On 7/16/12 4:36 AM, Nicolas DUGUE nicolas.du...@univ-orleans.fr wrote: Hi everybody, I'm new to Giraph so I have a few questions about how it works and so how to configure it to make it work as well as possible. We have settled a cluster of 6 servers with 24 cpu, 24GB of RAM and we want to use it to experiment with Giraph. Currently, we've made a few runs and we have some problems with memory, it seems that we don't give enough of it to the JVM (GC overhead, OutOfMemory, ...). Our experiments were benchmarks using the PageRank, we only succeed in running it on a 100 millions edges graph by running two virtual machines with 8GB of Ram on each of our server. Here are our questions : - What is the best ? Launching one VM with Giraph on each server and with 20GB of Ram OR launching two of its with 10GB of RAM for each ? - Are there a way to minimize the memory used by Hadoop to give more memory to the Giraph jobs ? - How is the graph distributed across the cluster ? Our graph may be a power-law graph with a few nodes with a very large amount of edges and a lot of nodes with a few edges. How Giraph will distribute this kind of graph ? Does it take in account the number of edges of each vertice ? Thanks in advance, Nicolas Dugué PhD student at the Univeristy of Orléans
Re: Adding rb to approved email addresses?
I tried adding the from emails to the d...@giraph.apache.org mailing list. Shouldn't that work? On 7/16/12 12:17 PM, Jakob Homan wrote: I don't believe so. The from list seems reasonable on each one: -- Forwarded message -- From: Avery Ching avery.ch...@gmail.com To: Avery Ching avery.ch...@gmail.com Cc: giraph giraph-...@incubator.apache.org, Alessandro Presta alessan...@fb.com On Mon, Jul 16, 2012 at 12:15 PM, Owen O'Malley omal...@apache.org wrote: On Mon, Jul 16, 2012 at 12:02 PM, Jakob Homan jgho...@gmail.com wrote: Anyone know what needs to be done to get the automated messages reviewboard is sending out whitelisted on the dev list? We're getting moderation requests for every one... Usually, if you use reply-all, it will bless that sender. Is each user showing up as a different sender? -- Owen
