Re: Issue in Aggregator
Dear Matthew, Yes my aggregator is commutative, and I want globally aggregated value from all workers, and not the locally aggregated. But I am getting locally aggregated value. My query is : Does Giraph call the aggregator class for the values returned by the local aggregators or not? - PuneetIIT Delhi, India On Sunday, November 9, 2014 5:43 AM, Matthew Saltz sal...@gmail.com wrote: Hi Puneet,It's unclear to me what you're wanting in terms of aggregator behavior. Are you saying you want an aggregator such that the final output is the aggregated value just for a particular worker? With an aggregator you should at least make sure the operations you're performing are commutative; that is, the order in which items are aggregated should not matter unless it is explicitly dealt with somehow. Otherwise you'll get unpredictable results. Best, Matthew SaltzEl 08/11/2014 15:05, Puneet Agarwal puagar...@yahoo.com escribió: Hi All,In my algo, I use an Aggregator which takes a Text value. I have written my custom aggregator class for this, as given below. public class MyAgg extends BasicAggregatorText {...} This works fine when running on my laptop with one worker.However, when running it on the cluster, sometimes it does not return the correctly aggregated value.It seems it is returning the locally aggregated value of one of the workers.While it should have used my logic to decide which of the aggregated values sent by various worker should be chosen as finally aggregated values.(But in fact I have not written such a code anywhere, it is therefore doing the best it could) Following is how is my analysis about this issue.a.I guess every worker aggregates the values locally.b. then there is a global aggregation step, which simply compares the values sent by various aggregators.c. For global aggregation it uses Text.compareTo() method. This method Text.compareTo() is a default Hadoop implementation and does not include the logic of my program.d. It seem it is because of the above the value returned by my aggregator in the cluster is actually not globally aggregated, but the locally aggregated value of one of the worker gets taken. If the above analysis is correct, following is how I think I can solve this.I should write my own class that implements Writable interface. In this class I would also write a compareTo method as a result things will start working fine. If it was using class MyAgg itself, to decide which of the values returned by various workers should be taken as globally aggregated value then this problem would not have occurred. I seek your guidance whether my analysis is correct. - PuneetIIT Delhi, India
Re: Compiling Giraph 1.1
I just built trunk with that command. Are you sure you're building latest trunk? On Fri, Nov 7, 2014 at 3:21 PM, Ryan freelanceflashga...@gmail.com wrote: Any updated thoughts on this? On Tue, Nov 4, 2014 at 5:59 PM, Ryan freelanceflashga...@gmail.com wrote: It's 'mvn -Phadoop_2 -fae -DskipTests clean install' Thanks, Ryan On Tue, Nov 4, 2014 at 2:02 PM, Roman Shaposhnik ro...@shaposhnik.org wrote: What's the exact compilation incantation you use? Thanks, Roman. On Tue, Nov 4, 2014 at 9:56 AM, Ryan freelanceflashga...@gmail.com wrote: I'm attempting to build, compile and install Giraph 1.1 on a server running CDH5.1.2. A few weeks ago I successfully compiled it by changing the hadoop_2 profile version to be 2.3.0-cdh5.1.2. I recently did a fresh install and was unable to build, compile and install (perhaps due to the latest code updates). The error seems to be related to the SaslNettyClient and SaslNettyServer. Any idea on fixes? Here's part of the error log: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure: [ERROR] /[myPath]/giraph/giraph-core/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java:[28,34] cannot find symbol [ERROR] symbol: class SaslPropertiesResolver [ERROR] location: package org.apache.hadoop.security ... [ERROR] /[myPath]/giraph/giraph-core/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java:[108,11] cannot find symbol [ERROR] symbol: variable SaslPropertiesResolver [ERROR] location: class org.apache.giraph.comm.netty.SaslNettyServer -- Claudio Martella
When do Giraph vertices receive their messages?
I am curious about how does Giraph receive messages before processing it I know that they use their accepted messages in the compute() method on the next superstep, but when do they receive it? If it is before the checkpoint process, is there any part in the documentation/code that I can see to understand it? Also, what mechanism that Giraph use to store messages before superstep S+1? Are they store it in a buffer or disk first? I still cannot find anything about this. Regards, Vincentius Martin
Re: When do Giraph vertices receive their messages?
These are some very interesting questions. I also would like to know the answers to these. - PuneetIIT Delhi, India On Monday, November 10, 2014 9:30 AM, Vincentius Martin vincentiusmar...@gmail.com wrote: I am curious about how does Giraph receive messages before processing it I know that they use their accepted messages in the compute() method on the next superstep, but when do they receive it? If it is before the checkpoint process, is there any part in the documentation/code that I can see to understand it?Also, what mechanism that Giraph use to store messages before superstep S+1? Are they store it in a buffer or disk first?I still cannot find anything about this. Regards, Vincentius Martin