Re: Release date for 1.1.0
Hi, Are there any advances so far on the 1.1.0 release schedule? Best, On Wed, Oct 30, 2013 at 12:57 AM, Roman Shaposhnik shaposh...@gmail.comwrote: On Tue, Oct 29, 2013 at 2:22 PM, Artie Pesh-Imam artie.pesh-i...@tapad.com wrote: Will this be published to a mvn repo? That'd be a pretty nice win. I know there's a task for it but its still not done yet. correct? AFAIR, it wasn't done on purpose for 1.0.0 release because of how build munges and also because the underlying hadoop dependencies tend to seep through. It is, however, a very good point to have those artifacts in Maven repo. But they have to be Hadoop-version specific. As a proposed RM for 1.1.0 I'll publish my plan shortly and it'll cover what artifacts I plan to deploy. Thanks, Roman. -- Ahmet Emre Aladağ
Release date for 1.1.0
Hi all, Is there an expected date for 1.1.0? There has been a lot of way taken since 1.0.0. -- Ahmet Emre Aladağ
Only one worker is running
Hi, I have a custom pagerank computation with inputs reading from Hbase and writing to it. I submit my job on a real distributed Hadoop cluster which can allocate 320 map jobs. I started my job with 100 workers. What I see is that only one of the workers are actually reading the input and gets out of memory: readVertexInputSplit: Loaded 200 vertices at 23706.551705289407 vertices/sec 10945068 edges at 129730.61549455763 edges/sec Memory (free/total/max) = 124.31M / 910.25M / 910.25M Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:300) at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:173) ... 16 more Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOfRange(Arrays.java:2694) at java.lang.String.init(String.java:203) at java.lang.String.substring(String.java:1913) at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:171) at java.net.URL.init(URL.java:614) at java.net.URL.init(URL.java:482) and the rest of the workers say: startSuperstep: WORKER_ONLY - Attempt=0, Superstep=-1 Master says: MASTER_ONLY - 99 finished out of 100 on superstep -1 What configuration should I solve this problem? I use: giraphConf.setWorkerConfiguration(1, 100, 85.0f); Thanks,
Multiple records for edges.
Hi, While the reader reads from Hbase, it encounters vertices with the same id. v1 - edges: v3,v5,v6 v2 - edges: ... v1 - edges: v4, v7, v9 ... I want to get finally: v1 - edges: v3,v4,v5,v6,v7,v9 but I get v1 - edges: v3,v5,v6 since vertex.initialize(id, value, edges) will not update the existing vertex. How can I achieve this?
Re: problem with running shortestpath example.
Hi, You have to specify EdgeInputFormat. For another class of mine I use the following command (wrting for just reference): hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.LinkRank.LinkRankVertex -vif org.apache.giraph.examples.LinkRank.LinkRankVertexInputFormat -eif org.apache.giraph.examples.LinkRank.LinkRankEdgeInputFormat -of org.apache.giraph.examples.LinkRank.LinkRankVertexOutputFormat -w 1 -vip nodes.txt -eip edges.txt -op linkOutput4 You can see the sample EdgeInputFormat on http://bit.ly/174RfHG. On 06/24/2013 11:33 AM, 김원배 wrote: and when I command like this, hadoop jar \ /home/wb/giraph-1.0.0/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar \ org.apache.giraph.GiraphRunner \ org.apache.giraph.examples.SimpleShortestPathsVertex \ -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ -vip /in/input \ -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ -op /outShortest \ -w 1 I get this result. 13/06/24 16:40:54 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
Edge value types don't match
Hi, I'm trying to implement LinkRank in Giraph. For now, I started with a copy of PageRank example and trying to run and improve it. My VertexInputFormat: Text Float My EdgeInputFormat: Text Text where Text are URLs and Float is the LinkRank score. Now I'm getting a type mismatch error (below). I could not find the source of the mismatch. When does one get this error? I'm defining the edge weights as NullWritable and Messages as FloatWritable. Here it says at one location I have NullWritable and some place Writable. But I can't see it. Thanks, My source code is: https://github.com/AGMLab/giraph/tree/trunk/giraph-examples/src/main/java/org/apache/giraph/examples/LinkRank [emre@localhost giraph-agmlab]$ hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.LinkRank.LinkRankVertex -vif org.apache.giraph.examples.LinkRank.LinkRankVertexInputFormat -eif org.apache.giraph.examples.LinkRank.LinkRankEdgeInputFormat -of org.apache.giraph.examples.LinkRank.LinkRankVertexOutputFormat -w 1 -vip nodes.txt -eip edges.txt -op linkOutput4 Exception in thread main java.lang.IllegalArgumentException: checkClassTypes: Edge value types don't match, vertex - class org.apache.hadoop.io.NullWritable, vertex input format - interface org.apache.hadoop.io.Writable at org.apache.giraph.job.GiraphConfigurationValidator.verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.java:245) at org.apache.giraph.job.GiraphConfigurationValidator.validateConfiguration(GiraphConfigurationValidator.java:122) at org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:154) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Hadoop version
Hi, I was wondering if newer Hadoop versions like 1.1.2 will be supported soon. Or is it easy to provide support?
