Re: Is Zookeeper a must for Giraph?
Zookeeper is required. That being said, you can have an external Zookeeper or Giraph can start one for you. It's your choice. Eli is the one to contact regarding Giraph on Hadoop 2.0.5. Any thoughts Eli? Avery On 6/24/13 5:22 PM, Chuan Lei wrote: It is not clear to me that whether Zookeeper is required or optional to Giraph. I wonder if it is possible to run Giraph without Zookeeper. If not the case, would the default Zookeeper work with Giraph? Is there anything has to be changed on Zookeeper. Another question is that I have the following error message when I ran Giraph on Hadoop-2.0.5 when I ran the PageRankBenchmark program. I saw similar posts on the mailing list, but it seems no clear answer to it yet. I would be grateful if someone can answer my question and resolve the issue. Error: java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_1372108933881_0002/_zkServer does not exist. at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153) Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_1372108933881_0002/_zkServer does not exist. at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:790) at org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:357) at org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:188) at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:60) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:90) ... 7 more Caused by: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_1372108933881_0002/_zkServer does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:405) at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:749) ... 11 more Regards, Chuan
Is Zookeeper a must for Giraph?
It is not clear to me that whether Zookeeper is required or optional to Giraph. I wonder if it is possible to run Giraph without Zookeeper. If not the case, would the default Zookeeper work with Giraph? Is there anything has to be changed on Zookeeper. Another question is that I have the following error message when I ran Giraph on Hadoop-2.0.5 when I ran the PageRankBenchmark program. I saw similar posts on the mailing list, but it seems no clear answer to it yet. I would be grateful if someone can answer my question and resolve the issue. Error: java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_1372108933881_0002/_zkServer does not exist. at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153) Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_1372108933881_0002/_zkServer does not exist. at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:790) at org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:357) at org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:188) at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:60) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:90) ... 7 more Caused by: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_1372108933881_0002/_zkServer does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:405) at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:749) ... 11 more Regards, Chuan
[no subject]
My run method:
args = new String[] {
"/user/root/giraph-input/gooftech.tsv", "/user/root/giraph-output",
"'Shanae Dailey'", "X:2181"};
if (args.length < 4) {
System.err.println(printUsage());
System.exit(1);
}
if (args.length > 4) {
System.err
.println("too many arguments. "
+ "Did you forget to quote the source ID name ('firstname lastname')");
System.exit(1);
}
String input = args[0];
String output = args[1];
String source_id = args[2];
String zooQuorum = args[3];
conf.addResource(new Path("/etc/hadoop/core-site.xml"));
GiraphConfiguration giraphConf = new GiraphConfiguration(getConf());
giraphConf.set(SOURCE_ID, source_id);
GiraphConfiguration.SPLIT_MASTER_WORKER.set(giraphConf, false);
GiraphConfiguration.USE_SUPERSTEP_COUNTERS.set(giraphConf, false);
GiraphConfiguration.CHECKPOINT_FREQUENCY.set(giraphConf, 0);
giraphConf.setVertexInputFormatClass(EmployeeRDFTextInputFormat.class);
giraphConf.setVertexOutputFormatClass(EmployeeShortestPathOutputFormat.class);
giraphConf.setComputationClass(EmployeeShortestPathVertex.class);
giraphConf.setWorkerConfiguration(1, 1, 100.0f);
giraphConf.setZooKeeperConfiguration(zooQuorum);
GiraphJob job = new GiraphJob(giraphConf,
"single-source shortest path for employee: " + source_id);
GiraphFileInputFormat.addVertexInputPath(giraphConf, new Path(input));
FileInputFormat.addInputPath(job.getInternalJob(), new Path(input));
FileOutputFormat.setOutputPath(job.getInternalJob(),
removeAndSetOutput(output));
return job.run(true) ? 0 : 1;
*But I keep getting following exceptions:*
13/06/24 14:57:03 INFO netty.NettyServer: start: Started server
communication server:
v525400ec6f25.hsltv3prod.cloud9.ibm.com/9.12.206.162:3 with up to 16
threads on bind attempt 0 with sendBufferSize = 32768 receiveBufferSize =
524288 backlog = 1
13/06/24 14:57:03 INFO netty.NettyServer: start: Started server
communication server:
v525400ec6f25.hsltv3prod.cloud9.ibm.com/9.12.206.162:30010 with up to 16
threads on bind attempt 1 with sendBufferSize = 32768 receiveBufferSize =
524288 backlog = 1
13/06/24 14:57:03 INFO netty.NettyClient: NettyClient: Using execution
handler with 8 threads after requestEncoder.
13/06/24 14:57:03 INFO netty.NettyClient: NettyClient: Using execution
handler with 8 threads after requestEncoder.
13/06/24 14:57:03 INFO master.BspServiceMaster: becomeMaster: I am now the
master!
13/06/24 14:57:03 INFO graph.GraphTaskManager: setup: Registering health of
this worker...
13/06/24 14:57:03 INFO master.BspServiceMaster:
/_hadoopBsp/job_local_0001/_vertexInputSplitDir already exists, no need to
create
13/06/24 14:57:03 INFO bsp.BspService: getJobState: Job state already
exists (/_hadoopBsp/job_local_0001/_masterJobState)
13/06/24 14:57:03 ERROR master.MasterThread: masterThread: Master algorithm
failed with NullPointerException
java.lang.NullPointerException
at java.lang.String.(String.java:593)
at
org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:598)
at
org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:693)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:99)
13/06/24 14:57:03 FATAL graph.GraphMapper: uncaughtException:
OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread,
msg = java.lang.NullPointerException, exiting...
java.lang.IllegalStateException: java.lang.NullPointerException
at org.apache.giraph.master.MasterThread.run(MasterThread.java:180)
Caused by: java.lang.NullPointerException
at java.lang.String.(String.java:593)
at
org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:598)
at
org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:693)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:99)
13/06/24 14:57:03 INFO bsp.BspService: getApplicationAttempt: Node
/_hadoopBsp/job_local_0001/_applicationAttemptsDir already exists!
13/06/24 14:57:03 INFO bsp.BspService: getApplicationAttempt: Node
/_hadoopBsp/job_local_0001/_applicationAttemptsDir already exists!
13/06/24 14:57:03 INFO messages.InMemoryMessageStoreFactory: newStore:
Using ByteArrayMessagesPerVertexStore since there is no combiner
13/06/24 14:57:03 INFO worker.BspServiceWorker: registerHealth: Created my
health node for attempt=0, superstep=0 with
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_workerHealthyDir/v525400ec6f25.hsltv3prod.cloud9.ibm.com_0
and workerInfo= Worker(hostname=v525400ec6f25.hsltv3prod.cloud9.ibm.com,
MRtaskID=0, port=30010)
13/06/24 14:57:03 INFO worker.BspServiceWorker: startSuperstep:
Master(hostname=v525400ec6f25.hsltv3prod.cloud9.ibm.com, MRtaskID=0,
port=3)
13/06/24 14:57:03 INFO worker.BspServiceWorker: startSuperstep: Ready for
computation on superstep 0 since worker selection and vertex range
assignments are done in
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_addressesAndPartitions
13/06/24 14:57:03 INFO netty.NettyClient: Using Netty
Re: Zeros appended to BytesWritable
This was the problem. Here a possible work around:
public static class MyBytes extends BytesWritable {
public MyBytes() {
super();
}
public MyBytes(byte[] data) {
super(data);
}
/**
* Set the size.
* @param size The new size.
*/
@Override
public void setSize(int size) {
boolean resize = size > getCapacity();
super.setSize(size);
if (resize) {
setCapacity(size);
}
}
}
2013/6/24 Christian Krause
> It seems this is related to this issue:
> http://stackoverflow.com/questions/15957870/why-byteswritable-setsizesize-make-the-space-of-bytes-to-1-5size
>
> But I am using the byte[] constructor, so I still don't get it...
>
> Christian
>
>
> 2013/6/24 Christian Krause
>
>> Hi,
>> I am using BytesWritables as messages and vertex IDs. The strange thing
>> is when I send a message or create a new vertex, a 0 is appended to my
>> original byte array. From my log I can see that I am sending the correct
>> data and in the next superstep I get my array with an appended 0. Does
>> anybody have an idea what might be causing that?
>>
>> Cheers,
>> Christian
>>
>
>
Re: Zeros appended to BytesWritable
It seems this is related to this issue: http://stackoverflow.com/questions/15957870/why-byteswritable-setsizesize-make-the-space-of-bytes-to-1-5size But I am using the byte[] constructor, so I still don't get it... Christian 2013/6/24 Christian Krause > Hi, > I am using BytesWritables as messages and vertex IDs. The strange thing is > when I send a message or create a new vertex, a 0 is appended to my > original byte array. From my log I can see that I am sending the correct > data and in the next superstep I get my array with an appended 0. Does > anybody have an idea what might be causing that? > > Cheers, > Christian >
Re: problem with running shortestpath example.
I am also a Giraph beginner. The default addresses of the web interfaces of your master nodes are localhost:50030 and localhost:50070 (assuming your localhost is your master). Just open them in a browser... Cheers, Christian
Re: problem with running shortestpath example.
Thank you all. However, Christian, could you explain more specific for me? Because I'm beginner :) please. 2013/6/24 Christian Krause > Hi, > You have a failed map task. Take a look at the web interface of the Job > Tracker. Open the logs of your job and look for exceptions. > > Cheers, > Christian > > > 2013/6/24 김원배 > >> Hi, I'm beginner on giraph. >> I have a problem with running shortestpath example. >> >> I'm using giraph-1.0.0 >> hadoop 0.20.203.0 >> maven 3.1.0-alpha-1 >> java 1.6.0_22 >> >> >> when I command like this, >> >> >> hadoop jar \ >> /home/wb/giraph-1.0.0/giraph-core/target/giraph-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar >> \ >> org.apache.giraph.GiraphRunner \ >> org.apache.giraph.examples.SimpleShortestPathsVertex \ >> -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ >> -vip ./in/input \ >> -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ >> -op ./outShortest \ >> -w 1 >> >> >> I get this result >> >> >> Exception in thread "main" java.lang.ClassNotFoundException: >> org.apache.giraph.examples.SimpleShortestPathsVertex >> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:186) >> at >> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:210) >> at >> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:147) >> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:616) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> >> >> >> >> and when I command like this, >> >> >> hadoop jar \ >> /home/wb/giraph-1.0.0/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar >> \ >> org.apache.giraph.GiraphRunner \ >> org.apache.giraph.examples.SimpleShortestPathsVertex \ >> -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ >> -vip /in/input \ >> -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ >> -op /outShortest \ >> -w 1 >> >> >> I get this result. >> >> >> 13/06/24 16:40:54 INFO utils.ConfigurationUtils: No edge input format >> specified. Ensure your InputFormat does not require one. >> 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format >> vertex index type is not known >> 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format >> vertex value type is not known >> 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format >> edge value type is not known >> 13/06/24 16:40:54 INFO job.GiraphJob: run: Since checkpointing is >> disabled (default), do not allow any task retries (setting >> mapred.map.max.attempts = 0, old value = 4) >> 13/06/24 16:40:56 INFO mapred.JobClient: Running job: >> job_201306241432_0005 >> 13/06/24 16:40:57 INFO mapred.JobClient: map 0% reduce 0% >> 13/06/24 16:41:18 INFO mapred.JobClient: map 50% reduce 0% >> 13/06/24 16:41:21 INFO mapred.JobClient: map 100% reduce 0% >> 13/06/24 16:51:29 INFO mapred.JobClient: map 50% reduce 0% >> 13/06/24 16:51:37 INFO mapred.JobClient: Job complete: >> job_201306241432_0005 >> 13/06/24 16:51:37 INFO mapred.JobClient: Counters: 6 >> 13/06/24 16:51:37 INFO mapred.JobClient: Job Counters >> 13/06/24 16:51:37 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1242845 >> 13/06/24 16:51:37 INFO mapred.JobClient: Total time spent by all >> reduces waiting after reserving slots (ms)=0 >> 13/06/24 16:51:37 INFO mapred.JobClient: Total time spent by all maps >> waiting after reserving slots (ms)=0 >> 13/06/24 16:51:37 INFO mapred.JobClient: Launched map tasks=2 >> 13/06/24 16:51:37 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 >> 13/06/24 16:51:37 INFO mapred.JobClient: Failed map tasks=1 >> >> >> I cannot find out what's the probelm. Thanks. >> >> >
Re: problem with running shortestpath example.
Hi, You have a failed map task. Take a look at the web interface of the Job Tracker. Open the logs of your job and look for exceptions. Cheers, Christian 2013/6/24 김원배 > Hi, I'm beginner on giraph. > I have a problem with running shortestpath example. > > I'm using giraph-1.0.0 > hadoop 0.20.203.0 > maven 3.1.0-alpha-1 > java 1.6.0_22 > > > when I command like this, > > > hadoop jar \ > /home/wb/giraph-1.0.0/giraph-core/target/giraph-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar > \ > org.apache.giraph.GiraphRunner \ > org.apache.giraph.examples.SimpleShortestPathsVertex \ > -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ > -vip ./in/input \ > -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ > -op ./outShortest \ > -w 1 > > > I get this result > > > Exception in thread "main" java.lang.ClassNotFoundException: > org.apache.giraph.examples.SimpleShortestPathsVertex > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:321) > at java.lang.ClassLoader.loadClass(ClassLoader.java:266) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:186) > at > org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:210) > at > org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:147) > at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > > > > > and when I command like this, > > > hadoop jar \ > /home/wb/giraph-1.0.0/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar > \ > org.apache.giraph.GiraphRunner \ > org.apache.giraph.examples.SimpleShortestPathsVertex \ > -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ > -vip /in/input \ > -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ > -op /outShortest \ > -w 1 > > > I get this result. > > > 13/06/24 16:40:54 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format > vertex index type is not known > 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format > vertex value type is not known > 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format > edge value type is not known > 13/06/24 16:40:54 INFO job.GiraphJob: run: Since checkpointing is disabled > (default), do not allow any task retries (setting mapred.map.max.attempts = > 0, old value = 4) > 13/06/24 16:40:56 INFO mapred.JobClient: Running job: job_201306241432_0005 > 13/06/24 16:40:57 INFO mapred.JobClient: map 0% reduce 0% > 13/06/24 16:41:18 INFO mapred.JobClient: map 50% reduce 0% > 13/06/24 16:41:21 INFO mapred.JobClient: map 100% reduce 0% > 13/06/24 16:51:29 INFO mapred.JobClient: map 50% reduce 0% > 13/06/24 16:51:37 INFO mapred.JobClient: Job complete: > job_201306241432_0005 > 13/06/24 16:51:37 INFO mapred.JobClient: Counters: 6 > 13/06/24 16:51:37 INFO mapred.JobClient: Job Counters > 13/06/24 16:51:37 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1242845 > 13/06/24 16:51:37 INFO mapred.JobClient: Total time spent by all > reduces waiting after reserving slots (ms)=0 > 13/06/24 16:51:37 INFO mapred.JobClient: Total time spent by all maps > waiting after reserving slots (ms)=0 > 13/06/24 16:51:37 INFO mapred.JobClient: Launched map tasks=2 > 13/06/24 16:51:37 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 > 13/06/24 16:51:37 INFO mapred.JobClient: Failed map tasks=1 > > > I cannot find out what's the probelm. Thanks. > >
Re: problem with running shortestpath example.
Hi, You have to specify EdgeInputFormat. For another class of mine I use the following command (wrting for just reference): hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.LinkRank.LinkRankVertex -vif org.apache.giraph.examples.LinkRank.LinkRankVertexInputFormat -eif org.apache.giraph.examples.LinkRank.LinkRankEdgeInputFormat -of org.apache.giraph.examples.LinkRank.LinkRankVertexOutputFormat -w 1 -vip nodes.txt -eip edges.txt -op linkOutput4 You can see the sample EdgeInputFormat on http://bit.ly/174RfHG. On 06/24/2013 11:33 AM, 김원배 wrote: and when I command like this, hadoop jar \ /home/wb/giraph-1.0.0/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar \ org.apache.giraph.GiraphRunner \ org.apache.giraph.examples.SimpleShortestPathsVertex \ -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ -vip /in/input \ -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ -op /outShortest \ -w 1 I get this result. 13/06/24 16:40:54 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
problem with running shortestpath example.
Hi, I'm beginner on giraph. I have a problem with running shortestpath example. I'm using giraph-1.0.0 hadoop 0.20.203.0 maven 3.1.0-alpha-1 java 1.6.0_22 when I command like this, hadoop jar \ /home/wb/giraph-1.0.0/giraph-core/target/giraph-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar \ org.apache.giraph.GiraphRunner \ org.apache.giraph.examples.SimpleShortestPathsVertex \ -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ -vip ./in/input \ -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ -op ./outShortest \ -w 1 I get this result Exception in thread "main" java.lang.ClassNotFoundException: org.apache.giraph.examples.SimpleShortestPathsVertex at java.net.URLClassLoader$1.run(URLClassLoader.java:217) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang.ClassLoader.loadClass(ClassLoader.java:321) at java.lang.ClassLoader.loadClass(ClassLoader.java:266) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:186) at org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:210) at org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:147) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) and when I command like this, hadoop jar \ /home/wb/giraph-1.0.0/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar \ org.apache.giraph.GiraphRunner \ org.apache.giraph.examples.SimpleShortestPathsVertex \ -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \ -vip /in/input \ -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \ -op /outShortest \ -w 1 I get this result. 13/06/24 16:40:54 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format vertex index type is not known 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format vertex value type is not known 13/06/24 16:40:54 WARN job.GiraphConfigurationValidator: Output format edge value type is not known 13/06/24 16:40:54 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 13/06/24 16:40:56 INFO mapred.JobClient: Running job: job_201306241432_0005 13/06/24 16:40:57 INFO mapred.JobClient: map 0% reduce 0% 13/06/24 16:41:18 INFO mapred.JobClient: map 50% reduce 0% 13/06/24 16:41:21 INFO mapred.JobClient: map 100% reduce 0% 13/06/24 16:51:29 INFO mapred.JobClient: map 50% reduce 0% 13/06/24 16:51:37 INFO mapred.JobClient: Job complete: job_201306241432_0005 13/06/24 16:51:37 INFO mapred.JobClient: Counters: 6 13/06/24 16:51:37 INFO mapred.JobClient: Job Counters 13/06/24 16:51:37 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1242845 13/06/24 16:51:37 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/06/24 16:51:37 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/06/24 16:51:37 INFO mapred.JobClient: Launched map tasks=2 13/06/24 16:51:37 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 13/06/24 16:51:37 INFO mapred.JobClient: Failed map tasks=1 I cannot find out what's the probelm. Thanks.
Zeros appended to BytesWritable
Hi, I am using BytesWritables as messages and vertex IDs. The strange thing is when I send a message or create a new vertex, a 0 is appended to my original byte array. From my log I can see that I am sending the correct data and in the next superstep I get my array with an appended 0. Does anybody have an idea what might be causing that? Cheers, Christian
