My run method:
args = new String[] {
"/user/root/giraph-input/gooftech.tsv", "/user/root/giraph-output",
"'Shanae Dailey'", "XXXXXXXXX:2181"};
if (args.length < 4) {
System.err.println(printUsage());
System.exit(1);
}
if (args.length > 4) {
System.err
.println("too many arguments. "
+ "Did you forget to quote the source ID name ('firstname lastname')");
System.exit(1);
}
String input = args[0];
String output = args[1];
String source_id = args[2];
String zooQuorum = args[3];
conf.addResource(new Path("/etc/hadoop/core-site.xml"));
GiraphConfiguration giraphConf = new GiraphConfiguration(getConf());
giraphConf.set(SOURCE_ID, source_id);
GiraphConfiguration.SPLIT_MASTER_WORKER.set(giraphConf, false);
GiraphConfiguration.USE_SUPERSTEP_COUNTERS.set(giraphConf, false);
GiraphConfiguration.CHECKPOINT_FREQUENCY.set(giraphConf, 0);
giraphConf.setVertexInputFormatClass(EmployeeRDFTextInputFormat.class);
giraphConf.setVertexOutputFormatClass(EmployeeShortestPathOutputFormat.class);
giraphConf.setComputationClass(EmployeeShortestPathVertex.class);
giraphConf.setWorkerConfiguration(1, 1, 100.0f);
giraphConf.setZooKeeperConfiguration(zooQuorum);
GiraphJob job = new GiraphJob(giraphConf,
"single-source shortest path for employee: " + source_id);
GiraphFileInputFormat.addVertexInputPath(giraphConf, new Path(input));
FileInputFormat.addInputPath(job.getInternalJob(), new Path(input));
FileOutputFormat.setOutputPath(job.getInternalJob(),
removeAndSetOutput(output));
return job.run(true) ? 0 : 1;
*But I keep getting following exceptions:*
13/06/24 14:57:03 INFO netty.NettyServer: start: Started server
communication server:
v525400ec6f25.hsltv3prod.cloud9.ibm.com/9.12.206.162:30000 with up to 16
threads on bind attempt 0 with sendBufferSize = 32768 receiveBufferSize =
524288 backlog = 1
13/06/24 14:57:03 INFO netty.NettyServer: start: Started server
communication server:
v525400ec6f25.hsltv3prod.cloud9.ibm.com/9.12.206.162:30010 with up to 16
threads on bind attempt 1 with sendBufferSize = 32768 receiveBufferSize =
524288 backlog = 1
13/06/24 14:57:03 INFO netty.NettyClient: NettyClient: Using execution
handler with 8 threads after requestEncoder.
13/06/24 14:57:03 INFO netty.NettyClient: NettyClient: Using execution
handler with 8 threads after requestEncoder.
13/06/24 14:57:03 INFO master.BspServiceMaster: becomeMaster: I am now the
master!
13/06/24 14:57:03 INFO graph.GraphTaskManager: setup: Registering health of
this worker...
13/06/24 14:57:03 INFO master.BspServiceMaster:
/_hadoopBsp/job_local_0001/_vertexInputSplitDir already exists, no need to
create
13/06/24 14:57:03 INFO bsp.BspService: getJobState: Job state already
exists (/_hadoopBsp/job_local_0001/_masterJobState)
13/06/24 14:57:03 ERROR master.MasterThread: masterThread: Master algorithm
failed with NullPointerException
java.lang.NullPointerException
at java.lang.String.<init>(String.java:593)
at
org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:598)
at
org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:693)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:99)
13/06/24 14:57:03 FATAL graph.GraphMapper: uncaughtException:
OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread,
msg = java.lang.NullPointerException, exiting...
java.lang.IllegalStateException: java.lang.NullPointerException
at org.apache.giraph.master.MasterThread.run(MasterThread.java:180)
Caused by: java.lang.NullPointerException
at java.lang.String.<init>(String.java:593)
at
org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:598)
at
org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:693)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:99)
13/06/24 14:57:03 INFO bsp.BspService: getApplicationAttempt: Node
/_hadoopBsp/job_local_0001/_applicationAttemptsDir already exists!
13/06/24 14:57:03 INFO bsp.BspService: getApplicationAttempt: Node
/_hadoopBsp/job_local_0001/_applicationAttemptsDir already exists!
13/06/24 14:57:03 INFO messages.InMemoryMessageStoreFactory: newStore:
Using ByteArrayMessagesPerVertexStore since there is no combiner
13/06/24 14:57:03 INFO worker.BspServiceWorker: registerHealth: Created my
health node for attempt=0, superstep=0 with
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_workerHealthyDir/v525400ec6f25.hsltv3prod.cloud9.ibm.com_0
and workerInfo= Worker(hostname=v525400ec6f25.hsltv3prod.cloud9.ibm.com,
MRtaskID=0, port=30010)
13/06/24 14:57:03 INFO worker.BspServiceWorker: startSuperstep:
Master(hostname=v525400ec6f25.hsltv3prod.cloud9.ibm.com, MRtaskID=0,
port=30000)
13/06/24 14:57:03 INFO worker.BspServiceWorker: startSuperstep: Ready for
computation on superstep 0 since worker selection and vertex range
assignments are done in
/_hadoopBsp/job_local_0001/_applicationAttemptsDir/0/_superstepDir/0/_addressesAndPartitions
13/06/24 14:57:03 INFO netty.NettyClient: Using Netty without
authentication.
13/06/24 14:57:03 INFO netty.NettyServer: start: Using Netty without
authentication.
13/06/24 14:57:03 INFO netty.NettyClient: connectAllAddresses: Successfully
added 1 connections, (1 total connected) 0 failed, 0 failures total.
*I also get binding exception but eventually zookeeper find a available
port:
*
13/06/24 14:57:03 WARN netty.NettyServer: start: Likely failed to bind on
attempt 0 to port 30000
org.jboss.netty.channel.ChannelException: Failed to bind to:
v525400ec6f25.hsltv3prod.cloud9.ibm.com/9.12.206.162:30000
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:298)
at org.apache.giraph.comm.netty.NettyServer.start(NettyServer.java:326)
at
org.apache.giraph.comm.netty.NettyWorkerServer.<init>(NettyWorkerServer.java:101)
at
org.apache.giraph.worker.BspServiceWorker.<init>(BspServiceWorker.java:184)
at
org.apache.giraph.graph.GraphTaskManager.instantiateBspService(GraphTaskManager.java:569)
at org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:207)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.ServerSocketChannelImpl.listen(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:125)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
I am using hadoop 1.1 and Giraph 1.1 (lastest build) on centos
Thanks
--Puneet
