Sorry I by accident sent that email before finishing it. I tested the compute method with just:
public void compute(Vertex<IntWritable, IntWritable, NullWritable> vertex, Iterable<IntWritable> messages) throws IOException { //check if its the first superstep if (getSuperstep() == 0) { if(isStart(vertex)) { vertex.setValue(new IntWritable((int) getSuperstep())); for (Edge<IntWritable, NullWritable> edge : vertex.getEdges()) { idQueue.addInt(edge.getTargetVertexId().get()); sendMessage(edge.getTargetVertexId(), new IntWritable(1)); } Collections.sort(idQueue.getArrayList()); //aggregate(ID_AGG, idQueue); } else { vertex.setValue(new IntWritable(Integer.MAX_VALUE)); } } else { } vertex.voteToHalt(); } //inner class public static class SimpleBFSMasterCompute extends MasterCompute { public void readFields(DataInput arg0) throws IOException { // TODO Auto-generated method stub } public void write(DataOutput arg0) throws IOException { // TODO Auto-generated method stub } @Override public void compute() { // TODO Auto-generated method stub } @Override public void initialize() throws InstantiationException, IllegalAccessException { // TODO Auto-generated method stub registerAggregator(ID_AGG, ArrayListAggregator.class); } } and it worked fine, and then tested it with the aggregate method uncommented and produced the same log statements as before. The values I pass into the aggregate method are: public static final String ID_AGG = "simplemastercompute.aggregator"; private MyArrayListWritable idQueue = new MyArrayListWritable(); code: MyArrayListWirtable class: http://pastebin.com/n4iDjp3j ArrayListAggregator class: http://pastebin.com/z7xjpZVU Hopefully this made my issue clearer. Kind regards, Ghufran On Thu, Apr 3, 2014 at 5:34 PM, ghufran malik <ghufran1ma...@gmail.com>wrote: > I just tested the compute method with just: > > > public void compute(Vertex<IntWritable, IntWritable, NullWritable> vertex, > Iterable<IntWritable> messages) throws IOException > { > //check if its the first superstep > if (getSuperstep() == 0) > { > > if(isStart(vertex)) > { > vertex.setValue(new IntWritable((int) getSuperstep())); > for (Edge<IntWritable, NullWritable> edge : vertex.getEdges()) > { > idQueue.addInt(edge.getTargetVertexId().get()); > sendMessage(edge.getTargetVertexId(), new IntWritable(1)); > } > Collections.sort(idQueue.getArrayList()); > aggregate(ID_AGG, idQueue); > } > else > { > vertex.setValue(new IntWritable(Integer.MAX_VALUE)); > } > } > > else{} > vertex.voteToHalt(); > } > > On Thu, Apr 3, 2014 at 5:24 PM, ghufran malik <ghufran1ma...@gmail.com>wrote: > >> After a while of those info jobs been printed this is printed out after: >> >> 14/04/03 17:01:25 INFO zookeeper.ClientCnxn: Unable to read additional >> data from server sessionid 0x145284966610002, likely server has closed >> socket, closing socket connection and attempting reconnect >> 14/04/03 17:01:26 INFO mapred.JobClient: map 50% reduce 0% >> 14/04/03 17:01:27 INFO zookeeper.ClientCnxn: Opening socket connection to >> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using >> SASL (unknown error) >> 14/04/03 17:01:27 WARN zookeeper.ClientCnxn: Session 0x145284966610002 >> for server null, unexpected error, closing socket connection and attempting >> reconnect >> java.net.ConnectException: Connection refused >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) >> 14/04/03 17:01:27 WARN zk.ZooKeeperExt: exists: Connection loss on >> attempt 0, waiting 5000 msecs before retrying. >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for >> /_hadoopBsp/job_201404031649_0001/_workerProgresses >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1069) >> at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:360) >> at >> org.apache.giraph.job.JobProgressTracker$2.run(JobProgressTracker.java:87) >> at java.lang.Thread.run(Thread.java:744) >> 14/04/03 17:01:29 INFO zookeeper.ClientCnxn: Opening socket connection to >> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using >> SASL (unknown error) >> 14/04/03 17:01:29 WARN zookeeper.ClientCnxn: Session 0x145284966610002 >> for server null, unexpected error, closing socket connection and attempting >> reconnect >> java.net.ConnectException: Connection refused >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) >> 14/04/03 17:01:30 INFO zookeeper.ClientCnxn: Opening socket connection to >> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using >> SASL (unknown error) >> 14/04/03 17:01:30 WARN zookeeper.ClientCnxn: Session 0x145284966610002 >> for server null, unexpected error, closing socket connection and attempting >> reconnect >> java.net.ConnectException: Connection refused >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) >> 14/04/03 17:01:31 INFO zookeeper.ClientCnxn: Opening socket connection to >> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using >> SASL (unknown error) >> 14/04/03 17:01:31 WARN zookeeper.ClientCnxn: Session 0x145284966610002 >> for server null, unexpected error, closing socket connection and attempting >> reconnect >> java.net.ConnectException: Connection refused >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) >> 14/04/03 17:01:31 INFO mapred.JobClient: Job complete: >> job_201404031649_0001 >> 14/04/03 17:01:31 INFO mapred.JobClient: Counters: 6 >> 14/04/03 17:01:31 INFO mapred.JobClient: Job Counters >> 14/04/03 17:01:31 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1235189 >> 14/04/03 17:01:31 INFO mapred.JobClient: Total time spent by all >> reduces waiting after reserving slots (ms)=0 >> 14/04/03 17:01:31 INFO mapred.JobClient: Total time spent by all maps >> waiting after reserving slots (ms)=0 >> 14/04/03 17:01:31 INFO mapred.JobClient: Launched map tasks=2 >> 14/04/03 17:01:31 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 >> 14/04/03 17:01:31 INFO mapred.JobClient: Failed map tasks=1 >> >> >> >> On Thu, Apr 3, 2014 at 5:05 PM, ghufran malik <ghufran1ma...@gmail.com>wrote: >> >>> My Giraph job gets stuck at this point and will not go any further, that >>> log is what is continually printed out every 5 seconds. This log is coming >>> from the CombinedWorkerProgress class: >>> >>> else if (isComputeSuperstep()) { >>> sb.append("Compute superstep ").append(currentSuperstep).append(": >>> "); >>> sb.append(verticesComputed).append(" out of ").append( >>> verticesToCompute).append(" vertices computed; "); >>> sb.append(partitionsComputed).append(" out of ").append( >>> partitionsToCompute).append(" partitions computed"); >>> >>> So the data is loaded in fine and then the Giraph job gets stuck in >>> superstep 0 for some reason? >>> >>> public void compute(Vertex<IntWritable, IntWritable, NullWritable> >>> vertex, Iterable<IntWritable> messages) throws IOException >>> { >>> //check if its the first superstep >>> if (getSuperstep() == 0) >>> { >>> >>> if(isStart(vertex)) >>> { >>> vertex.setValue(new IntWritable((int) getSuperstep())); >>> for (Edge<IntWritable, NullWritable> edge : vertex.getEdges()) >>> { >>> idQueue.addInt(edge.getTargetVertexId().get()); >>> sendMessage(edge.getTargetVertexId(), new IntWritable(1)); >>> } >>> Collections.sort(idQueue.getArrayList()); >>> aggregate(ID_AGG, idQueue); >>> } >>> else >>> { >>> vertex.setValue(new IntWritable(Integer.MAX_VALUE)); >>> } >>> } >>> >>> Thats the code I wrote for the first superstep. I ran this code before >>> without the aggregate and it worked, so I think my problem is related the >>> the aggregator/master. >>> >>> Kind regards, >>> >>> Ghufran >>> >>> >>> On Thu, Apr 3, 2014 at 4:40 PM, Rob Vesse <rve...@dotnetrdf.org> wrote: >>> >>>> How is that an error? >>>> >>>> That's just some informational log statements from Giraph, you'll need >>>> to provide the actual error message/describe the issue to get help with >>>> your problem >>>> >>>> Rob >>>> >>>> From: ghufran malik <ghufran1ma...@gmail.com> >>>> Reply-To: <user@giraph.apache.org> >>>> Date: Thursday, 3 April 2014 16:09 >>>> To: <user@giraph.apache.org> >>>> Subject: Master/Agreggators >>>> >>>> Hi, >>>> >>>> I received the error: >>>> >>>> 14/04/03 16:01:07 INFO mapred.JobClient: map 100% reduce 0% >>>> 14/04/03 16:01:11 INFO job.JobProgressTracker: Data from 1 workers - >>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions >>>> computed; min free memory on worker 1 - 106.6MB, average 106.6MB >>>> 14/04/03 16:01:16 INFO job.JobProgressTracker: Data from 1 workers - >>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions >>>> computed; min free memory on worker 1 - 106.6MB, average 106.6MB >>>> 14/04/03 16:01:21 INFO job.JobProgressTracker: Data from 1 workers - >>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions >>>> computed; min free memory on worker 1 - 106.6MB, average 106.6MB >>>> 14/04/03 16:01:26 INFO job.JobProgressTracker: Data from 1 workers - >>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions >>>> computed; min free memory on worker 1 - 106.59MB, average 106.59MB >>>> >>>> >>>> After trying to run a computation class I made that makes use of an >>>> agreggator and master. I remember getting a similar error when I tried >>>> SimplePageRank which also makes uses of a master and a agreggator. >>>> >>>> Does anyone know why I receive this error and how to fix it? >>>> >>>> Kind regards, >>>> >>>> Ghufran >>>> >>>> >>> >> >