[
https://issues.apache.org/jira/browse/GIRAPH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis Larroque updated GIRAPH-1101:
---------------------------------------
Description:
If two workers (or more) are proccesing the same vertice on same superstep (for
example, doing mulple BFS at the same time, could lead to it, depending of the
data of course), the entire superstep hangs, every workers say something like
this:
{quote}
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Future result not
ready yet java.util.concurrent.FutureTask@23a1ef14
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Future result not
ready yet java.util.concurrent.FutureTask@23a1ef14
{quote}
And the master says:
{quote}
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got
finished worker list = [], size = 0, worker list =
[Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1,
port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal,
MRtaskID=2, port=30002),
Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3,
port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal,
MRtaskID=5, port=30005)], size = 4 from
/_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got
finished worker list = [], size = 0, worker list =
[Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1,
port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal,
MRtaskID=2, port=30002),
Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3,
port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal,
MRtaskID=5, port=30005)], size = 4 from
/_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: 0 out of 4
workers finished on superstep 4 on path
/_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: Waiting on
[ip-172-31-23-12.sa-east-1.compute.internal_2,
ip-172-31-23-9.sa-east-1.compute.internal_5,
ip-172-31-23-11.sa-east-1.compute.internal_3,
ip-172-31-23-9.sa-east-1.compute.internal_1]
16/07/29 21:48:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52{quote}
was:
If two workers (or more) are proccesing the same vertice on same superstep (for
example, doing mulple BFS at the same time, could lead to it, depending of the
data of course), the entire superstep hangs, every workers say something like
this:
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Future result not
ready yet java.util.concurrent.FutureTask@23a1ef14
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Future result not
ready yet java.util.concurrent.FutureTask@23a1ef14
And the master says:
{quote}
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got
finished worker list = [], size = 0, worker list =
[Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1,
port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal,
MRtaskID=2, port=30002),
Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3,
port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal,
MRtaskID=5, port=30005)], size = 4 from
/_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
{/quote}
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got
finished worker list = [], size = 0, worker list =
[Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1,
port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal,
MRtaskID=2, port=30002),
Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3,
port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal,
MRtaskID=5, port=30005)], size = 4 from
/_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: 0 out of 4
workers finished on superstep 4 on path
/_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: Waiting on
[ip-172-31-23-12.sa-east-1.compute.internal_2,
ip-172-31-23-9.sa-east-1.compute.internal_5,
ip-172-31-23-11.sa-east-1.compute.internal_3,
ip-172-31-23-9.sa-east-1.compute.internal_1]
16/07/29 21:48:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
> Giraph hangs indefinitely when two or more workers process the same vertice
> on the same superstep
> -------------------------------------------------------------------------------------------------
>
> Key: GIRAPH-1101
> URL: https://issues.apache.org/jira/browse/GIRAPH-1101
> Project: Giraph
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Jose Luis Larroque
> Priority: Minor
>
> If two workers (or more) are proccesing the same vertice on same superstep
> (for example, doing mulple BFS at the same time, could lead to it, depending
> of the data of course), the entire superstep hangs, every workers say
> something like this:
> {quote}
> 16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Future result not
> ready yet java.util.concurrent.FutureTask@23a1ef14
> 16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Waiting for
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
> 16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Future result not
> ready yet java.util.concurrent.FutureTask@23a1ef14
> {quote}
> And the master says:
> {quote}
> 16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
> MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
> 16/07/29 21:43:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got
> finished worker list = [], size = 0, worker list =
> [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1,
> port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal,
> MRtaskID=2, port=30002),
> Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3,
> port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal,
> MRtaskID=5, port=30005)], size = 4 from
> /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
> 16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
> MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
> 16/07/29 21:43:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:48:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got
> finished worker list = [], size = 0, worker list =
> [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1,
> port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal,
> MRtaskID=2, port=30002),
> Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3,
> port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal,
> MRtaskID=5, port=30005)], size = 4 from
> /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
> 16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: 0 out of
> 4 workers finished on superstep 4 on path
> /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
> 16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: Waiting
> on [ip-172-31-23-12.sa-east-1.compute.internal_2,
> ip-172-31-23-9.sa-east-1.compute.internal_5,
> ip-172-31-23-11.sa-east-1.compute.internal_3,
> ip-172-31-23-9.sa-east-1.compute.internal_1]
> 16/07/29 21:48:19 INFO yarn.GiraphYarnTask: [STATUS: task-0]
> MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
> 16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of
> false
> 16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Waiting for
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
