Pavel Kuznetsov created IGNITE-9447:
---------------------------------------
Summary: Benchmarks hangs intemittently due to distributed race
condition.
Key: IGNITE-9447
URL: https://issues.apache.org/jira/browse/IGNITE-9447
Project: Ignite
Issue Type: Bug
Components: sql
Reporter: Pavel Kuznetsov
Assignee: Pavel Kuznetsov
If we run more than one yardstick driver, benchmark hangs intermittently.
We've got yardstick's base driver class
org.apache.ignite.yardstick.IgniteAbstractBenchmark it has logic to wait all
the nodes in the cluster.
{noformat}
final CountDownLatch nodesStartedLatch = new CountDownLatch(1);
ignite().events().localListen(new IgnitePredicate<Event>() {
@Override public boolean apply(Event gridEvt) {
if (nodesStarted())
nodesStartedLatch.countDown();
return true;
}
}, EVT_NODE_JOINED);
if (!nodesStarted()) {
println(cfg, "Waiting for " + (args.nodes() - 1) + " nodes to
start...");
nodesStartedLatch.await();
}
{noformat}
This code is executed on every driver node.
If we want to close local ignite instance just after cluster is ready
(containing expected number of nodes), sometimes we'll have dead lock:
1) cluster contains N-1 nodes, all nodes are waiting for the Nth node.
2) Nth node is connected, cluster receives message, waitForNodes code of Nth
node is not executed.
3) N-1 nodes got this message and stop waiting.
4) N-1 thinks that cluster is ready and call ignite.close() on their local
instances
5) Nth node starts waiting for cluster to contain number of nodes, but N-1 of
them closed their instances
6) Nth node is waiting infinitely.
We can avoid this problem if we use distributed CountDownLatch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
