Hi,
I am trying to implement a mechanism so that clients can recover from a full
cluster shutdown.
Basically on CLIENT_RECONNECTION, my code closes a QueryCursor and then
tries issuing another query.
The reconnection handling code is executed by a "disco-event-worker-#..."
thread; and when executing IgniteKernal.getOrCreateCache (from the discovery
thread); the call never returns. (stacktrace below).
If the very same code is executed asynchronously (not in this
"discovery-worker" thread), then the IgniteKernal.getOrCreateCache executes
normally.
I suspect the cause of this problem is that there some sort of deadlock,
caused by calling getOrCreateCache from this thread before the reconnection
event (that we are processing) is forwarded to all other listeners.
Would this make sense/is that expected?
Thanks!
*Register to the reconnection event*
KeepAliveQueryListener keepAliveListener = new
KeepAliveQueryListener(ignite, queryBuilder, listener);
ignite.events().localListen(keepAliveListener,
EventType.EVT_CLIENT_NODE_RECONNECTED);
*event handling:*
class KeepAliveQueryListener { ...
@Override
public boolean apply(DiscoveryEvent de) {
exec(() -> {
logger.info("on event: type={} ", de.name());
if (de.type() == EventType.EVT_CLIENT_NODE_RECONNECTED) {
logger.warn("restarting CQ on client reconnection..");
startQuery();
logger.warn("restarted CQ");
}
});
return true;
}
}
private static final Executor es = Executors.newFixedThreadPool(4);
private void exec(Runnable r) {
// asynch vs sync
r.run();
//es.execute(r);
}
class QueryBuilder { ...
public IgniteCache<BinaryObject, BinaryObject> getCache() {
logger.info("getCache name={} called..", table);
IgniteCache<BinaryObject, BinaryObject> res =
ignite.getOrCreateCache(table).withKeepBinary();
logger.info("getCache name={} done", table);
return res;
}
}
*stacktrace*
"disco-event-worker-#122%listener-client%" #167 prio=5 os_prio=0
tid=0x000000002019f000 nid=0x7934 waiting on condition [0x0000000040fbe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3107)
at cachelistener.impl.ignite.QueryBuilder.getCache(QueryBuilder.java:85)
at
cachelistener.impl.ignite.KeepAliveQueryListener.startQuery(KeepAliveQueryListener.java:43)
at
cachelistener.impl.ignite.KeepAliveQueryListener.lambda$0(KeepAliveQueryListener.java:67)
at
cachelistener.impl.ignite.KeepAliveQueryListener$$Lambda$28/1019213101.run(Unknown
Source)
at
cachelistener.impl.ignite.KeepAliveQueryListener.exec(KeepAliveQueryListener.java:56)
at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:62)
at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:1)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1477)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:873)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:858)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2703)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2920)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2732)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
