Hi, I am trying to implement a mechanism so that clients can recover from a full cluster shutdown. Basically on CLIENT_RECONNECTION, my code closes a QueryCursor and then tries issuing another query.
The reconnection handling code is executed by a "disco-event-worker-#..." thread; and when executing IgniteKernal.getOrCreateCache (from the discovery thread); the call never returns. (stacktrace below). If the very same code is executed asynchronously (not in this "discovery-worker" thread), then the IgniteKernal.getOrCreateCache executes normally. I suspect the cause of this problem is that there some sort of deadlock, caused by calling getOrCreateCache from this thread before the reconnection event (that we are processing) is forwarded to all other listeners. Would this make sense/is that expected? Thanks! *Register to the reconnection event* KeepAliveQueryListener keepAliveListener = new KeepAliveQueryListener(ignite, queryBuilder, listener); ignite.events().localListen(keepAliveListener, EventType.EVT_CLIENT_NODE_RECONNECTED); *event handling:* class KeepAliveQueryListener { ... @Override public boolean apply(DiscoveryEvent de) { exec(() -> { logger.info("on event: type={} ", de.name()); if (de.type() == EventType.EVT_CLIENT_NODE_RECONNECTED) { logger.warn("restarting CQ on client reconnection.."); startQuery(); logger.warn("restarted CQ"); } }); return true; } } private static final Executor es = Executors.newFixedThreadPool(4); private void exec(Runnable r) { // asynch vs sync r.run(); //es.execute(r); } class QueryBuilder { ... public IgniteCache<BinaryObject, BinaryObject> getCache() { logger.info("getCache name={} called..", table); IgniteCache<BinaryObject, BinaryObject> res = ignite.getOrCreateCache(table).withKeepBinary(); logger.info("getCache name={} done", table); return res; } } *stacktrace* "disco-event-worker-#122%listener-client%" #167 prio=5 os_prio=0 tid=0x000000002019f000 nid=0x7934 waiting on condition [0x0000000040fbe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140) at org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3107) at cachelistener.impl.ignite.QueryBuilder.getCache(QueryBuilder.java:85) at cachelistener.impl.ignite.KeepAliveQueryListener.startQuery(KeepAliveQueryListener.java:43) at cachelistener.impl.ignite.KeepAliveQueryListener.lambda$0(KeepAliveQueryListener.java:67) at cachelistener.impl.ignite.KeepAliveQueryListener$$Lambda$28/1019213101.run(Unknown Source) at cachelistener.impl.ignite.KeepAliveQueryListener.exec(KeepAliveQueryListener.java:56) at cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:62) at cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:1) at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1477) at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:873) at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:858) at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341) at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2703) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2920) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2732) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:748) -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/