Possible race on node shutdown

[Andrey Davydov]

Hello,   Yesterday we got error on node shutdown while test our system:   2019-12-03 18:49:53,653 [pool-228-thread-1] INFO   r.s.d.m.c.m.PoisonPill:54 - Poison pill works on e45190a6-de52-4e43-b710-1ee8cd1f60a6 2019-12-03 18:49:53,657 [pool-228-thread-1] INFO   r.s.d.m.c.m.ClusterLifecycleBean:61 - IgniteLifecycleBean 331582218 handle event: BEFORE_NODE_STOP 2019-12-03 18:49:53,658 [pool-228-thread-1] INFO   r.s.d.m.c.w.j.JettyStarter:145 - Stop http interface. 2019-12-03 18:49:53,658 [pool-228-thread-1] INFO   r.s.d.m.c.w.j.JettyStarter:149 - Http interface stopped. 2019-12-03 18:49:53,658 [pool-228-thread-1] INFO   r.s.d.m.c.m.ClusterLifecycleBean:105 - IgniteLifecycleBean 331582218 finish handle event: BEFORE_NODE_STOP [18:49:53] Topology snapshot [ver=4, locNode=a8ee74de, servers=2, clients=0, state=INACTIVE, CPUs=4, offheap=7.2GB, heap=4.0GB] [18:49:53] Coordinator changed [prev=TcpDiscoveryNode [id=e45190a6-de52-4e43-b710-1ee8cd1f60a6, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1575398986885, loc=false, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], cur=TcpDiscoveryNode [id=a8ee74de-6d37-4413-9418-08ec903bb974, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1575398993662, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]] [18:49:53]   ^-- Baseline [id=0, size=3, _online_=2, offline=1] [18:49:53] Topology snapshot [ver=4, locNode=67aa9f86, servers=2, clients=0, state=INACTIVE, CPUs=4, offheap=7.2GB, heap=4.0GB] [18:49:53] Coordinator changed [prev=TcpDiscoveryNode [id=e45190a6-de52-4e43-b710-1ee8cd1f60a6, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1575398989326, loc=false, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], cur=TcpDiscoveryNode [id=a8ee74de-6d37-4413-9418-08ec903bb974, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1575398989326, loc=false, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]] [18:49:53]   ^-- Baseline [id=0, size=3, _online_=2, offline=1] 2019-12-03 18:49:53,689 [grid-nio-worker-tcp-comm-1-#1107%TestNode-0%] ERROR  :135 - Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException: null              at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2158) ~[ignite-core-2.7.5.jar:2.7.5]              at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794) [ignite-core-2.7.5.jar:2.7.5]              at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.5.jar:2.7.5]              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] 2019-12-03 18:49:54,033 [grid-nio-worker-tcp-comm-1-#1107%TestNode-0%] ERROR  :127 - JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]]   We can’t find any way for reproduce.   Andrey.