Hello, We start test our system on Ignite 2.8.1 and got very strange log. As I understand, It was error in system (on 2.7.6 this test works without any problem, I will investigate it later) and it was NPE during handling of this error: [01:08:11] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]] [01:08:11] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. 2020-06-11 01:08:22,677 [tcp-disco-msg-worker-[crd]-#1225%TestNode-0%] WARN :119 - Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-3, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022]]] org.apache.ignite.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-3, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1810) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2858) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7759) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [ignite-core-2.8.1.jar:2.8.1] [01:08:22] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-3, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022]]] 2020-06-11 01:08:22,685 [tcp-disco-msg-worker-[crd]-#1225%TestNode-0%] WARN :119 - Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022]]] org.apache.ignite.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1810) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2858) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7759) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [ignite-core-2.8.1.jar:2.8.1] [01:08:22] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022]]] 2020-06-11 01:08:22,685 [nio-acceptor-tcp-comm-#16311%TestNode-1%] WARN :119 - Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TestNode-1, finished=false, heartbeatTs=1591837692022]]] org.apache.ignite.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TestNode-1, finished=false, heartbeatTs=1591837692022] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1810) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:3024) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2963) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] [01:08:22] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TestNode-1, finished=false, heartbeatTs=1591837692022]]] 2020-06-11 01:08:22,685 [tcp-disco-msg-worker-[crd]-#1225%TestNode-0%] WARN :119 - Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-2, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022]]] org.apache.ignite.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-2, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1810) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2858) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7759) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [ignite-core-2.8.1.jar:2.8.1] [01:08:22] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=grid-nio-worker-tcp-comm-2, igniteInstanceName=TestNode-0, finished=false, heartbeatTs=1591837692022]]] 2020-06-11 01:08:22,686 [nio-acceptor-tcp-comm-#16311%TestNode-1%] ERROR :135 - Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.NullPointerException]] java.lang.NullPointerException: null at org.apache.ignite.internal.processors.diagnostic.DiagnosticProcessor.onFailure(DiagnosticProcessor.java:109) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.failure.FailureProcessor.process(FailureProcessor.java:188) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.failure.FailureProcessor.process(FailureProcessor.java:146) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1808) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:3024) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2963) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] 2020-06-11 01:08:22,687 [nio-acceptor-tcp-comm-#16311%TestNode-1%] ERROR :135 - Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: GridWorker [name=nio-acceptor-tcp-comm, igniteInstanceName=TestNode-1, finished=true, heartbeatTs=1591837692022]]] org.apache.ignite.IgniteException: GridWorker [name=nio-acceptor-tcp-comm, igniteInstanceName=TestNode-1, finished=true, heartbeatTs=1591837692022] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1810) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169) [ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153) [ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] Andrey. |
- NPE during printing failure information Andrey Davydov
- Re: NPE during printing failure information akorensh
- RE: NPE during printing failure information Andrey Davydov
- Re: NPE during printing failure information Ilya Kasnacheev