Re: What this error says, and when it happens (Ignite 2.9.0, Java 8, Ubuntu 14.04)? SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT

2022-04-02 Thread Ilya Shishkov 
Hi Aliaksandr,

Were there any other errors or warnings in the ignite log? Maybe, there
were messages about 'too long JVM pauses'?

пт, 1 апр. 2022 г. в 17:46, Aliaksandr Kazlou :

> Hi, recently in prod we started to observe the following error below.
> What is it? And what could be the cause? Are there any recommendations
> where to look/how to fix?
>
>
> 2022-04-01 12:03:08.535  WARN   --- [server.app%] ROOT
>   : Possible failure suppressed accordingly to a configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
> o.a.i.IgniteException: GridWorker [name=sys-stripe-8, igniteInstanceName=
> server.app, finished=false, heartbeatTs=1648814504489]]]
> org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-8,
> igniteInstanceName=server.app, finished=false, heartbeatTs=1648814504489]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1806)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1801)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$1(ServerImpl.java:2970)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:8057)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3086)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7995)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
> 2022-04-01 12:03:08.541  WARN   --- [er.app%]
> o.a.i.i.p.cache.CacheDiagnosticManager   : Page locks dump:
> Thread=[name=exchange-worker-#67%server.app%, id=115], state=TIMED_WAITING
>

What this error says, and when it happens (Ignite 2.9.0, Java 8, Ubuntu 14.04)? SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT

2022-04-01 Thread Aliaksandr Kazlou 
Hi, recently in prod we started to observe the following error below.
What is it? And what could be the cause? Are there any recommendations
where to look/how to fix?


2022-04-01 12:03:08.535  WARN   --- [server.app%] ROOT
: Possible failure suppressed accordingly to a configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-8, igniteInstanceName=
server.app, finished=false, heartbeatTs=1648814504489]]]
org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-8,
igniteInstanceName=server.app, finished=false, heartbeatTs=1648814504489]
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1806)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1801)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$1(ServerImpl.java:2970)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:8057)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3086)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7995)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
2022-04-01 12:03:08.541  WARN   --- [er.app%]
o.a.i.i.p.cache.CacheDiagnosticManager   : Page locks dump:
Thread=[name=exchange-worker-#67%server.app%, id=115], state=TIMED_WAITING