Hi, Can we keep SYSTEM_WORKER_TERMINATION in ignoredFailureTypes, is there any impact. Want to ignore these casting exceptions instead of the server going down.
2024-03-16T18:37:15,313][ERROR][sys-stripe-56-#57%EDIFCustomerCCNDR%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.ClassCastException: o.a.i.i.processors.cache.distributed.near.GridNearGetRequest cannot be cast to o.a.i.i.GridJobExecuteRequest]] Thanks, Gangaiah ________________________________ From: Gangaiah Gundeboina <gangaiah.gundebo...@ril.com> Sent: 17 March 2024 15:26 To: user@ignite.apache.org <user@ignite.apache.org>; Gangaiah Gundeboina <gangaiah.gundebo...@ril.com> Subject: Re: Server node down due to ClassCastException: o.a.i.i.processors.cache.distributed.near.GridNearGetRequest cannot be cast to o.a.i.i.GridJobExecuteRequest Hi Stephen/Mikhail, We are getting this issue frequently and bit urgent, server node going out of the cluster. Could you please help us. Thanks, Gangaiah ________________________________ From: Gangaiah.Gundeboina.ril.com via user <user@ignite.apache.org> Sent: 16 March 2024 11:29 To: user@ignite.apache.org <user@ignite.apache.org> Subject: [External]Server node down due to ClassCastException: o.a.i.i.processors.cache.distributed.near.GridNearGetRequest cannot be cast to o.a.i.i.GridJobExecuteRequest Caution: The e-mail below is from an external source. Please do not open attachments or click links unless this email comes from a known sender and you know the content is safe. Hi, One of the 8 server node cluster down suddenly, printed error not that much helpful to find the reason. Given below the log entries, please help us here to find the reason. ################################################################################ 2024-03-16T07:18:52,489][INFO ][db-checkpoint-thread-#556%EDIFCustomerMAINDR%][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=90c6ab62-a441-4cd3-8b6d-fbef550fbe4c, startPtr=FileWALPointer [idx=5463768, fileOff=51537417, len=56705], checkpointLockWait=0ms, checkpointLockHoldTime=21ms, walCpRecordFsyncDuration=4ms, pages=62810, reason='timeout'] 2024-03-16T07:18:53,565][INFO ][db-checkpoint-thread-#556%EDIFCustomerMAINDR%][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=90c6ab62-a441-4cd3-8b6d-fbef550fbe4c, pages=62810, markPos=FileWALPointer [idx=5463768, fileOff=51537417, len=56705], walSegmentsCleared=4, walSegmentsCovered=[5463764 - 5463767], markDuration=43ms, pagesWrite=491ms, fsync=585ms, total=1119ms] 2024-03-16T07:19:11,191][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=5463768, segIdx=8, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463768.wal] 2024-03-16T07:19:11,318][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463768.wal] 2024-03-16T07:19:33,988][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=5463769, segIdx=9, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463769.wal] 2024-03-16T07:19:34,121][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463769.wal] 2024-03-16T07:20:18,852][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=5463770, segIdx=0, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463770.wal] 2024-03-16T07:20:19,000][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463770.wal] 2024-03-16T07:21:15,373][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=5463771, segIdx=1, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000001.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463771.wal] 2024-03-16T07:21:15,500][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000001.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463771.wal] 2024-03-16T07:21:52,493][INFO ][db-checkpoint-thread-#556%EDIFCustomerMAINDR%][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=d12adda8-2202-4192-bbeb-f5083d3eabc9, startPtr=FileWALPointer [idx=5463772, fileOff=44231024, len=56705], checkpointLockWait=0ms, checkpointLockHoldTime=25ms, walCpRecordFsyncDuration=5ms, pages=61176, reason='timeout'] 2024-03-16T07:21:53,433][INFO ][db-checkpoint-thread-#556%EDIFCustomerMAINDR%][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=d12adda8-2202-4192-bbeb-f5083d3eabc9, pages=61176, markPos=FileWALPointer [idx=5463772, fileOff=44231024, len=56705], walSegmentsCleared=4, walSegmentsCovered=[5463768 - 5463771], markDuration=45ms, pagesWrite=485ms, fsync=454ms, total=984ms] 2024-03-16T07:22:12,689][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=5463772, segIdx=2, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000002.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463772.wal] 2024-03-16T07:22:12,855][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000002.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463772.wal] 2024-03-16T07:22:46,007][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=5463773, segIdx=3, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000003.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463773.wal] 2024-03-16T07:22:46,135][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000003.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463773.wal] 2024-03-16T07:23:21,043][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=5463774, segIdx=4, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000004.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463774.wal] 2024-03-16T07:23:21,176][INFO ][wal-file-archiver%EDIFCustomerMAINDR-#466%EDIFCustomerMAINDR%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000004.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000005463774.wal] [2024-03-16T07:23:30,071][ERROR][sys-stripe-101-#102%EDIFCustomerMAINDR%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.ClassCastException: o.a.i.i.processors.cache.distributed.near.GridNearGetRequest cannot be cast to o.a.i.i.GridJobExecuteRequest]] java.lang.ClassCastException: org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetRequest cannot be cast to org.apache.ignite.internal.GridJobExecuteRequest at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) ~[ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) [ignite-core-2.7.6.jar:2.7.6] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.6.jar:2.7.6] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_351] 2024-03-16T07:23:30,071][ERROR][sys-stripe-101-#102%EDIFCustomerMAINDR%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.ClassCastException: o.a.i.i.processors.cache.distributed.near.GridNearGetRequest cannot be cast to o.a.i.i.GridJobExecuteRequest]] 2024-03-16T07:23:30,080][WARN ][sys-stripe-101-#102%EDIFCustomerMAINDR%][FailureProcessor] No deadlocked threads detected. 2024-03-16T07:23:32,895][WARN ][jvm-pause-detector-worker][IgniteKernal%EDIFCustomerMAINDR] Possible too long JVM pause: 2809 milliseconds. 2024-03-16T07:23:32,912][WARN ][sys-stripe-101-#102%EDIFCustomerMAINDR%][FailureProcessor] Thread dump at 2024/03/16 07:23:32 IST Thread [name="JMX server connection timeout 185576", id=185576, state=TIMED_WAITING, blockCnt=191, waitCnt=192] Lock [object=[I@20ce31fa, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168) at java.lang.Thread.run(Thread.java:750) Thread [name="sys-#89619%EDIFCustomerMAINDR%", id=185575, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2fa3cce5, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750).............................. [2024-03-16T07:23:32,919][ERROR][sys-stripe-101-#102%EDIFCustomerMAINDR%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.ClassCastException: o.a.i.i.processors.cache.distributed.near.GridNearGetRequest cannot be cast to o.a.i.i.GridJobExecuteRequest]] 2024-03-16T07:23:32,919][ERROR][sys-stripe-101-#102%EDIFCustomerMAINDR%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.ClassCastException: o.a.i.i.processors.cache.distributed.near.GridNearGetRequest cannot be cast to o.a.i.i.GridJobExecuteRequest]] ################################################################################################# Thanks, Gangaiah "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."