Re: JVM Halt on Null Pointer Exception in GridDhtTxAbstractEnlistFuture
Hello! Do you happen to use SQL MVCC? https://apacheignite.readme.io/docs/multiversion-concurrency-control#section-multiversion-concurrency-control Looks like you've stumbled on a previously unknown issue. Can you please file a ticket against Apache Ignite JIRA, with some description of steps to reproduce? Regards, -- Ilya Kasnacheev чт, 23 мая 2019 г. в 12:19, garima.j : > Hello, > > We have a 3 node cluster in production for Apache Ignite version 2.7. RAM > 128GB. A Spark streaming service (with thick Ignite clients) writes data > into Ignite cache in a transaction (first get and then put). > Now, 3 spark clients stopped and hence NODE_FAILED events were received. My > Ignite crashed on one node and had the below stack trace : > > [2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near > enlist > request from unknown node (will ignore) [txId=GridCacheVersion > [topVer=169659586, order=1558471024158, nodeOrder=23], > node=1be3bce3-7220-45bc-9863-4f16d97ea22b] > [2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager] > Failed > processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b, > msg=GridNearTxEnlistRequest [threadId=5872, > futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false, > miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b, > topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0], > lockVer=GridCacheVersion [topVer=169659586, order=1558471024158, > nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240, > crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176], > cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000, > taskNameHash=0, op=UPSERT, needRes=false]] > java.lang.NullPointerException: null > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.(GridDhtTxAbstractEnlistFuture.java:237) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.(GridDhtTxEnlistFuture.java:84) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) > [ignite-core-2.7.0.jar:2.7.0] > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > [ignite-core-2.7.0.jar:2.7.0] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] > [2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error > detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler > [ignoredFailure
Re: JVM Halt on Null Pointer Exception in GridDhtTxAbstractEnlistFuture
Hey, Please try to disable the critical failures handler as written here: http://apache-ignite-users.70518.x6.nabble.com/Error-Running-Gridgain-s-LoadCaches-java-application-td28276.html Let us know if it works. - Denis On Thu, May 23, 2019 at 2:19 AM garima.j wrote: > Hello, > > We have a 3 node cluster in production for Apache Ignite version 2.7. RAM > 128GB. A Spark streaming service (with thick Ignite clients) writes data > into Ignite cache in a transaction (first get and then put). > Now, 3 spark clients stopped and hence NODE_FAILED events were received. My > Ignite crashed on one node and had the below stack trace : > > [2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near > enlist > request from unknown node (will ignore) [txId=GridCacheVersion > [topVer=169659586, order=1558471024158, nodeOrder=23], > node=1be3bce3-7220-45bc-9863-4f16d97ea22b] > [2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager] > Failed > processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b, > msg=GridNearTxEnlistRequest [threadId=5872, > futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false, > miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b, > topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0], > lockVer=GridCacheVersion [topVer=169659586, order=1558471024158, > nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240, > crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176], > cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000, > taskNameHash=0, op=UPSERT, needRes=false]] > java.lang.NullPointerException: null > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.(GridDhtTxAbstractEnlistFuture.java:237) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.(GridDhtTxEnlistFuture.java:84) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) > [ignite-core-2.7.0.jar:2.7.0] > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > [ignite-core-2.7.0.jar:2.7.0] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] > [2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error > detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler > [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], > failureCtx=FailureContext [type=CRITICAL_ERROR, > err=java.lang.NullPo
JVM Halt on Null Pointer Exception in GridDhtTxAbstractEnlistFuture
Hello, We have a 3 node cluster in production for Apache Ignite version 2.7. RAM 128GB. A Spark streaming service (with thick Ignite clients) writes data into Ignite cache in a transaction (first get and then put). Now, 3 spark clients stopped and hence NODE_FAILED events were received. My Ignite crashed on one node and had the below stack trace : [2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near enlist request from unknown node (will ignore) [txId=GridCacheVersion [topVer=169659586, order=1558471024158, nodeOrder=23], node=1be3bce3-7220-45bc-9863-4f16d97ea22b] [2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager] Failed processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b, msg=GridNearTxEnlistRequest [threadId=5872, futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false, miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b, topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0], lockVer=GridCacheVersion [topVer=169659586, order=1558471024158, nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240, crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176], cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000, taskNameHash=0, op=UPSERT, needRes=false]] java.lang.NullPointerException: null at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.(GridDhtTxAbstractEnlistFuture.java:237) ~[ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.(GridDhtTxEnlistFuture.java:84) ~[ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061) ~[ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112) ~[ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229) ~[ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227) ~[ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) [ignite-core-2.7.0.jar:2.7.0] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.0.jar:2.7.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] [2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]] java.lang.NullPointerException: null Please help and let me know why this failure happened. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/