Hello,
We have a 3 node cluster in production for Apache Ignite version 2.7. RAM
128GB. A Spark streaming service (with thick Ignite clients) writes data
into Ignite cache in a transaction (first get and then put).
Now, 3 spark clients stopped and hence NODE_FAILED events were received. My
Ignite crashed on one node and had the below stack trace :
[2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near enlist
request from unknown node (will ignore) [txId=GridCacheVersion
[topVer=169659586, order=1558471024158, nodeOrder=23],
node=1be3bce3-7220-45bc-9863-4f16d97ea22b]
[2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager] Failed
processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
msg=GridNearTxEnlistRequest [threadId=5872,
futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false,
miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
lockVer=GridCacheVersion [topVer=169659586, order=1558471024158,
nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240,
crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176],
cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000,
taskNameHash=0, op=UPSERT, needRes=false]]
java.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.<init>(GridDhtTxAbstractEnlistFuture.java:237)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.<init>(GridDhtTxEnlistFuture.java:84)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.7.0.jar:2.7.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
[2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.NullPointerException]]
java.lang.NullPointerException: null
Please help and let me know why this failure happened.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
