Hello,
Getting below errors continuously on Ignite node. Please help me with
explaining the below exception when it occurs and why?
ERROR org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - Failed
to send message to remote node [node=TcpDiscoveryNode
[id=6cc31b37-7870-413a-b75a-95b4371fc542,
consistentId=mtnclzcom_1679291151812, addrs=ArrayList [10.233.102.12,
127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, /10.233.102.12:47500,
mtn-kafka-stream-6b74bc74f-6d8jw/13.93.181.185:47500], discPort=47500,
order=2, intOrder=2, lastExchangeTime=1679291157038, loc=false,
ver=2.10.0#20210310-sha1:bc24f6ba, isClient=false], msg=GridIoMessage
[plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
skipOnTimeout=false, msg=GridNearTxPrepareRequest
[futId=528e59df681-cef0f6c9-0046-4006-a7c4-9c6514ce6210, miniId=1,
topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1],
subjId=6eaca65c-cf03-42d9-8819-31e0825fa86a, taskNameHash=0, txLbl=null,
flags=[implicitSingle], super=GridDistributedTxPrepareRequest
[threadId=275, concurrency=OPTIMISTIC, isolation=READ_COMMITTED,
writeVer=GridCacheVersion [topVer=290770946, order=1679291313990,
nodeOrder=1], timeout=0, reads=null, writes=PredicateCollectionView
[IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=190,
val=IGNITE_CRUD_ROW_KEY_1234_5678_91011_DLR_CLZ_COM, hasValBytes=true],
cacheId=628281256], val=TxEntryValueHolder [val=com.google.gson.JsonObject
[idHash=85330180, hash=1866240837, members=LinkedHashMap
{ACTIVE_STATUS=com.google.gson.JsonPrimitive [idHash=1159180188,
hash=31429505, value=1], CML_ACCEPTED=com.google.gson.JsonPrimitive
[idHash=1673278339, hash=31429505, value=1],
CML_DATE=com.google.gson.JsonPrimitive [idHash=1909260391,
hash=-1099433161, value=2023-03-20T05:43:23.423]}], op=CREATE],
prevVal=TxEntryValueHolder [val=null, op=NOOP], oldVal=TxEntryValueHolder
[val=null, op=NOOP], entryProcessorsCol=null, ttl=-1,
conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null,
filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=true,
entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry
[super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=190,
val=IGNITE_CRUD_ROW_KEY_1234_5678_91011_DLR_CLZ_COM, hasValBytes=true],
val=null, ver=GridCacheVersion [topVer=0, order=0, nodeOrder=0],
hash=-661118938, extras=null, flags=0]]], prepared=0, locked=false,
nodeId=6cc31b37-7870-413a-b75a-95b4371fc542, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=2, partUpdateCntr=0,
serReadVer=null, xidVer=GridCacheVersion [topVer=290770946,
order=1679291313990, nodeOrder=1]]], dhtVers=null, txSize=0, plc=2,
txState=null, flags=last, super=GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=290770946, order=1679291313990, nodeOrder=1],
committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage
[cacheId=0, super=GridCacheMessage [msgId=120793, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1],
err=null, skipPrepare=false]]]]]]]
org.apache.ignite.IgniteCheckedException: Failed to connect to node due to
unrecoverable exception (is node still alive?). Make sure that each
ComputeTask and cache Transaction has a timeout set in order to prevent
parties from waiting forever in case of network issues
[nodeId=6cc31b37-7870-413a-b75a-95b4371fc542, addrs=[/10.233.102.12:47100,
mtn-kafka-stream-6b74bc74f-6d8jw/13.93.181.185:47100, /127.0.0.1:47100],
err= class org.apache.ignite.IgniteCheckedException: Remote node does not
observe current node in topology : 6cc31b37-7870-413a-b75a-95b4371fc542]
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:627)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:691)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1255)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:689)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:453)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:228)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1186)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1133)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2101)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2196)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1296)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.sendReply(IgniteTxHandler.java:1573)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1327)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:135)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:244)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:242)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.10.0.jar:2.10.0]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to
node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=6cc31b37-7870-413a-b75a-95b4371fc542, addrs=[/10.233.102.12:47100,
mtn-kafka-stream-6b74bc74f-6d8jw/13.93.181.185:47100, /127.0.0.1:47100]]
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:609)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:691)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1255)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:689)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:453)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:228)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1186)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1133)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2101)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2196)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1296)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.sendReply(IgniteTxHandler.java:1573)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1327)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:135)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:244)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:242)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
~[ignite-core-2.10.0.jar:2.10.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.10.0.jar:2.10.0]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
Caused by: java.net.SocketTimeoutException
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:129) ~[?:1.8.0_332]
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:465)
~[ignite-core-2.10.0.jar:2.10.0]
... 30 more
Caused by: org.apache.ignite.IgniteCheckedException: Remote node does not
observe current node in topology : 6cc31b37-7870-413a-b75a-95b4371fc542
at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:505)
~[ignite-core-2.10.0.jar:2.10.0]
... 30 more
--
*Regards,*
*Abhishek Ubhe*
