Hi Ignite devs,
*Context*:
We are migrating to Apache Ignite 2.18 as the IMDG technology in our
service. Our service is deployed in kubernetes and is composed by a
deployment of frontend pods that handle user requests (these scale
horizontally according to the demand) and a statefulset composed by 6 pods
(Ignite nodes) on the backend. The frontend pods rely on Ignite thick
clients to communicate with the IMDG, so that we can leverage near-cache
capabilities. The caches atomicity mode are configured to be atomic.
*What happened?*
It seems we had some network instability in the cluster and several
frontend pods lost their connection to the backend. In addition, backend-5
failed, causing topology versions to change. TopVer 1553 was the last
exchange that ever completed when checking backend-0 pod logs (coordinator
at that point):
{"timestamp":"2026-10-22
12:34:20,088","logLevel":"INFO","threadName":"exchange-worker-#53%ibackend-0%","loggerName":"org.apache.ignite.internal.exchange.time","message":"Finished
exchange init [topVer=AffinityTopologyVersion [topVer=1551, minorTopVer=0],
crd=true]"}
{"timestamp":"2026-10-22
12:34:28,185","logLevel":"INFO","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.exchange.time","message":"Finished
exchange init [topVer=AffinityTopologyVersion [topVer=1552, minorTopVer=0],
crd=true]"}
{"timestamp":"2026-10-22
12:34:29,921","logLevel":"INFO","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.exchange.time","message":"Finished
exchange init [topVer=AffinityTopologyVersion [topVer=1553, minorTopVer=0],
crd=true]"}
During the churn, we could see backend-0 pod (node id
45e4c48a-6815-4107-8278-c66f5e9a720b) logging that it was not able to send
messages to the frontend thick-clients, as they were rejected by them. For
example:
*{"timestamp":"2026-10-22
12:34:42,726","logLevel":"ERROR","threadName":"sys-stripe-7-#8%backend-0%","loggerName":"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi","message":"Failed
to send message to remote node [node=TcpDiscoveryNode
[id=06a1fc26-8833-4f1f-afc0-2abdfdd3f4eb,
consistentId=front-z2-5585b6f78b-qs9cl, addrs=ArrayList [<REDACTED>],
sockAddrs=null, discPort=0, order=1533, intOrder=774, loc=false,
ver=2.18.0#20260423-sha1:d49adada, isClient=true, dataCenterId=null],
msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false,
timeout=0, skipOnTimeout=false, msg=GridDhtAtomicSingleUpdateRequest [...],
prevVal=null, super=GridDhtAtomicAbstractUpdateRequest [onRes=false,
nearNodeId=62853a54-8550-43d0-b580-72d1d634b643, nearFutId=671771,
flags=near|hasRes]]]], class org.apache.ignite.IgniteCheckedException:
Failed to connect to node due to unrecoverable exception (is node still
alive?). Make sure that each ComputeTask and cache Transaction has a
timeout set in order to prevent parties from waiting forever in case of
network issues [nodeId=06a1fc26-8833-4f1f-afc0-2abdfdd3f4eb,
addrs=<REDACTED>, err= class org.apache.ignite.IgniteCheckedException:
Remote node does not observe current node in topology :
06a1fc26-8833-4f1f-afc0-2abdfdd3f4eb]*
Our understanding is that *flags=near|hasRes* indicates the coordinator was
trying to send notifications to the thick-client nodes.
On the frontend pods (e.g.: qs9cl - corresponds to node 06a1fc26) we could
then see a message saying they detected backend-0 node failed:
*{"timestamp":"2026-10-22
12:34:27,772","logLevel":"WARN","threadName":"disco-event-worker-#45%front-z2-5585b6f78b-qs9cl%","loggerName":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","message":"Node
FAILED: TcpDiscoveryNode [id=45e4c48a-6815-4107-8278-c66f5e9a720b,
consistentId=backend-0, addrs=<REDACTED>, sockAddrs=null, discPort=47500,
order=1512, intOrder=765, loc=false, ver=2.18.0#20260423-sha1:d49adada,
isClient=false, dataCenterId=null]"}*
*{"timestamp":"2026-10-22
12:34:27,814","logLevel":"WARN","threadName":"grid-nio-worker-tcp-comm-0-#23%front-z2-5585b6f78b-qs9cl%","loggerName":"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi","message":"Close
incoming connection, unknown node
[nodeId=45e4c48a-6815-4107-8278-c66f5e9a720b,
ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker
[super=AbstractNioClientWorker [idx=0, bytesRcvd=189860895, bytesSent=0,
bytesRcvd0=2877, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0,
igniteInstanceName=front-z2-5585b6f78b-qs9cl, finished=false,
heartbeatTs=1792672467803, hashCode=267317747, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#23%front-z2-5585b6f78b-qs9cl%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=65 lim=65 cap=32768],
inRecovery=null, outRecovery=null, closeSocket=true,
outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1,
maxMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.MaxValueMetric@dff208b7,
super=GridNioSessionImpl [locAddr=<REDACTED>, rmtAddr=<REDACTED>,
createTime=1792672467803, closeTime=0, bytesSent=4599, bytesRcvd=2877,
bytesSent0=4599, bytesRcvd0=2877, sndSchedTime=1792672467803,
lastSndTime=1792672467803, lastRcvTime=1792672467803, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@1ef17b8d, directMode=true],
GridConnectionBytesVerifyFilter, SSL filter], accepted=true,
markedForClose=false]]]"} *
So it's clear the communication between them was broken.
Eventually backend-5 (node id f7678a98-2cf8-410f-a898-6e29229e8fda)
rejoins, topVer=1554 exchange starts and never finishes (we could not find
log indicating the exchange finished):
{"timestamp":"2026-10-22
12:35:00,951","logLevel":"INFO","threadName":"disco-event-worker-#51%backend-0%","loggerName":"org.apache.ignite.internal.managers.discovery.GridDiscoveryManager","message":"Added
new node to topology: TcpDiscoveryNode
[id=f7678a98-2cf8-410f-a898-6e29229e8fda, consistentId=backend-5,
addrs=<REDACTED>, discPort=47500, order=1554, intOrder=786, loc=false,
ver=2.18.0#20260423-sha1:d49adada, isClient=false, dataCenterId=null]"}
{"timestamp":"2026-10-22
12:35:00,952","logLevel":"INFO","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.exchange.time","message":"Started
exchange init [topVer=AffinityTopologyVersion [topVer=1554, minorTopVer=0],
crd=true, evt=NODE_JOINED, evtNode=f7678a98-2cf8-410f-a898-6e29229e8fda,
customEvt=null, allowMerge=true, exchangeFreeSwitch=false]"}
Backend-0 then starts logging that it failed to wait for partition release
frequently:
{"timestamp":"2026-10-22
12:35:10,955","logLevel":"WARN","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.diagnostic","message":"Failed
to wait for partition release future [topVer=AffinityTopologyVersion
[topVer=1554, minorTopVer=0], node=45e4c48a-6815-4107-8278-c66f5e9a720b]"}
{"timestamp":"2026-10-22
12:35:10,963","logLevel":"WARN","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.diagnostic","message":"Latch
manager state: ExchangeLatchManager [serverLatches=ConcurrentHashMap
{CompletableLatchUid [id=exchange, topVer=AffinityTopologyVersion
[topVer=1554, minorTopVer=0]]=ServerLatch [permits=1, pendingAcks=HashSet
[45e4c48a-6815-4107-8278-c66f5e9a720b], super=CompletableLatch
[id=CompletableLatchUid [id=exchange, topVer=AffinityTopologyVersion
[topVer=1554, minorTopVer=0]]]]}, clientLatches=ConcurrentHashMap {}]"}
Backend-0 seems to be awaiting on itself at this point, as there are 2
AtomicUpdateReleaseFuture.
These are the futures we see in the logs:
{"timestamp":"2026-10-22
12:35:10,957","logLevel":"WARN","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture","message":"Partition
release future: PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=1554, minorTopVer=0], totalFutures=5,
futures=[ExplicitLockReleaseFuture [topVer=AffinityTopologyVersion
[topVer=1554, minorTopVer=0], totalFutures=0, futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=1554,
minorTopVer=0], totalFutures=2, futures=[GridDhtAtomicSingleUpdateFuture
[allUpdated=true, super=GridDhtAtomicAbstractUpdateFuture [futId=2458706,
resCnt=2, addedReader=true, dhtRes=TransformMapView
{9bb269d6-11bb-4cc7-85f8-91d0d43fcecf=[res=true, size=1, nearSize=0],
9b513009-3bc3-47c6-b2f3-21cd8d96b8d4=[res=true, size=1, nearSize=0],
4aa6bd4c-9f00-4b53-bd7c-dcac56dab975=[res=false, size=0, nearSize=1]}]],
GridDhtAtomicSingleUpdateFuture [allUpdated=true,
super=GridDhtAtomicAbstractUpdateFuture [futId=62694, resCnt=7,
addedReader=true, dhtRes=TransformMapView
{9b513009-3bc3-47c6-b2f3-21cd8d96b8d4=[res=true, size=1, nearSize=0],
91158478-87f6-45b7-b428-2cbb10c074bc=[res=false, size=0, nearSize=1],
4aa6bd4c-9f00-4b53-bd7c-dcac56dab975=[res=false, size=0, nearSize=1],
53dbc7fb-69ae-476d-b13a-562f79edb4bd=[res=false, size=0, nearSize=1],
da18f110-896b-418a-be26-c01cd0be7720=[res=false, size=0, nearSize=1],
06a1fc26-8833-4f1f-afc0-2abdfdd3f4eb=[res=false, size=0, nearSize=1],
9440b479-f6a3-4e4e-bf84-835f33e0eda1=[res=true, size=0, nearSize=1],
df30a2d5-e82f-44bf-be0a-4c5fa14de89d=[res=true, size=1, nearSize=0],
4cb0eb8c-bec3-40a5-ae6b-c73cc322411f=[res=true, size=0, nearSize=1],
62853a54-8550-43d0-b580-72d1d634b643=[res=true, size=0, nearSize=1],
9e2d35bd-c6b0-42f0-9efa-a08790fed26a=[res=true, size=0, nearSize=1],
9a2dc81a-5675-48f1-879d-ed17787de29f=[res=false, size=0, nearSize=1],
5ebe852c-b733-4049-af8d-d61cf86bb84e=[res=true, size=0, nearSize=1]}]]]],
DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=1554,
minorTopVer=0], totalFutures=0, futures=[]], LocalTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=1554, minorTopVer=0],
totalFutures=0, futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=1554, minorTopVer=0],
totalFutures=1, futures=[RemoteTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=1554, minorTopVer=0],
totalFutures=0, futures=[]]]]]]"}
All the pods whose res=false are frontend thick clients who considered the
coordinator had failed.
Hours later, we could see the same futures on the coordinator queue:
{"timestamp":"2026-10-22
16:47:41,352","logLevel":"WARN","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.diagnostic","message":">>>
GridDhtAtomicSingleUpdateFuture [allUpdated=true,
super=GridDhtAtomicAbstractUpdateFuture [futId=62694, resCnt=7,
addedReader=true, dhtRes=TransformMapView
{9b513009-3bc3-47c6-b2f3-21cd8d96b8d4=[res=true, size=1, nearSize=0],
91158478-87f6-45b7-b428-2cbb10c074bc=[res=false, size=0, nearSize=1],
4aa6bd4c-9f00-4b53-bd7c-dcac56dab975=[res=false, size=0, nearSize=1],
53dbc7fb-69ae-476d-b13a-562f79edb4bd=[res=false, size=0, nearSize=1],
da18f110-896b-418a-be26-c01cd0be7720=[res=false, size=0, nearSize=1],
06a1fc26-8833-4f1f-afc0-2abdfdd3f4eb=[res=false, size=0, nearSize=1],
9440b479-f6a3-4e4e-bf84-835f33e0eda1=[res=true, size=0, nearSize=1],
df30a2d5-e82f-44bf-be0a-4c5fa14de89d=[res=true, size=1, nearSize=0],
4cb0eb8c-bec3-40a5-ae6b-c73cc322411f=[res=true, size=0, nearSize=1],
62853a54-8550-43d0-b580-72d1d634b643=[res=true, size=0, nearSize=1],
9e2d35bd-c6b0-42f0-9efa-a08790fed26a=[res=true, size=0, nearSize=1],
9a2dc81a-5675-48f1-879d-ed17787de29f=[res=false, size=0, nearSize=1],
5ebe852c-b733-4049-af8d-d61cf86bb84e=[res=true, size=0, nearSize=1]}]]"}
And the PME had not completed yet:
{"timestamp":"2026-10-22
16:47:41,350","logLevel":"WARN","threadName":"exchange-worker-#53%backend-0%","loggerName":"org.apache.ignite.internal.diagnostic","message":"Failed
to wait for partition release future [topVer=AffinityTopologyVersion
[topVer=1554, minorTopVer=0], node=45e4c48a-6815-4107-8278-c66f5e9a720b]"}
If the above analysis is correct, is it expected that missing acks from the
thick clients would block the PME? Can this be configured somehow, so it
timeouts rather than hanging? Clients might die for a number of reasons and
our expectation would be that the IMDG would be resilient to those events.
If any additional information is required, please let me know and I'm happy
to provide it.
Thanks & Best Regards,
Alessandro Parolin
