Vyacheslav Koptilin created IGNITE-17507:
--------------------------------------------
Summary: Failed to wait for partition map exchange on some clients
Key: IGNITE-17507
URL: https://issues.apache.org/jira/browse/IGNITE-17507
Project: Ignite
Issue Type: Bug
Reporter: Vyacheslav Koptilin
Assignee: Vyacheslav Koptilin
We have scenario with several client and server nodes, which can stuck on PME
after start:
* Start some server nodes
* Trigger rebalance
* Start some client and server nodes
* Some of the client nodes stuck with Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion…
Deep investigation of the logs showed, that the root cause of the stuck PME on
client is the race between joining new client node and receiving stale
CacheAffinityChangeMessage on a client, which causes PME, but when other old
nodes receive this CacheAffinityChangeMessage, they skip it because of some
optimization.
Optimization can be found in the method
CacheAffinitySharedManager#onDiscoveryEvent, we save lastAffVer = topVer; for
old nodes, but because of some race lastAffVer for the problem client node is
null when we reach CacheAffinitySharedManager#onCustomEvent and we schedule
invalid PME in msg.exchangeNeeded(exchangeNeeded);, but other nodes skip this
PME
The possible fix is that we can try to make the _CacheAffinityChangeMessage
_mutable (mutable discovery custom message). It allows to modify the message
before sending it across the ring. This approach does not require to make a
decision to apply or skip the message on client nodes, the required flag will
be transferred from a server node. In case of using Zookeeper Discovery, there
is no ability to mutate discovery messages. However is is possible to mutate
the message on the coordinator node. This is quite enough for our case.
TeamCity does not demonstrates any issue with this approach.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
