mustafaAlmonayer opened a new pull request, #25611:
URL: https://github.com/apache/camel/pull/25611
------------------------------------------------------------------------------------------------------------------
Problem Description:
------------------------------------------------------------------------------------------------------------------
When the ZooKeeper cluster becomes unavailable, the Camel route managed by
"ClusteredRoutePolicy" never stops, even though the node has lost its ZooKeeper
session. Beyond the route not stopping, this also introduces a split-brain
scenario: if one node becomes isolated from ZooKeeper while others remain
connected, a new leader election runs on the healthy side and a second node
wins leadership and starts the same route — now two nodes are running the same
route simultaneously with no coordination, which can cause data corruption,
duplicate processing, or conflicting writes depending on what the route does.
------------------------------------------------------------------------------------------------------------------
Root Cause:
------------------------------------------------------------------------------------------------------------------
The root cause is a timing issue between two threads in
"ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()". When
ZooKeeper goes down, Curator's "ConnectionStateManager" thread fires a
"SUSPENDED"/"LOST" state change, which causes "LeaderSelectorListenerAdapter"
to interrupt the thread blocked inside "takeLeadership()". The "BlockingTask"
handles the interrupt correctly and exits its loop. Execution then reaches the
leadership-lost event at:
fireLeadershipChangedEvent(getLeader().orElse(null)); //
ZooKeeperClusterView.java line 155
This fires "ClusteredRoutePolicy.leadershipChanged()", which, regardless of
the argument passed, always calls back into:
setLeader(clusterView.getLocalMember().isLeader()); //
ClusteredRoutePolicy.java line 376
Which resolves to: leaderSelector.hasLeadership() //
CuratorLocalMember.isLeader(), line 162
This returns "true" at this point because Curator's guarantees that
"hasLeadership()" only becomes "false" after "takeLeadership()" returns to the
"LeaderSelector" internals. The event fires from inside "takeLeadership()", so
the answer is always "true", "ClusteredRoutePolicy" sees no leadership change,
and the route keeps running indefinitely.
------------------------------------------------------------------------------------------------------------------
Consequences:
------------------------------------------------------------------------------------------------------------------
The consequence is:
- The [isolated node] keeps running the route because "hasLeadership()" is
still "true" at the moment the event fires
- The [healthy side] elects a new leader, which also starts the same route
- Both nodes now process the same workload simultaneously with no mutual
exclusion — a classic [split-brain]
------------------------------------------------------------------------------------------------------------------
Solution:
------------------------------------------------------------------------------------------------------------------
Introduce a "volatile boolean leader" flag inside "ZooKeeperClusterView"
that is owned and controlled by the view itself, rather than delegating to
"leaderSelector.hasLeadership()".
Change "CuratorLocalMember.isLeader()" to return this flag instead.
In "takeLeadership()", set the flag to "true" before firing the
leadership-gained event, and in the "finally" block set it to "false" before
firing the leadership-lost event. This guarantees that when
"ClusteredRoutePolicy" calls back into "isLeader()" during the event, it reads
"false", which causes "stopManagedRoutes()" to be called and the route to stop
correctly before any other node can win the election and start it.
Additionally, "leaderSelector.autoRequeue()" should be called in "doStart()"
so that after losing leadership due to a ZooKeeper disconnect, the node
automatically re-enters the election when ZooKeeper reconnects and the route
can start again on whichever node wins.
- "ZooKeeperClusterView": add "volatile boolean leader" field
- "ZooKeeperClusterView.CuratorLocalMember.isLeader()": return "leader" flag
instead of "leaderSelector.hasLeadership()"
- "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()": set
"leader = true" on entry, wrap task in "try/finally", set "leader = false"
before firing the lost event
- "ZooKeeperClusterView.doStart()": add "leaderSelector.autoRequeue()" to
re-enter election after reconnect
------------------------------------------------------------------------------------------------------------------
How to reproduce:
------------------------------------------------------------------------------------------------------------------
1. Start two instances of a Camel application using
"ZooKeeperClusterService" with "ClusteredRouteController"
2. Confirm one instance is leader and its route is running
3. Isolate the leader node from all ZooKeeper nodes (e.g. firewall rules or
kill ZK nodes)
4. Observe on the healthy side: a new leader is elected and its route starts
5. Observe on the isolated node: the route never stops — both nodes are now
running the same route simultaneously (split-brain)
6. Expected: the isolated node's route stops as soon as its ZooKeeper
session is lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
