shashank created CAMEL-26577:
--------------------------------
Summary: camel-core - RebalancingCamelClusterService: every
rebalancing pass takes a reference on the views and never releases it, so the
view of a removed clustered route keeps the leadership and a released view is
started again
Key: CAMEL-26577
URL: https://issues.apache.org/jira/browse/CAMEL-26577
Project: Camel
Issue Type: Bug
Components: camel-kubernetes, camel-core
Reporter: shashank
Assignee: shashank
{{RebalancingCamelClusterService}} (camel-support) wraps a
{{CamelPreemptiveClusterService}} (the Kubernetes cluster service) and runs
{{reconcile()}} on its own scheduled thread every period. Each pass looks at
every namespace through {{delegate.getView(namespace)}}, in {{members()}},
{{owned()}} and {{setDisabled()}}, several times per pass, and never calls
{{releaseView}}.
{{AbstractCamelClusterService.getView}} is not a plain lookup: it takes a
reference on the view (and starts it on the first one), and {{releaseView}}
stops the view when the last reference is released
({{CamelClusterService.releaseView}}: "Release a view if it has no references";
CAMEL-25089 made sure a released view is not started again by a restart of the
service). So with the rebalancing service:
* a namespace whose clustered route is removed keeps its view running: the
{{ClusteredRoutePolicy}} releases its reference, but the rebalancer holds one
more per pass. This node stays in the election, and when it is the leader of
that namespace no other node runs the route (the rebalancer even counts it as
one of its partitions);
* a view that was already released (stopped) is started again by the next pass,
as its {{getView}} takes the first reference again;
* {{members()}} then also compares the members of views nobody uses.
Found with a TLA+ model of the reference count of one namespace with three
actors (the route policy retaining at onInit and releasing at onRemove, the
rebalancing thread, the service stop/start). Property: once the last route
using the namespace is removed and the rebalancer is between passes, the view
is stopped and holds no reference. Short trace on main: route added (count 1,
view started) -> route removed (count 0, view stopped) -> reconcile pass
{{getView}} (count 1, view started again) -> pass ends, nothing released -> the
view keeps running with no user. The same with the pass before the removal: the
count never drops to 0. The fixed model (look only at views in use, release
every reference taken) holds; both paths are reachable (witnesses).
Reproduced on the real classes with {{DefaultCamelContext}}, a
{{ClusteredRoutePolicy}} route and a test {{CamelPreemptiveClusterService}}
extending {{AbstractCamelClusterService}} (no Kubernetes): after
{{removeRoute}} the view is still {{Started}}; a view released through the API
is {{Started}} again after one {{reconcile()}}. Both tests are deterministic
(the passes are run explicitly, the period is one hour).
Proposed fix: the rebalancer takes a reference only on the views that are in
use (a package-private lookup of {{AbstractCamelClusterService}} that does not
create or start a view) and releases each one in a {{finally}}; namespaces with
no user are not partitions to balance. Delegates that do not extend
{{AbstractCamelClusterService}} get {{getView}}/{{releaseView}} pairs.
Not part of this issue (minor): {{doStop}} does not wait for a pass in flight,
which reschedules itself on the {{serializedExecutor}} field, so after a quick
stop/start one more chain of passes runs on the new executor.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
