[
https://issues.apache.org/jira/browse/NIFI-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097886#comment-18097886
]
David Handermann commented on NIFI-16120:
-----------------------------------------
Thanks for following up and verifying the solution in your environment
[~karmnik], great to hear that the 5-second cache for role leader status
achieves the desired significant reduction in Kubernetes API requests at scale.
Thanks again for raising this issue and providing the helpful interaction along
the way!
> KubernetesLeaderElectionManager.isLeader() does an uncached Kubernetes API
> GET on the scheduling hot path, causing excessive lease requests
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16120
> URL: https://issues.apache.org/jira/browse/NIFI-16120
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.5.0, 2.10.0
> Reporter: Michal S
> Assignee: David Handermann
> Priority: Major
> Fix For: 2.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h2. Summary
> With
> {{nifi.cluster.leader.election.implementation=KubernetesLeaderElectionManager}},
> NiFi issues a live, uncached GET on the leader-election Lease on *every*
> call to {{isLeader()}}. Because {{isLeader()}} is on the processor scheduling
> hot path, a cluster with many "Primary Node Only" processors generates
> hundreds of Lease GET requests per second per node against the Kubernetes API
> server.
> h2. Affects
> Confirmed on 2.5.0. The relevant code is unchanged on current {{main}}, so
> all 2.x releases using Kubernetes leader election are affected. Not present
> with ZooKeeper (CuratorLeaderElectionManager), which answers isLeader() from
> local state.
> h2. Root cause
> {{org.apache.nifi.kubernetes.leader.election.KubernetesLeaderElectionManager.isLeader(String)}}
> calls {{getLeader(roleName)}} ->
> {{StandardLeaderElectionCommandProvider.findLeader()}} ->
> {{kubernetesClient.leases()...get()}} (a live API GET) on every invocation.
> The manager already keeps a local {{roleLeaders}} map, updated by the fabric8
> {{LeaderElector}} {{onNewLeader}} callback ({{setRoleLeader}}), but
> {{isLeader()}} ignores it.
> Call path on the hot loop:
> {{ConnectableTask.isRunOnCluster()}} -> {{FlowController.isPrimary()}} ->
> {{KubernetesLeaderElectionManager.isLeader()}} -> getLeader() -> live Lease
> GET.
> This runs for every scheduling invocation of a Primary-Node-Only processor
> (on non-primary nodes it is invoked and skipped repeatedly), so lease GET
> volume scales with (running Primary-Node-Only processors x scheduling
> frequency x concurrent tasks).
> h2. Impact / evidence
> On a 2-node EKS cluster with a few dozen active Primary-Node-Only processors:
> ~32,000 Lease GET/min per non-leader node (~640/s), all HTTP 200 -> ~46M
> Kubernetes audit events/day. Thread dumps show many "Timer-Driven Process
> Thread" stacks in isRunOnCluster -> isPrimary -> getLeader ->
> BaseOperation.get. Stopping the flows dropped the rate ~99% instantly; an
> identical but idle cluster stayed at baseline. Reproduced independently of
> fabric8 client version (7.3.1 / 7.7.0 / 7.8.0) and HTTP client implementation
> (JDK HttpClient and OkHttp) - it is not a client issue.
> h2. Proposed fix
> Make {{isLeader()}} read the locally-tracked leader from {{roleLeaders}}
> (already maintained by the onNewLeader callback), with a one-time live
> fallback only before the first observation. This mirrors
> CuratorLeaderElectionManager and removes the API GET from the hot path.
> {code:java}
> } else {
> String leaderId = roleLeaders.get(roleName);
> if (leaderId == null) {
> leaderId = getLeader(roleName).orElse(null); // one-time fallback
> before first observation
> }
> leader = participantId.equals(leaderId);
> ...
> }
> {code}
> Verified: steady-state Lease GET drops from ~32k/min to baseline (a few
> hundred/min) even under a synthetic load of 21 Primary-Node-Only processors
> at ~12k tasks/s. A short-TTL cache on getLeader() is an alternative. I can
> supply a pull request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
