Chhida opened a new pull request, #22826:
URL: https://github.com/apache/kafka/pull/22826
This PR addresses a critical performance issue in StandardAuthorizerData
where every authorization request scans ALL ACLs for the matching resource
type, resulting in O(N) complexity per request.
On a production cluster with ~7000 ACLs and ~500 topics, this causes
~3.5 million iterations per MetadataRequest (504 topics × ~7000 ACLs),
saturating all data-plane-kafka-request-handler threads at 94.8% CPU.
**_(data-plane-kafka-request-handler-0" #80 daemon prio=5 os_prio=0
cpu=17207014.77ms elapsed=18143.19s tid=0x00007f76d50aa800 nid=0x8e7bb runnable
[0x00007f7681480000]
java.lang.Thread.State: RUNNABLE
at
org.apache.kafka.metadata.authorizer.StandardAcl.kafkaPrincipal(StandardAcl.java:106)
at
org.apache.kafka.metadata.authorizer.StandardAuthorizerData.findResult(StandardAuthorizerData.java:493)
at
org.apache.kafka.metadata.authorizer.StandardAuthorizerData.checkSection(StandardAuthorizerData.java:428)
at
org.apache.kafka.metadata.authorizer.StandardAuthorizerData.findAclRule(StandardAuthorizerData.java:367)
at
org.apache.kafka.metadata.authorizer.StandardAuthorizerData.authorize(StandardAuthorizerData.java:247)
at
org.apache.kafka.metadata.authorizer.StandardAuthorizer.authorize(StandardAuthorizer.java:143)
at kafka.server.AuthHelper.filterByAuthorized(AuthHelper.scala:113)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:1350)
at kafka.server.KafkaApis.handle(KafkaApis.scala:192)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:159)
at java.lang.Thread.run([email protected]/Thread.java:829)_**
Root cause (from thread dump):
HashSet.contains → KafkaPrincipal.hashCode → String.hashCode
at StandardAuthorizerData.findResult (StandardAuthorizerData.java:493)
Fix:
- Add a principal-level index (aclsByPrincipal) to AclCache, mapping
principal strings (e.g. "User:bob") to their ACL sets
- Refactor findAclRule/checkSection/findResult to use this index with
a skipPrincipalCheck flag, avoiding the expensive HashSet.contains
when the principal index guarantees membership
- Add matchesUpTo == 0 guard in checkSection to prevent potential
infinite loop with empty substring
Performance improvement (reproduction with 7001 ACLs, 504 topics):
- Before: 381% CPU, 5/8 handlers stuck in findAclRule
- After: ~30% CPU, 0/8 handlers in findAclRule (all idle)
Validated in production on Kafka 3.9.1 (backport) with
allow.everyone.if.no.acl.found=false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
