Matthias J. Sax created KAFKA-8821:
--------------------------------------
Summary: Avoid pattern subscription to allow for stricter ACL
settings
Key: KAFKA-8821
URL: https://issues.apache.org/jira/browse/KAFKA-8821
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
To avoid triggering auto topic creation (if `auto.create.topic.enable=true` on
the brokers), Kafka Streams uses consumer pattern subscription. For this case,
the consumer requests all metadata from the brokers and does client side
filtering.
However, if users want to set ACL to restrict a Kafka Streams application, this
may results in broker side ERROR logs that some metadata cannot be provided.
The only way to avoid those broker side ERROR logs is to grant corresponding
permissions.
As of 2.3 release it's possible to disable auto topic creation client side (via
https://issues.apache.org/jira/browse/KAFKA-7320). Kafka Streams should use
this new feature (note, that broker version 0.11 is required) to allow users to
set strict ACLs without getting flooded with ERROR logs on the broker.
The proposal is that by default Kafka Streams disables auto-topic create client
side (optimistically) and uses regular subscription (not pattern subscription).
If an older broker is used, users need to explicitly enable
`allow.auto.create.topic` client side. If we detect this setting, we switch
back to pattern based subscription.
If users don't enable auto topic create client side and run with an older
broker, we would just rethrow the exception to the user, adding some context
information on how to fix the issue.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
