Paolo Moriello created KAFKA-9806:
-------------------------------------
Summary: Unable to consume when __consumer_offsets is created
under insufficient cluster ACLs
Key: KAFKA-9806
URL: https://issues.apache.org/jira/browse/KAFKA-9806
Project: Kafka
Issue Type: Bug
Components: consumer, controller, core
Reporter: Paolo Moriello
Assignee: Paolo Moriello
h2. Description
The __consumer_offsets topic is automatically created the first time
consumption happens on a cluster. When a consumer start to consume from a
topic, it first submits a FindCoordinator request, which tries to get metadata
from the __consumer_offsets topic. If this does not exist, it gets created. On
topic creation, the controller assigning partitions to the other brokers
through UpdateMetadata and LeaderAndIsr requests.
If there aren't sufficient cluster-level ACLs, UpdateMetadata and LeaderAndIsr
are not authorized by peer brokers; so, the requests fail, but the
auto-creation of the topic succeeds and zk-nodes are filled with the related
metadata. No-operation is performed by the controller on handling of failed
UpdateMetadata and LeaderAndIsr, so __consumer_offset topic does not get
deleted.
When ACLs are removed/fixed, __consumer_offset is in a bad state for the topics
we tried to consume on under insufficient ACLs, and consumption is not
possible: consumer does not return any data. No errors are thrown by the
consumer, nor by the brokers. By enabling DEBUG logs on the consumer we only
see that: 'The coordinator is not available.'
Note: this is fixed with a broker restart or __consumer_offsets partition
reassignment, since both operations trigger UpdateMetadata and LeaderAndIsr
requests on the topic.
h2. Steps to reproduce
Clean Kafka cluster (no __consumer_offset) with more than 1 broker, and SSL
enabled for inter-broker communication.
+ I had allow.everyone.if.no.acl.found set to true on my test cluster.
# Create a topic
# Set insufficient cluster level ACLs. Which precise ACL does not matter as
long as ClusterAction initiated by administrative messages between brokers is
denied.
# Consume on the topic -> this request will fail with unauthorized errors on
the client side as well
# Remove ACLs or set ACLs to allow inter-broker communication
# Produce/consume on topic. Consumer won't be able to read any data.
Step 3 triggers creation of __consumer_offset topic. In addition to the client
side errors, it is possible to see related UpdateMetadata and LeaderAndIsr
exceptions in the broker logs. However at this point you can verify that
__consumer_offset topic is created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
