On Tue, Apr 7, 2020, at 08:08, Paolo Moriello wrote: > Hi Colin, > > Thanks for your interest in this. I agree with you, this change could break > compatibility. However, changing the source principal is non trivial in > this case. In fact, here the problem is not in the internal topic creation > - which succeeds - but in the two subsequent LeaderAndIsr and > UpdateMetadata requests. > > When a consumer tries to consume for the first time, the creation of > internal topic completes, zk-nodes are filled with the necessary metadata, > and this triggers a ZkPartitionStateMachine (PartitionStateMachine.scala) > update which, in turn, makes the ControllerChannelManager > (ControllerChannelManager.scala) send LeaderAndIsr and UpdateMetadata > requests to the brokers; (I can be wrong, but I believe that this requests > are already being executed with broker principal). These requests fail > because we authorize the cluster operation there, so the __consumer_offsets > topic remains in a bad state.
I might be misunderstanding something here, but it seems to me that if LeaderAndIsrRequest or UpdateMetadataRequest are failing with authorization errors, then there is a configuration problem on the cluster which doesn't have anything to do with the __consumer_offsets topic. > > Is there a reason to not authorize the operation for find coordinator > requests as well? To be clear, we can't change the authorization for FindCoordinatorRequest. best, Colin > > Thanks, > Paolo > > On Mon, 6 Apr 2020 at 23:58, Colin McCabe <cmcc...@apache.org> wrote: > > > Hi Paolo, > > > > Thanks for finding this issue. > > > > Unfortunately, you certainly can't add a new permission requirement to an > > existing RPC without breaking compatibility. So the current solution in > > the PR will not work. However, you should be able to have the broker > > create the topic using its own principal rather than the caller's. > > Basically the equivalent of a doAs block (I forget how we do this exactly, > > but we do have some way of doing it). > > > > best, > > Colin > > > > > > On Mon, Apr 6, 2020, at 02:56, Paolo Moriello wrote: > > > Hello everybody, > > > > > > I've opened a Jira to fix a bug on creation of internal topics. This > > > happens when the topics are created under insufficient ACLs: eg. > > > __consumer_offset is created but subsequent updateMetadata and leaderIsr > > > requests fail; the topic is than in an inconsistent state and it is > > > impossible to consume. > > > > > > Jira: https://issues.apache.org/jira/browse/KAFKA-9806 > > > > > > A simple fix to solve this problem is to authorize the cluster operation > > > before creating these topics. I've submitted a PR with the fix: > > > https://github.com/apache/kafka/pull/8415 > > > > > > Please take a look and let me know if you have any feedback. > > > Thanks, > > > Paolo > > > > > >
