Hi Dom, Thanks for reporting the issue. Yes, I think this is a regression issue. I've created a bug in JIRA: KAFKA-13612 <https://issues.apache.org/jira/browse/KAFKA-13612>. I'll work on a fix for it.
Thank you. Luke On Fri, Jan 21, 2022 at 9:59 PM Dominic Evans <[email protected]> wrote: > Hi, > > Kafka 2.8.x changed the behaviour of auto topic creation (KAFKA-9751) so > that it's now done asynchronously by a forwarded request to the controller > rather than being immediately started by the broker that received the > MetadataRequest. In 2.7.x and earlier the client used to get > LEADER_NOT_AVAILABLE on the first MetadataResponse as the topic creation > was immediately in-progress and the partitions and leaders were just being > assigned. With 2.8.x onward, instead they now get > UNKNOWN_TOPIC_OR_PARTITION until the Controller has actioned the create. > That seems fine and is easy to handle. > > However, historically the __consumer_offsets topic was special cased such > that it would get auto-created on a Metadata request even if normal topic > auto-creation (`auto.create.topics.enable`) was disabled in the broker. > Does anyone know if something has changed there? > > If I startup a brand new empty 2.8.x cluster and just send > MetadataRequests for the __consumer_offsets topic I always get > UNKNOWN_TOPIC_OR_PARTITION responses and the auto-creation doesn't seem to > ever get actioned for that alone. As you'd expect, if I start up a consumer > group the JoinGroup is sufficient to cause the __consumer_offsets topic to > be created. On 2.7.x and earlier, just sending MetadataRequests is > sufficient for the __consumer_offsets topic to be created. > > The behaviour can be observed using (e.g.,) kcat (neé kafkacat) after > starting a brand new cluster with auto.create.topics.enable=false: > > kcat -L -b localhost:29091,localhost:29092,localhost:29093 -t > __consumer_offsets -d metadata,topic > > And no auto-creation ever happens despite allowAutoCreate true in the > Metadata request body: > > Kafka (Metadata v4 Request) > Length: 42 > API Key: Metadata (3) > API Version: 4 > Correlation ID: 3 > Client ID: rdkafka > Topic > Topic Name: __consumer_offsets > Allow Auto Topic Creation: True > > Is this a regression or an intentional change in behaviour? I couldn't > find any reference to this in the changelog / release notes. > > Best regards, > Dom > -- > Dominic Evans > IBM Software Group, Hursley Park, UK
