[ https://issues.apache.org/jira/browse/KAFKA-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754976#comment-17754976 ]
Luke Chen commented on KAFKA-15353: ----------------------------------- I'm setting this issue as blocker for v3.5.2 and v3.6.0. Let me know if you have any thoughts. Sorry that I'm going to attend a conference the following days, so I can't submit patch for this issue soon. Welcome to take it over if anyone has available cycle. Thanks. cc [~calvinliu] [~junrao] [~dajac] > Empty ISR returned from controller after AlterPartition request > --------------------------------------------------------------- > > Key: KAFKA-15353 > URL: https://issues.apache.org/jira/browse/KAFKA-15353 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 3.5.0 > Reporter: Luke Chen > Priority: Blocker > Fix For: 3.6.0, 3.5.2 > > > In > [KIP-903|https://cwiki.apache.org/confluence/display/KAFKA/KIP-903%3A+Replicas+with+stale+broker+epoch+should+not+be+allowed+to+join+the+ISR], > (more specifically this [PR|https://github.com/apache/kafka/pull/13408]), we > bumped the AlterPartitionRequest version to 3 to use `NewIsrWithEpochs` field > instead of `NewIsr` one. And when building the request, we'll manually > convert/downgrade the request into the older version for backward > compatibility > [here|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/AlterPartitionRequest.java#L85-L96], > to extract ISR info from `NewIsrWithEpochs` and then fill in the `NewIsr` > field, and then clear the `NewIsrWithEpochs` field. > > The problem is, when the AlterPartitionRequest sent out for the first time, > there's some transient error (ex: NOT_CONTROLLER), we'll retry. On the retry, > we'll build the AlterPartitionRequest again. But this time, the request data > is the one that converted above. At this point, when we try to extract the > ISR from `NewIsrWithEpochs`, we'll get empty. So, we'll send out an > AlterPartition request with empty ISR, and impacting the kafka availability. > From the log, I can see this: > {code:java} > [2023-08-16 03:57:55,122] INFO [Partition test_topic-1 broker=3] ISR updated > to (under-min-isr) and version updated to 9 (kafka.cluster.Partition) > {code} > > This will happen when users trying to upgrade from versions < 3.5.0 to 3.5.0 > or later. During the rolling upgrade, there will be some nodes in v3.5.0, and > some are not. So, for the node in v3.5.0 will try to build an old version of > AlterPartitionRequest. And then, if it happen to have some transient error > during the AlterPartitionRequest send, the ISR will be empty and no producers > will be able to write data to the partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)