[ https://issues.apache.org/jira/browse/KAFKA-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852643#comment-16852643 ]
Richard Yu commented on KAFKA-8421: ----------------------------------- Might try to take a hack at this one. It certainly is an interesting issue. :) > Allow consumer.poll() to return data in the middle of rebalance > --------------------------------------------------------------- > > Key: KAFKA-8421 > URL: https://issues.apache.org/jira/browse/KAFKA-8421 > Project: Kafka > Issue Type: Improvement > Components: consumer > Reporter: Guozhang Wang > Priority: Major > > With KIP-429 in place, today when a consumer is about to send join-group > request its owned partitions may not be empty, meaning that some of its > fetched data can still be returned. Nevertheless, today the logic is strict: > {code} > if (!updateAssignmentMetadataIfNeeded(timer)) { > return ConsumerRecords.empty(); > } > {code} > I.e. if the consumer enters a rebalance it always returns no data. > As an optimization, we can consider letting consumers to still return > messages that still belong to its owned partitions even when it is within a > rebalance, because we know it is safe that no one else would claim those > partitions in this rebalance yet, and we can still commit offsets if, after > this rebalance, the partitions need to be revoked then. > One thing we need to take care though is the rebalance timeout, i.e. when > consumer's processing those records they may not call the next poll() in time > (think: Kafka Streams num.iterations mechanism), which may leads to consumer > dropping out of the group during rebalance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)