[ https://issues.apache.org/jira/browse/KAFKA-12980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376870#comment-17376870 ]
Jason Gustafson commented on KAFKA-12980: ----------------------------------------- [~ChrisEgerton] I do not think it requires a KIP. It is a slight behavioral change, but it does not really affect API usage. > Allow consumers to return from poll when position advances due to aborted > transactions > -------------------------------------------------------------------------------------- > > Key: KAFKA-12980 > URL: https://issues.apache.org/jira/browse/KAFKA-12980 > Project: Kafka > Issue Type: Improvement > Components: consumer > Reporter: Chris Egerton > Assignee: Chris Egerton > Priority: Major > > When {{Consumer::poll}} is invoked on a topic with an open transaction, and > then that transaction is aborted, {{poll}} does not return unless there are > other records available in that topic after the aborted transaction. > Instead, {{poll}} could return in this case, even when no records are > available. > This facilitates reads to the end of a topic where the end offsets of a topic > are listed and then a consumer for that topic is polled until its > [position|https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#position(org.apache.kafka.common.TopicPartition)] > is at or beyond each of those offsets (for example, [Connect does > this|https://github.com/apache/kafka/blob/fce771579c3e20f20949c4c7e0a5e3a16c57c7f0/connect/runtime/src/main/java/org/apache/kafka/connect/util/KafkaBasedLog.java#L322-L345] > when reading to the end of any of its internal topics). > We could update the existing language in the docs for {{Consumer::poll}} from > {quote}This method returns immediately if there are records available. > {quote} > to > {quote}This method returns immediately if there are records available or if > the position advances past control records. > {quote} > > A workaround for existing users who would like to see this is to use short > poll intervals and manually check the consumer's position in between each > poll, but this is fairly tedious and may lead to excess CPU and network > utilization depending on the latency requirements for knowing when the end of > the topic has been reached. -- This message was sent by Atlassian Jira (v8.3.4#803005)