[
https://issues.apache.org/jira/browse/KAFKA-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-16966:
------------------------------
Description:
In {{{}initWithCommittedOffsetsIfNeeded{}}}, the behavior of the existing
{{LegacyKafkaConsumer}} is to allow reuse only if the partitions for the
_current_ request equal those of the _previous_ request *exactly*
([source|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java?rgh-link-date=2024-06-14T16%3A43%3A11Z#L927]).
That logic is the basis for the behavior used in the
{{{}AsyncKafkaConsumer{}}}.
The proposed change is to allow for request reuse if the partitions for the
_current_ request are a subset of those of the _previous_ request. This
introduces a subtle difference in behavior between the two {{Consumer}}
implementations, so we need to decided if we want to change both
implementations or just {{{}AsyncKafkaConsumer{}}}. Also, the specific case
that the request reuse logic solves is when the user has passed in a very low
timeout value in a tight {{poll()}} loop, which suggests the partitions
wouldn't be changing between those loops.
was:
In {{{}initWithCommittedOffsetsIfNeeded{}}}, the behavior of the existing
{{LegacyKafkaConsumer}} is to allow reuse only if the partitions for the
_current_ request equal those of the _previous_ request *exactly*
([source|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java?rgh-link-date=2024-06-14T16%3A43%3A11Z#L927]).
That logic is the basis for the behavior used in the
{{{}AsyncKafkaConsumer{}}}.
It's unlikely to cause any problems, but it introduces a subtle difference in
behavior between the two {{Consumer}} implementations. Also, the specific case
that the request reuse logic solves is when the user has passed in a very low
timeout value in a tight {{poll()}} loop, which suggests the partitions
wouldn't be changing between those loops.
> Allow offset commit fetch to reuse previous request if partitions are a subset
> ------------------------------------------------------------------------------
>
> Key: KAFKA-16966
> URL: https://issues.apache.org/jira/browse/KAFKA-16966
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
> Affects Versions: 3.8.0
> Reporter: Kirk True
> Assignee: Kirk True
> Priority: Major
> Labels: consumer-threading-refactor
> Fix For: 3.9.0
>
>
> In {{{}initWithCommittedOffsetsIfNeeded{}}}, the behavior of the existing
> {{LegacyKafkaConsumer}} is to allow reuse only if the partitions for the
> _current_ request equal those of the _previous_ request *exactly*
> ([source|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java?rgh-link-date=2024-06-14T16%3A43%3A11Z#L927]).
> That logic is the basis for the behavior used in the
> {{{}AsyncKafkaConsumer{}}}.
> The proposed change is to allow for request reuse if the partitions for the
> _current_ request are a subset of those of the _previous_ request. This
> introduces a subtle difference in behavior between the two {{Consumer}}
> implementations, so we need to decided if we want to change both
> implementations or just {{{}AsyncKafkaConsumer{}}}. Also, the specific case
> that the request reuse logic solves is when the user has passed in a very low
> timeout value in a tight {{poll()}} loop, which suggests the partitions
> wouldn't be changing between those loops.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
