[
https://issues.apache.org/jira/browse/KAFKA-12487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383609#comment-17383609
]
Konstantine Karantasis commented on KAFKA-12487:
------------------------------------------------
Changing the default consumer protocol to be the cooperative protocol has been
postponed for 3.1.0. Given that we are past the code freeze for 3.0, I'm
postponing this issue to 3.1.0 while keeping its blocker status for this
release. The change is not trivial and would be good to have enough time to
test before we release.
> Sink connectors do not work with the cooperative consumer rebalance protocol
> ----------------------------------------------------------------------------
>
> Key: KAFKA-12487
> URL: https://issues.apache.org/jira/browse/KAFKA-12487
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 3.0.0, 2.4.2, 2.5.2, 2.8.0, 2.7.1, 2.6.2
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Blocker
> Fix For: 3.0.0
>
>
> The {{ConsumerRebalanceListener}} used by the framework to respond to
> rebalance events in consumer groups for sink tasks is hard-coded with the
> assumption that the consumer performs rebalances eagerly. In other words, it
> assumes that whenever {{onPartitionsRevoked}} is called, all partitions have
> been revoked from that consumer, and whenever {{onPartitionsAssigned}} is
> called, the partitions passed in to that method comprise the complete set of
> topic partitions assigned to that consumer.
> See the [WorkerSinkTask.HandleRebalance
> class|https://github.com/apache/kafka/blob/b96fc7892f1e885239d3290cf509e1d1bb41e7db/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L669-L730]
> for the specifics.
>
> One issue this can cause is silently ignoring to-be-committed offsets
> provided by sink tasks, since the framework ignores offsets provided by tasks
> in their {{preCommit}} method if it does not believe that the consumer for
> that task is currently assigned the topic partition for that offset. See
> these lines in the [WorkerSinkTask::commitOffsets
> method|https://github.com/apache/kafka/blob/b96fc7892f1e885239d3290cf509e1d1bb41e7db/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L429-L430]
> for reference.
>
> This may not be the only issue caused by configuring a sink connector's
> consumer to use cooperative rebalancing. Rigorous unit and integration
> testing should be added before claiming that the Connect framework supports
> the use of cooperative consumers with sink connectors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
