Qinghui Xu created KAFKA-9982:
---------------------------------
Summary: [kafka-connect] Source connector does not guarantee at
least once delivery
Key: KAFKA-9982
URL: https://issues.apache.org/jira/browse/KAFKA-9982
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 2.5.0
Reporter: Qinghui Xu
In kafka-connect runtime, the WorkerSourceTask is responsible for sending
records to the destination topics and managing the source offset commit.
Committed offsets are then used later for recovery of tasks during rebalance or
restart.
But there are two concerns when looking into the WorkerSourceTask
implementation:
* When producer fail to send records, there's no retry but just skipping
offset commit and then execute next loop (poll for new records)
* The offset commit and effectively sending records over network are in fact
asynchronous, which means the offset commit could happen before records are
received by brokers, and a rebalance/restart in this gap could lead to message
loss.
The conclusion is thus that the source connector does not support at least once
semantics by default (without the plugin implementation making extra effort
itself). I consider this as a bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
