Aleksandr Savonin created FLINK-41603:
-----------------------------------------
Summary: Kafka exactly-once sink can lose records after a commit
timeout during recovery
Key: FLINK-41603
URL: https://issues.apache.org/jira/browse/FLINK-41603
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Reporter: Aleksandr Savonin
During recovery, the Flink Kafka sink can acknowledge a transaction that Kafka
never committed. This can cause data loss for read_committed consumers.
The issue can occur when a committer restores two transactions, A and B, from
the same checkpoint, for example after scaling down. The following sequence
reproduces it:
1. Flink calls commitTransaction() for A. The call exceeds max.block.ms, so
Flink schedules a retry. Kafka can still complete the commit in the background.
2. Kafka completes A before Flink processes B.
3. Flink reuses the same producer for B. The Kafka client returns A’s pending
success result, and Flink marks B as committed without sending a commit request
for B.
4. Flink retries A. B has already been marked complete, so it is never retried.
Kafka eventually aborts B when its transaction expires.
I reproduced this with Kafka 4.2.0. Flink reported success for both
transactions, but Kafka showed that B was aborted. A read_uncommitted consumer
saw B’s record, while a read_committed consumer saw zero records.
Expected behavior: Flink should acknowledge each transaction only after Kafka
confirms the commit for that specific transaction. If a commit times out, its
retry should use the producer that issued it, including when several restored
transactions are pending.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)