Aleksandr Savonin created FLINK-41598:
-----------------------------------------

             Summary: Overlapping transactional ID prefixes can cause data loss 
in KafkaSink POOLING mode
                 Key: FLINK-41598
                 URL: https://issues.apache.org/jira/browse/FLINK-41598
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka
            Reporter: Aleksandr Savonin


With exactly-once delivery and TransactionNamingStrategy.POOLING, a KafkaSink 
writer lists ongoing Kafka transactions on startup to clean up transactions 
left by a previous attempt. It uses a startsWith check to identify its 
transactions, so distinct transactional ID prefixes can match the same ID.

To reproduce, start a sink with the prefix "sink-2" and let it write a record 
in an open transaction such as "sink-2-0-0". Before that transaction commits, 
start a second sink with the prefix "sink" on the same topic. The second sink 
can abort "sink-2-0-0" and fence the first producer. If the transaction was 
awaiting checkpoint commit, its records can be lost from the committed Kafka 
output.

The same prefix check can also prevent startup. For example, if another 
producer has an open transaction named "sink-a-b", the writer tries to parse 
"a" as a subtask ID and throws NumberFormatException.

Cleanup should only consider IDs that match the connector’s generated format 
for a configured or restored prefix: <prefix>-<subtaskId>-<offset>. A writer 
with prefix "sink" should ignore both "sink-2-0-0" and "sink-a-b" and leave 
those producers able to commit.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to