Elias Levy created FLINK-4027:
---------------------------------
Summary: FlinkKafkaProducer09 sink can lose messages
Key: FLINK-4027
URL: https://issues.apache.org/jira/browse/FLINK-4027
Project: Flink
Issue Type: Bug
Components: Kafka Connector
Affects Versions: 1.0.3
Reporter: Elias Levy
Priority: Critical
The FlinkKafkaProducer09 sink appears to not offer at-least-once guarantees.
The producer is publishing messages asynchronously. A callback can record
publishing errors, which will be raised when detected. But as far as I can
tell, there is no barrier to wait for async errors from the sink when
checkpointing or to track the event time of acked messages to inform the
checkpointing process.
If a checkpoint occurs while there are pending publish requests, and the
requests return a failure after the checkpoint occurred, those message will be
lost as the checkpoint will consider them processed by the sink.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
