[ https://issues.apache.org/jira/browse/FLINK-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855319#comment-15855319 ]
Tzu-Li (Gordon) Tai commented on FLINK-5704: -------------------------------------------- FLINK-3679 and FLINK-5583 would probably change the {{DeserializationSchema}} interface. We should include that as part of the Kafka consumer usage migration. > Deprecate FlinkKafkaConsumer constructors in favor of improvements to > decoupling from Kafka offset committing > ------------------------------------------------------------------------------------------------------------- > > Key: FLINK-5704 > URL: https://issues.apache.org/jira/browse/FLINK-5704 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector, Streaming Connectors > Reporter: Tzu-Li (Gordon) Tai > > With FLINK-3398 and FLINK-4280, the {{FlinkKafkaConsumer}} will be able to > completely operate independently of committed offsets in Kafka. > I.e., > (1) *Starting position*: when starting, the consumer can choose to not use > any committed offsets in Kafka as the starting position > (2) *Committing offsets back to Kafka*: the consumer can completely opt-out > of committing offsets back to Kafka > However, our current default behaviour for (1) is to respect committed > offsets, and (2) is to always have offset committing. Users still have to > call the respective setter configuration methods to change this. > I think we should deprecate the current constructors in favor of new ones > with default behaviours (1) start from the latest record, without respecting > Kafka offsets, and (2) don't commit offsets. > With this change, users explicitly call the config methods of FLINK-3398 and > FLINK-4280 to *enable* respecting committed offsets for Kafka, instead of > _disabling_ it. They would want to / need to enable it, only when perhaps to > migrate from a non-Flink consuming application, or they wish to expose the > internal checkpointed offsets to measure consumer lag using Kafka toolings. > The main advantage for this change is that the API of {{FlinkKafkaConsumer}} > can speak for itself that it does not depend on committed offsets in Kafka > (this is a misconception that users frequently have), and that exactly-once > depends solely on offsets checkpointed internally using Flink's checkpointing > mechanics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)