Tzu-Li (Gordon) Tai created FLINK-5704:
------------------------------------------
Summary: Deprecate FlinkKafkaConsumer constructors in favor of
improvements to decoupling from Kafka offset committing
Key: FLINK-5704
URL: https://issues.apache.org/jira/browse/FLINK-5704
Project: Flink
Issue Type: Improvement
Components: Kafka Connector, Streaming Connectors
Reporter: Tzu-Li (Gordon) Tai
With FLINK-3398 and FLINK-4280, the {{FlinkKafkaConsumer}} will be able to
completely operate independently of committed offsets in Kafka.
I.e.,
1) when starting, the consumer can choose to not use any committed offsets in
Kafka as the starting position
2) the consumer can completely opt-out of committing offsets back to Kafka
However, our current default behaviour for 1) is to respect committed offsets,
and 2) is to always have offset committing. Users still have to call the
respective setter configuration methods to change this.
I think we should deprecate the current constructors in favor of new ones with
default behaviours 1) start from the latest record, without respecting Kafka
offsets, and 2) don't commit offsets.
With this change, users explicitly call the config methods of FLINK-3398 and
FLINK-4280 to *enable* respecting committed offsets for Kafka, instead of
**disabling** it. They would want to / need to enable it, only when perhaps to
migrate from a non-Flink consuming application, or they wish to expose the
internal checkpointed offsets to measure consumer lag using Kafka toolings.
The main advantage for this change is that the API of {{FlinkKafkaConsumer}}
can speak for itself that it does not depend on committed offsets in Kafka
(this is a misconception that users frequently have), and that exactly-once
depends solely on offsets checkpointed internally using Flink's checkpointing
mechanics.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
