RE: FlinkKafkaConsumer -> KafkaSource State Migration
Thanks Fabian, That was the information I was missing. (Late reply ... same here, FlinkForward 😊 ) Thias -Original Message- From: Fabian Paul Sent: Donnerstag, 28. Oktober 2021 08:38 To: Schwalbe Matthias Cc: Mason Chen ; user Subject: Re: FlinkKafkaConsumer -> KafkaSource State Migration Hi, Sorry for the late reply but most of use were involved in the Flink Forward conference. The upgrade strategies for the Kafka sink and source are pretty similar. Source and sink do not rely on state migration but leveraging Kafka as source of truth. When running with FlinkKafkaConsumer Mason pointed out correctly that you have stop the job with a save point and set `setCommittedOffsetsOnCheckpoint(true)` [1]. For the FlinkKafkaProducer it is similar on a final savepoint the producer will finalize all pending transactions and submit them to Kafka. The KafkaSink can start without the need of any state migration because there should not be any pending transactions anymore. I do not think you must use `allowNonRestoredState` because there shouldn’t be any state anymore after stopping with a savepoint the source or sink. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#upgrading-to-the-latest-connector-version Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
Re: FlinkKafkaConsumer -> KafkaSource State Migration
Hi, Sorry for the late reply but most of use were involved in the Flink Forward conference. The upgrade strategies for the Kafka sink and source are pretty similar. Source and sink do not rely on state migration but leveraging Kafka as source of truth. When running with FlinkKafkaConsumer Mason pointed out correctly that you have stop the job with a save point and set `setCommittedOffsetsOnCheckpoint(true)` [1]. For the FlinkKafkaProducer it is similar on a final savepoint the producer will finalize all pending transactions and submit them to Kafka. The KafkaSink can start without the need of any state migration because there should not be any pending transactions anymore. I do not think you must use `allowNonRestoredState` because there shouldn’t be any state anymore after stopping with a savepoint the source or sink. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#upgrading-to-the-latest-connector-version
RE: FlinkKafkaConsumer -> KafkaSource State Migration
I would also be interested on instructions/discussion on how to state-migrate from pre-unified sources/sinks to unified ones (Kafka) 😊 Thias From: Mason Chen Sent: Mittwoch, 27. Oktober 2021 01:52 To: user Subject: FlinkKafkaConsumer -> KafkaSource State Migration Hi all, I read these instructions for migrating to the KafkaSource: https://nightlies.apache.org/flink/flink-docs-release-1.14/release-notes/flink-1.14/#deprecate-flinkkafkaconsumer. Do we need to employ any uid/allowNonRestoredState tricks if our Flink job is also stateful outside of the source? Or what is the mechanism that resolves the topic/partition/offsets in the stateful upgrade? Will restoring from FlinkKafkaConsumer cause an exception due to incompatibility of the union state to the current (what is it again)? Best, Mason Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
