Re: Checkpoint recovery and state external to flink
Hi Yun, This is good information. Thank you. However looks like it only applies to SinkFunction. Any thoughts for when intermediate operators are also interacting with external systems? Thanks. Ajay From: Yun Tang Date: Tuesday, March 5, 2019 at 4:04 AM To: "Aggarwal, Ajay" , "user@flink.apache.org" Subject: Re: Checkpoint recovery and state external to flink Hi Ajay I think two phase commit protocol could solve your concern for the exactly-once external system, Flink already support this feature in some sinks [1], e.g. you could refer to [2] to know which version of Kafaka producer could support exactly-once. [1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html [2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html Best Yun Tang From: Aggarwal, Ajay Sent: Tuesday, March 5, 2019 4:08 To: user@flink.apache.org Subject: Checkpoint recovery and state external to flink What happens when the flink job interacts with a user managed database and hence has some state outside of flink? In these situations when a flink job is recovered from last successful checkpoint, this external state will not be in sync with the recovered flink state. In most cases it will be ahead of the recovered flink state. Any recommendations or best practices to follow here? I am assuming it must be very common for flink applications to interact with external systems (databases, message systems etc.)
Re: Checkpoint recovery and state external to flink
Hi Ajay I think two phase commit protocol could solve your concern for the exactly-once external system, Flink already support this feature in some sinks [1], e.g. you could refer to [2] to know which version of Kafaka producer could support exactly-once. [1] https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html [2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html Best Yun Tang From: Aggarwal, Ajay Sent: Tuesday, March 5, 2019 4:08 To: user@flink.apache.org Subject: Checkpoint recovery and state external to flink What happens when the flink job interacts with a user managed database and hence has some state outside of flink? In these situations when a flink job is recovered from last successful checkpoint, this external state will not be in sync with the recovered flink state. In most cases it will be ahead of the recovered flink state. Any recommendations or best practices to follow here? I am assuming it must be very common for flink applications to interact with external systems (databases, message systems etc.)
Checkpoint recovery and state external to flink
What happens when the flink job interacts with a user managed database and hence has some state outside of flink? In these situations when a flink job is recovered from last successful checkpoint, this external state will not be in sync with the recovered flink state. In most cases it will be ahead of the recovered flink state. Any recommendations or best practices to follow here? I am assuming it must be very common for flink applications to interact with external systems (databases, message systems etc.)
