For point (1) above: Its up to user to have proper sink and source to choose to have exactly once semantics as per the documentation: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/guarantees.html If we choose the supported source and sink combinations duplicates will be avoided.
For point (2) If the communication breaks across Job manager and task manager during the save point or checkpoint operation, checkpoint/save point will be declined. We can't have them Regards Bhaskar On Sat, Mar 14, 2020 at 4:54 PM shravan <mysore.damoda...@microfocus.com> wrote: > Our understanding is to stop job with savepoint, all the task manager > will persist their state during savepoint. If a Task Manager receives a > shutdown signal while savepoint is being taken, does it complete the > savepoint before shutdown ? > [Ans ] Why task manager is shutdown suddenly? Are you saying about handling > unpredictable shutdown while taking > savepoint? In that case You can also use retained check point. In case > current checkpoint has issues because of shutdown > you will have previous checkpoint. So that you can use it. Now you will > have > 2 options, either savepoint/checkpoint. One of them > will always be available. > *[Followup Question]* When the processing service is shutdown say for > maintenance as it is a graceful shutdown we are looking at means to avoid > duplicates as exactly once message processing is guaranteed by our service > . > We are already starting the job based on checkpoint or savepoint whichever > is the latest. When the job is started from last good checkpoint it results > in duplicates. > > The job manager K8S service is configured as remote job manager address > in Task Manager. This service may not be available during savepoint, will > this affect the communication between Task Manager and Job Manager during > savepoint ? > [Ans] you can go for HA right? Where you can run more than one jobmanager > so > that one is always service is available > *[Followup Question]* As we mentioned above processing service is shut down > for maintenance. > > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
