Re: Upgrading a job in rolling mode

2022-06-29 Thread Robin Cassan
Hi Weihua! Thanks so much for your answer, it's pretty apparent that it's absolutely not trivial to do with your explanation. Have you (or any other) been able to apply something close to this Green/Blue deployment idea from this Zero Downtime Upgrade presentation: https://www.youtube.com/watch?v=H

Re: Upgrading a job in rolling mode

2022-06-29 Thread Weihua Hu
Hi, Robin This is a good question, but Flink can't do rolling upgrades. I'll try to explain the cost of Flink's support for RollingUpgrade. 1. There is a shuffle connection between tasks in a Region, and in order to ensure the consistency of the data processed by the upstream and downstream tasks

Upgrading a job in rolling mode

2022-06-29 Thread Robin Cassan
Hi all! We are running a flink cluster on kubernetes and deploying a single job on it through "flink run ". Whenever we want to modify the jar, we cancel the job and run the "flink run" command again, with the new jar, and the retained checkpoint URL from the first run. This works well, but this ad