Thank you Piotr! One last question - let's assume my source is a Kafka topic - if I stop via the CLI with a savepoint in Flink 1.9, but do not use that savepoint when restarting my job - the job would continue from the last offset that has been committed in Kafka and thus I would also not experience a loss of data in my sink. Is that correct?
Best, Tobi On Fri, Feb 28, 2020 at 3:17 PM Piotr Nowojski <pi...@ververica.com> wrote: > Yes, that’s correct. There shouldn’t be any data loss. Stop with savepoint > is a solution to make sure, that if you are stopping a job (either > permanently or temporarily) that all of the results are published/committed > to external systems before you actually stop the job. > > If you just cancel/kill/crash a job, in some rare cases (if a checkpoint > was completing at the time cluster was crashing), some records might not be > committed before the cancellation/kill/crash happened. Also note that > doesn’t mean there is a data loss, just those records will be published > once you restore your job from a checkpoint. If you want to stop the job > permanently, that might not happen, hence we need stop with savepoint. > > Piotrek > > On 28 Feb 2020, at 15:02, Kaymak, Tobias <tobias.kay...@ricardo.ch> wrote: > > Thank you! For understanding the matter: When I have a streaming pipeline > (reading from Kafka, writing somewhere) and I click "cancel" and after that > I restart the pipeline - I should not expect any data to be lost - is that > correct? > > Best, > Tobias > > On Fri, Feb 28, 2020 at 2:51 PM Piotr Nowojski <pi...@ververica.com> > wrote: > >> Thanks for confirming that Yadong. I’ve created a ticket for that [1]. >> >> Piotrek >> >> [1] https://issues.apache.org/jira/browse/FLINK-16340 >> >> On 28 Feb 2020, at 14:32, Yadong Xie <vthink...@gmail.com> wrote: >> >> Hi >> >> 1. the old stop button was removed in flink 1.9.0 since it could not >> work properly as I know >> 2. if we have the feature of the stop with savepoint, we could add it to >> the web UI, but it may still need some work on the rest API to support the >> new feature >> >> >> Best, >> Yadong >> >> >> Piotr Nowojski <pi...@ververica.com> 于2020年2月28日周五 下午8:49写道： >> >>> Hi, >>> >>> I’m not sure. Maybe Yadong (CC) will know more, but to the best of my >>> knowledge and research: >>> >>> 1. In Flink 1.9 we switched from the old webUI to a new one, that >>> probably explains the difference you are seeing. >>> 2. The “Stop” button in the old webUI, was not working properly - that >>> was not stop with savepoint, as stop with savepoint is a relatively new >>> feature. >>> 3. Now that we have stop with savepoint (it can be used from CLI as you >>> wrote), probably we could expose this feature in the new UI as well, unless >>> it’s already exposed somewhere? Yadong, do you know an answer for that? >>> >>> Piotrek >>> >>> On 27 Feb 2020, at 13:31, Kaymak, Tobias <tobias.kay...@ricardo.ch> >>> wrote: >>> >>> Hello, >>> >>> before Flink 1.9 I was able to "Stop" a streaming pipeline - after >>> clicking that button in the webinterface it performed a clean shutdown. Now >>> with Flink 1.9 I just see the option to cancel it. >>> >>> However, using the commandline flink stop -d >>> 266c5b38cf9d8e61a398a0bef4a1b350 still does the trick. So the >>> functionality is there. >>> >>> Has the button been removed on purpose? >>> >>> Best, >>> Tobias >>> >>> >>> >> > > -- > > Tobias Kaymak > Data Engineer > Data Intelligence > > tobias.kay...@ricardo.ch > www.ricardo.ch > Theilerstrasse 1a, 6300 Zug > > >
