Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hello, before Flink 1.9 I was able to "Stop" a streaming pipeline - after clicking that button in the webinterface it performed a clean shutdown. Now with Flink 1.9 I just see the option to cancel it. However, using the commandline flink stop -d 266c5b38cf9d8e61a398a0bef4a1b350 still does the tri

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hi, I’m not sure. Maybe Yadong (CC) will know more, but to the best of my knowledge and research: 1. In Flink 1.9 we switched from the old webUI to a new one, that probably explains the difference you are seeing. 2. The “Stop” button in the old webUI, was not working properly - that was not st

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Thanks for confirming that Yadong. I’ve created a ticket for that [1]. Piotrek [1] https://issues.apache.org/jira/browse/FLINK-16340 > On 28 Feb 2020, at 14:32, Yadong Xie wrote: > > Hi > > 1. the old stop button was removed in flink 1.9.0

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Thank you! For understanding the matter: When I have a streaming pipeline (reading from Kafka, writing somewhere) and I click "cancel" and after that I restart the pipeline - I should not expect any data to be lost - is that correct? Best, Tobias On Fri, Feb 28, 2020 at 2:51 PM Piotr Nowojski wr

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Yes, that’s correct. There shouldn’t be any data loss. Stop with savepoint is a solution to make sure, that if you are stopping a job (either permanently or temporarily) that all of the results are published/committed to external systems before you actually stop the job. If you just cancel/kil

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Thank you Piotr! One last question - let's assume my source is a Kafka topic - if I stop via the CLI with a savepoint in Flink 1.9, but do not use that savepoint when restarting my job - the job would continue from the last offset that has been committed in Kafka and thus I would also not experien

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hi Tobi, In this case, the job would indeed continue from the last offset that has been committed in Kafka (assuming that you are using the `startFromGroupOffsets` start position) for the specified group id. However, do keep in mind that those offsets are not consistent with the offsets written in

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hi Tobi, No, FlinkKafkaConsumer is not using committed Kafka’s offsets for recovery. Offsets where to start from are stored in the checkpoint itself. Updating the offsets back to Kafka is an optional, purely cosmetic thing from the Flink’s perspective, so the job will start from the correct off

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hi, let me refine my question: My pipeline is generated from Beam, so the Flink pipeline is a translated Beam pipeline. When I update my Apache Beam pipeline code, working with a snapshot in Flink to stop the pipeline is not an option, as the snapshot will use the old representation of the the Fli

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hi, Sorry for my previous slightly confusing response, please take a look at the response from Gordon. Piotrek > On 2 Mar 2020, at 12:05, Kaymak, Tobias wrote: > > Hi, > > let me refine my question: My pipeline is generated from Beam, so the Flink > pipeline is a translated Beam pipeline. W

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Thank you! One last question regarding Gordons response. When a pipeline stops consuming and cleanly shuts down and there is no error during that process, and then it gets started again and uses the last committed offset in Kafka - there should be no data loss - or am I missing something? In what

Re: Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hi Kaymak, To answer your last question: there will be no data loss in that scenario you described, but there could be duplicate processed records. With checkpointing enabled, the Flink Kafka consumer does not commit offsets back to Kafka until offsets in Flink checkpoints have been persisted. T