Hi,
We are trying to test the recovery mechanism of Flink with Kafka and HDFS sink
during failures.
I’ve killed the job after processing some messages and restarted the same job
again. Some of the messages I am seeing are processed more than once and not
following the exactly once semantics.
.org>"
mailto:user@flink.apache.org>>
Subject: Flink recovery
Hi,
We are trying to test the recovery mechanism of Flink with Kafka and HDFS sink
during failures.
I’ve killed the job after processing some messages and restarted the same job
again. Some of the messages I am seeing
een
>
> From: "Madhire, Venkat Naveen Kumar Reddy"
> Reply-To: "user@flink.apache.org"
> Date: Friday, May 13, 2016 at 10:58 AM
> To: "user@flink.apache.org"
> Subject: Flink recovery
>
> Hi,
>
> We are trying to test the recovery mechanis
gt;"
mailto:user@flink.apache.org>>
Date: Friday, May 13, 2016 at 4:13 PM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Flink recovery
Hi,
Flink's exactly-once semantics do not mean that events are processed
exact
Reply-To: "user@flink.apache.org"
> Date: Friday, May 13, 2016 at 4:13 PM
> To: "user@flink.apache.org"
> Subject: Re: Flink recovery
>
> Hi,
>
> Flink's exactly-once semantics do not mean that events are processed
> exactly-once but that events
gt;"
mailto:user@flink.apache.org>>
Date: Friday, May 13, 2016 at 4:26 PM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Flink recovery
Hi Naveen,
the RollingFileSink supports exactly-once output. So you should be good
t; pipeline.
>
>
>
> Thanks,
> Naveen
>
> From: Fabian Hueske
> Reply-To: "user@flink.apache.org"
> Date: Friday, May 13, 2016 at 4:26 PM
>
> To: "user@flink.apache.org"
> Subject: Re: Flink recovery
>
> Hi Naveen,
>
> the Ro
"user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Date: Saturday, May 14, 2016 at 4:10 AM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Flink recovery
The behavior of the
link.apache.org"
> Date: Saturday, May 14, 2016 at 4:10 AM
> To: "user@flink.apache.org"
> Subject: Re: Flink recovery
>
> The behavior of the RollingFileSink depends on the capabilities of the
> file system.
> If the file system does not support to truncate
>> Thank you.
>>
>> From: Fabian Hueske
>> Reply-To: "user@flink.apache.org"
>> Date: Saturday, May 14, 2016 at 4:10 AM
>> To: "user@flink.apache.org"
>> Subject: Re: Flink recovery
>>
>> The behavior of the RollingFileSink de
mailto:user@flink.apache.org>>
Subject: Re: Flink recovery
Hi Naveen,
I think cancelling a job is not the right approach for testing our exactly-once
guarantees. By cancelling a job, you are discarding the state of your job.
Restarting from scratch (without using a savepoint) will cause dupli
e, application
> upgrade and node failure.
>
>
>
> Thanks,
> Naveen
>
> From: Robert Metzger
> Reply-To: "user@flink.apache.org"
> Date: Tuesday, May 17, 2016 at 6:58 AM
> To: "user@flink.apache.org"
> Subject: Re: Flink recovery
>
gt;"
mailto:user@flink.apache.org>>
Date: Tuesday, May 17, 2016 at 10:02 AM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Flink recovery
Hi,
Savepoints are exactly for that use case:
https://ci.apache.o
er
> Reply-To: "user@flink.apache.org"
> Date: Tuesday, May 17, 2016 at 10:02 AM
>
> To: "user@flink.apache.org"
> Subject: Re: Flink recovery
>
> Hi,
>
> Savepoints are exactly for that use case:
> https://ci.apache.org/projects/flink/flink-docs
14 matches
Mail list logo