Re: data loss after implementing checkpoint

Hi Sridhar, Stephan already covered the correct sequence of actions in order for your second program to know its correct starting point. As far as the active/inactive rules are concerned, as Nico pointed out you have to somehow store in the backend which rules are active and which are not

Re: data loss after implementing checkpoint

Maybe to clear up some confusion here: - Flink recovers from the latest checkpoint after a failure - If you stopping/cancelling a Flink job and submit the job again, it does not automatically pick up the latest checkpoint. Flink does not know that the second program is a continuation of the

Re: data loss after implementing checkpoint

Hi Sridhar, sorry for not coming back to you earlier and tbh, I'm no expert on this field either. I don't see this enabling/disabling of rules in the CEP library overview at [1]. How do you do this? You'll probably have to create a stateful operator [2] to store this state in Flink. Maybe

Re: data loss after implementing checkpoint

The CEP library runs Correction; the CEP implemented using the CEP library runs . On Wed, Jul 19, 2017 at 10:08 AM, Sridhar Chellappa wrote: > A follow up question on this. I have a Complex Event processor implemented > using the CEP library (1.3.0). The CEP

Re: data loss after implementing checkpoint

A follow up question on this. I have a Complex Event processor implemented using the CEP library (1.3.0). The CEP library runs a variety of rules that are configured (enable/disable rule) VIA REST APIs. Now, if my application crashes and recovers (or is cancelled and restarted), will my

Re: data loss after implementing checkpoint

(back to list) state.checkpoints.dir is a configuration parameter which you set in the flink configuration itself (see [1]). This will be used for checkpoint metadata only (for RocksDB and Fs) while the checkpoints themselves are stored in the given directory. Nico [1]

Re: data loss after implementing checkpoint

Hi Aftab, looks like what you want is either an externalized checkpoint with RETAIN_ON_CANCELLATION mode [1] or a savepoint [2]. Ordinary checkpoints are deleted when the job is cancelled and only serve as a fault tolerance layer in case something goes wrong, i.e. machines fail, so that the

Re: data loss after implementing checkpoint

Hi, I think you need to create a savepoint and restore from there. https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/savepoints.html Checkpoint are for automatic recovery within the lifetime of a job, they're deleted when you stop the job manually. Regards, Kien On

data loss after implementing checkpoint

Hi, I am new to flink. I am facing issue implementing checkpoint. checkpoint related code: long checkpointInterval = 5000; StreamExecutionEnvironment env = StreamUtils.getEnvironment(params); //specify backend //env.setStateBackend(new FsStateBackend("s3n://xxx/flink-state/"), true));