Hi Prasanna, I'd like to add my two cents here. I would not say using the incremental checkpoint is always the best choice. It might have its downsides when restoring from the checkpoint as it will have to apply all the deltas. Therefore restoring from a non-incremental checkpoint might be faster.
As Yun Tang, mentioned the incremental checkpoints are supported by RocksDB only. You don't necessarily need the RocksDB state backend in all cases. If you are sure that the state will fit into the memory (it is probably the case for such a simple job, especially if the map function is stateless), you should be good with the Filesystem state backend[1]. This state backend should be faster as it does not need to spill anything to disk and keeps everything in a deserialized form during the runtime. You might also find this short post[2] helpful. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/state/state_backends.html#the-fsstatebackend [2] https://www.ververica.com/blog/stateful-stream-processing-apache-flink-state-backends On 08/07/2020 05:25, Yun Tang wrote: > Hi Prasanna > > Using incremental checkpoint is always better than not as this is > faster and less memory consumed. > However, incremental checkpoint is only supported by RocksDB > state-backend. > > > Best > Yun Tang > ------------------------------------------------------------------------ > *From:* Prasanna kumar <prasannakumarram...@gmail.com> > *Sent:* Tuesday, July 7, 2020 20:43 > *To:* dev@flink.apache.org <dev@flink.apache.org>; user > <u...@flink.apache.org> > *Subject:* Check pointing for simple pipeline > > Hi , > > I have pipeline. Source-> Map(JSON transform)-> Sink.. > > Both source and sink are Kafka. > > What is the best checkpoint ing mechanism? > > Is setting checkpoints incremental a good option? What should be > careful of? > > I am running it on aws emr. > > Will checkpoint slow the speed? > > Thanks, > Prasanna.
signature.asc
Description: OpenPGP digital signature
