Re: Flink with Ceph as the persistent storage

2017-12-05 Thread Gyula Fóra
It would be the same as with any other form of async checkpointing. No direct blocking of processing but the network traffic might indirectly affect it to some extent :) Jayant Ameta ezt írta (időpont: 2017. dec. 5., K, 12:15): > If the checkpointing to Ceph happens asynchronously, does it still

Re: Flink with Ceph as the persistent storage

2017-12-05 Thread Jayant Ameta
If the checkpointing to Ceph happens asynchronously, does it still have any impact on the stream processing? Jayant Ameta On Tue, Dec 5, 2017 at 4:34 PM, Gyula Fóra wrote: > Hi, > > To my understanding Ceph as in http://ceph.com/ceph-storage/ is a block > based object storage system. You can u

Re: Flink with Ceph as the persistent storage

2017-12-05 Thread Gyula Fóra
Hi, To my understanding Ceph as in http://ceph.com/ceph-storage/ is a block based object storage system. You can use it mounted to your server and will behave as a local file system to most extent but will be shared in the cluster. The performance might not be as good as with HDFS to our experie

Flink with Ceph as the persistent storage

2017-12-05 Thread Jayant Ameta
Hi, Flink documents suggests that Ceph can be used as a persistent storage for states. https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/checkpointing.html Considering that Ceph is a transactional database, wouldn't it have adverse effect on Flink's performance?