Till and Stephan, thanks for your clarification.
@Till One more question, from what I have read about the checkpointing [1],
the list operations don't seem likely to be performed frequently, so
storing state backend on s3 shouldn't have any severe impact on flink
performance. Is this assumption right?
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.2/internals/stream_checkpointing.html
-- Ayush
On Fri, May 12, 2017 at 1:05 AM Stephan Ewen <se...@apache.org> wrote:
> Small addition to Till's comment:
>
> In the case where file:// points to a mounted distributed file system
> (NFS, MapRFs, ...), then it actually works. The important thing is that the
> filesystem where the checkpoints go is replicated (fault tolerant) and
> accessible from all nodes.
>
> On Thu, May 11, 2017 at 2:16 PM, Till Rohrmann <trohrm...@apache.org>
> wrote:
>
>> Hi Ayush,
>>
>> you’re right that RocksDB is the recommend state backend because of the
>> above-mentioned reasons. In order to make the recovery properly work, you
>> have to configure a shared directory for the checkpoint data via
>> state.backend.fs.checkpointdir. You can basically configure any file
>> system which is supported by Hadoop (no HDFS required). The reason is that
>> we use Hadoop to bridge between different file systems. The only thing you
>> have to make sure is that you have the respective file system
>> implementation in your class path.
>>
>> I think you can access Windows Azure Blob Storage via Hadoop [1]
>> similarly to access S3, for example.
>>
>> If you use S3 to store your checkpoint data, then you will benefit from
>> all the advantages of S3 but also suffer from its drawbacks (e.g. that list
>> operations are more costly). But these are not specific to Flink.
>>
>> A URL like file:// usually indicates a local file. Thus, if your Flink
>> cluster is not running on a single machine, then this won’t work.
>>
>> [1] https://hadoop.apache.org/docs/stable/hadoop-azure/index.html
>>
>> Cheers,
>> Till
>>
>>
>> On Thu, May 11, 2017 at 10:41 AM, Ayush Goyal <ay...@helpshift.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I had a few questions regarding checkpoint storage options using
>>> RocksDBStateBackend. In the flink 1.2 documentation, it is the
>>> recommended state
>>> backend due to it's ability to store large states and asynchronous
>>> snapshotting.
>>> For high availabilty it seems HDFS is the recommended store for state
>>> backend
>>> data. In AWS deployment section, it is also mentioned that s3 can be
>>> used for
>>> storing state backend data.
>>>
>>> We don't want to depend on a hadoop cluster for flink deployment, so I
>>> had
>>> following questions:
>>>
>>> 1. Can we use any storage backend supported by flink for storing RocksDB
>>>
>>> StateBackend data with file urls: there are quite a few supported as
>>> mentioned here:
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/filesystems.html
>>> and here:
>>> https://github.com/apache/flink/blob/master/docs/dev/batch/connectors.md
>>>
>>> 2. Is there some work already done to support Windows Azure Blob Storage
>>> for
>>> storing State backend data? There are some docs here:
>>> https://github.com/apache/flink/blob/master/docs/dev/batch/connectors.md
>>> can we utilize this for that?
>>>
>>> 3. If utilizing S3 for state backend, is there any performance impact?
>>>
>>> 4. For high availability can we use a NFS volume for state backend, with
>>>
>>> "file://" urls? Will there be any performance impact?
>>>
>>> PS: I posted this email earlier via nabble, but it's not showing up in
>>> apache archive. So sending again. Apologies if it results in multiple
>>> threads.
>>>
>>> -- Ayush
>>>
>>
>>
>
