Hi, I think atomic rename is not part of the requirements. I'll add +Stephan who recently wrote this document in case he has any additional input.
Cheers, Aljoscha On Thu, 16 Feb 2017 at 23:28 Vijay Srinivasaraghavan <vijikar...@yahoo.com> wrote: > Following up on my question regarding backed Filesystem (HCFS) > requirements. Appreciate any inputs. > > --- > Regarding the Filesystem abstraction support, we are planning to use a > distributed file system which complies with Hadoop Compatible File System > (HCFS) standard in place of standard HDFS. > > According to the documentation ( > https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/filesystems.html), > persistence gurantees is listed as one of the main requirement and to be > precises it qualifies both visibility and durability gurantees. > > My question is, > > 1) Are we expecting the file system to support "Atomic Rename" > characteristics? I believe checkpoint mechanism involves in renaming the > files and will that have an impact if "atomic rename" is not guranteed by > the underlying file system? > > 2) How does one certify Flink with HCFS (in place of standard HDFS) in > terms of the scenarios/usecase that needs to be tested? Is there any > general guidance on this? > --- > > Regards > Vijay > > > On Wednesday, February 15, 2017 11:28 AM, Vijay Srinivasaraghavan < > vijikar...@yahoo.com> wrote: > > > Hello, > > Regarding the Filesystem abstraction support, we are planning to use a > distributed file system which complies with Hadoop Compatible File System > (HCFS) standard in place of standard HDFS. > > According to the documentation ( > https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/filesystems.html), > persistence gurantees is listed as one of the main requirement and to be > precises it qualifies both visibility and durability gurantees. > > My question is, > > 1) Are we expecting the file system to support "Atomic Rename" > characteristics? I believe checkpoint mechanism involves in renaming the > files and will that have an impact if "atomic rename" is not guranteed by > the underlying file system? > > 2) How does one certify Flink with HCFS (in place of standard HDFS) in > terms of the scenarios/usecase that needs to be tested? Is there any > general guidance on this? > > Thanks > Vijay > > >
