Hi everyone, I'd like to start a discussion on FLIP-598: Gen2 Datalake SDK Native FileSystem for Azure [1].
This FLIP is part of the broader effort to move Flink filesystems away from Hadoop dependencies, alongside FLIP-597 [2] (common abstractions) and FLIP-555 [3] (native S3). It proposes flink-azure-fs-native, a new filesystem module built directly on the Azure Storage File DataLake SDK v12 [4] to replace the Hadoop-based flink-fs-azure. The long-term goal is for each major cloud to have a lean, SDK-native filesystem plugin that ships without Hadoop Common, reduces CVE exposure, and can adopt cloud SDK improvements directly. The FLIP covers authentication, configuration, read/write paths, directory operations and RecoverableWriter. Flink state is bi-directionally compatible with the Hadoop-based module to allow safe migration. Looking forward to your feedback. Best regards, Aleksandr Iushmanov [1] https://cwiki.apache.org/confluence/x/DQHuGQ [2] https://cwiki.apache.org/confluence/x/9gDuGQ [3] https://cwiki.apache.org/confluence/x/uYqmFw [4] https://techcommunity.microsoft.com/blog/azurestorageblog/announcing-the-azure-storage-v12-client-libraries/1482394
