Hi everyone, I'd like to start a discussion on FLIP-601: Cloud-Agnostic Client-Side Encryption for SDK-Native Filesystems [1].
For some production Flink use cases server-side encryption is insufficient or impractical. For example, using SSE with private networking in Azure requires enabling the "Trusted Microsoft Services" exception on the key vault, broadening access beyond the client. In GCS, similar challenges arise with the need to allow both ingress and egress traffic. In some deployments the cloud provider cannot be trusted at all and data must be encrypted before it leaves the client process. This FLIP builds on the SDK-native filesystem effort (FLIP-597 [2], FLIP-555 [3], FLIP-598 [4]) to add an optional client-side encryption (CSE) layer with pluggable key providers that works with any SDK-native filesystem plugin. The FLIP proposes two new modules under flink-filesystems/: - flink-fs-cse-common — cloud-agnostic interfaces and configuration. Defines KeyProviderFactory, KeyProvider, CseStreamFactory, and EncryptionMetadata. - flink-fs-cse-aes-gcm — AES-256-GCM implementation backed by BouncyCastle FIPS. CSE is transparent to checkpoint coordinators and state backends — the filesystem layer encrypts on write and decrypts on read. Configuration uses three fs.cse.* keys: a factory class, a write key ID, and optional encryption context entries. This gives three operational modes: disabled (no factory), read-only decryption (factory but no write key), and full CSE (both configured). There are a few points I'd like to highlight for discussion: 1. BouncyCastle FIPS ships as a separate JAR in the filesystem plugin directory (not shaded). This requires adding Flink to the ASF export classifications page [5] and filing a BIS/NSA notification per ASF crypto policy [6]. Are there concerns about the export compliance process or the BC FIPS packaging approach? 2. FileSystem has no close() lifecycle hook, so the KeyProvider (which may hold connections to a key management service) lives for the JVM lifetime. Should a FileSystem.close() hook be added as part of this FLIP, or addressed separately? Looking forward to your feedback. Best regards, Aleksandr Iushmanov [1] https://cwiki.apache.org/confluence/x/P4E_Gg [2] https://cwiki.apache.org/confluence/x/9gDuGQ [3] https://cwiki.apache.org/confluence/x/uYqmFw [4] https://cwiki.apache.org/confluence/x/DQHuGQ [5] https://www.apache.org/licenses/exports/ [6] https://infra.apache.org/crypto.html
