Hi Aleksandr, I have been applying the proposed interfaces to the native GCS prototype. I encountered a few questions that I would like to understand better. Some may come from gaps in my prototype or my understanding of the intended layering, so I would appreciate corrections and guidance.
1. Scope and interface layering Would it be helpful for the FLIP to show three explicit layers? - Existing Flink contracts, such as FileSystem and RecoverableWriter - Shared object-storage mechanics such as position tracking and ranged reopening - Provider-specific SDK adapters and capabilities My current impression is that ReadContext, StreamContext, InputStreamOpener, InputStreamExtension, and RawAndWrappedInputStreams overlap somewhat. Could we initially reduce this to a position-aware stream factory or handle, and add further context only when a second use case requires it? I would also like to understand the reason for placing the implementations in flink-core rather than a small flink-object-storage-fs-base module. There may be plugin classloading constraints that I have missed. 2. Thread safety and cancellation In the GCS prototype, locking every read affected the throughput. There is also a Cancellation concern: when a lock is held during a blocking SDK read, a Concurrent close cannot acquire the lock to abort that operation. Would a narrower contract be sufficient: reads and writes remain thread- Is confined, while close or abort is safe to invoke concurrently during task cancellation? If full thread safety is required, it would help to document the Flink caller that requires it. I will share the GCS benchmark results and a blocked-read cancellation test so that we can evaluate this with data rather than assumptions. 3. Stream ownership, CRT, and encryption I understand the need to support normal SDK streams, AWS CRT, buffering, and future client-side encryption through one model. The remaining question for me is lifecycle ownership. S3 may require aborting a response rather than closing and draining it, while GCS and Azure expose different channel semantics. RawAndWrappedInputStreams may therefore need more explicit rules for ownership, close ordering, and abort behaviour. Could the opener return an owned handle containing the readable stream and its abort/close operation? This might let each provider adapt its native stream without introducing an intermediate buffer. I will test this approach with both the GCS prototype and the native S3 implementation. For client-side encryption, I am still gathering requirements from the security discussion. Random access will require more than metadata injection because logical offsets, encrypted chunk boundaries, integrity tags, and physical object lengths must be considered. I will share the header-injection and range-read requirements once that discussion concludes. 4. Existing open questions For RecoverableWriter, my current understanding is that RecoverableWriter itself is already the cloud-neutral abstraction, while the recovery protocol should remain provider-specific. S3 uses multipart state, whereas the existing GCS writer uses immutable component objects and composes so that older checkpoint handles remain valid after writing continues. Did you envisage a new common implementation here, or only shared utilities and contract tests? For PathsCopyingFileSystem, I believe the existing interface already provides the common capability. Each provider could implement it independently when an optimised transfer path is available. For ObjectStorageOperations, I understand the motivation, especially given the declining availability of reliable emulators. One possible incremental approach would be to use provider-local SDK adapters initially, and share FileSystem behaviour tests, and extract a common operations interface once S3, GCS and Azure demonstrate a stable intersection. If the intended interface is narrower than a complete object-store abstraction, documenting the exact operations and semantics would help me evaluate it against GCS. 5. Exception classification I am interested in connecting this work with the exception-classification mechanism. It would be useful to align common categories such as authentication, not-found, precondition failure, throttling, timeout, and cancellation while retaining the original SDK exception as the cause. Should this classification be part of FLIP-597, or should the filesystem preserve provider exceptions and let the classification plugin handle them in a follow-up? I would be happy to prepare a comparison of the AWS, GCS, and Azure exception models. One final clarification concerns migration. I assume deprecation will be evaluated separately for each cloud plugin rather than applying to flink-fs- hadoop-shaded generally, because HDFS and the remaining Hadoop-based integrations still require it. We should also document that in-progress RecoverableWriter State may not be portable between Hadoop and native implementations. I will follow up with the GCS prototype, performance notes, the S3/GCS behaviour comparison, and the encryption requirements. I am also happy to help refine the interface documentation or add shared contract tests. These questions are intended to help us arrive at the smallest abstraction that preserves correctness and performance across providers. I support the Hadoop-less direction, and I am keen to learn from the Azure and S3 perspectives so that we can help finalise FLIP-597. Bests, Samrat On Thu, Jun 25, 2026 at 4:39 PM Aleksandr Iushmanov <[email protected]> wrote: > Hi everyone, > > I'd like to start a discussion on FLIP-597: Hadoop-less Flink Filesystems > [1]. > > As FLIP-555 [2] established, Flink's Hadoop-based filesystem plugins > carry significant maintenance burdens: transitive dependency conflicts, > classpath issues, and CVE exposure. FLIP-555 addressed this for S3 by > introducing a native SDK-based filesystem. This FLIP extends that effort > as an umbrella for all cloud providers (Azure, GCS, and potentially > others) with the end goal to deprecate Hadoop-based variants. > > The FLIP proposes common, cloud-agnostic I/O abstractions placed in > flink-core that all SDK-native filesystem implementations share as > a step towards Hadoop-less Flink deployments: > > - ObjectStorageInputStream / ObjectStorageOutputStream: thread-safe > stream implementations. > - InputStreamExtension / InputStreamOpener: composable read pipeline > supporting buffering, decryption via decoration > - WriteContext / ReadContext: metadata descriptors enabling features > like client-side encryption without changing the FileSystem API > > There are a number of open questions for discussion: > 1. Should a cloud-agnostic RecoverableWriter abstraction be part of > this FLIP given that implementation across clouds may be too specific > for common abstractions to be useful. > 2. Should PathsCopyingFileSystem (bulk copy) be in scope? Transfer managers > for clouds vary. And currently we only support this implementation in > AWS. > 3. Should this FLIP include an ObjectStorageOperations interface > (analogous to FLIP-555's S3AccessHelper) for a generalized > ObjectStorageFileSystem base, or should we defer premature > generalisation > of testing setup? Main motivation for this would be the ability to > verify > common FS parts with testing doubles with reduced reliance on other > libraries > like MinIO [3] (archived), LocalStack [4] (archived), Azurite [5] (no > SDK V2 support) > that demonstrates a decline in support. > > Looking forward to your feedback. > > Best regards, > Aleksandr Iushmanov > > [1] https://cwiki.apache.org/confluence/x/9gDuGQ > [2] https://cwiki.apache.org/confluence/x/uYqmFw > [3] https://github.com/minio/minio > [4] https://github.com/localstack/localstack > [5] https://github.com/Azure/Azurite >
