Hi Flink dev community,
I'd like to start a discussion about the long-term support scope of
flink-s3-fs-native.
Background
==========
Recently I submitted PR #27964 / FLINK-39485:
"NativeS3FileSystem: support bucket-root paths (empty object key)".
The change fixes an IllegalArgumentException: "Key cannot be empty"
when using s3://bucket (without any object key) on S3-compatible stores
like MinIO. It has ZERO impact on pure Amazon S3, as AWS S3 already
supports this case.
During review:
- spuru9 suggested adding a test or documenting manual validation.
- Samrat suggested we should first discuss with the community whether
we want to continue 100% support for MinIO / S3-compatible filesystems
in native-s3-fs, given the added code complexity and maintenance overhead.
Current Situation
=================
- FLIP-555 (which introduced native-s3-fs) was primarily motivated by
providing a modern, high-performance, Hadoop-free S3 implementation
based on AWS SDK v2, with better alignment to Amazon S3.
- The documentation mentions support for "S3 compliant object stores
such as MinIO and IBM COS", but the exact level of support (strict
AWS S3 vs best-effort compatible) has never been clearly defined.
Proposal
========
I suggest we define clear support tiers for the S3 filesystems:
1. flink-s3-fs-native:
- Primary / strict support for Amazon S3.
- Best-effort (minimal) support for S3-compatible stores.
- Goal: Keep the code clean and maintain high performance.
2. flink-s3-fs-hadoop:
- Main place for broader S3-compatible support (MinIO, Alibaba OSS,
Ceph, etc.).
This approach would reduce long-term maintenance overhead in
native-s3-fs and make expectations clear to users.
Questions for the community
===========================
- Do we agree on introducing explicit support tiers?
- Should native-s3-fs stay focused on AWS S3 first, with compatible
stores being secondary?
- For the current small PR (#27964), since it has no side effects on
AWS S3 and improves usability for many users, can we merge it first
while we discuss the broader policy?
Looking forward to your thoughts.
Thanks,
macdoor
GitHub PR: https://github.com/apache/flink/pull/27964
JIRA: FLINK-39485
