rmoff commented on code in PR #860: URL: https://github.com/apache/flink-web/pull/860#discussion_r3465922732
########## docs/content/posts/2026-06-14-announcing-native-s3-fs.md: ########## @@ -0,0 +1,200 @@ +--- +title: "Introducing Flink's Native S3 FileSystem: Built for Performance, Designed for Production" +date: "2026-06-14T08:00:00.000Z" +slug: "announcing-native-s3-fs" +url: "/2026/06/14/announcing-native-s3-fs/" +authors: +- gaborgsomogyi: + name: "Gabor Somogyi" +- Samrat002: + name: "Samrat Deb" +aliases: +- /news/2026/06/14/announcing-native-s3-fs.html +--- + +Apache Flink relies on the underlying filesystem for much of its work: reading and writing application data, materializing streaming sinks, and storing checkpoints and savepoints for recovery. For years, S3 support in Flink meant choosing between two Hadoop-based plugins, each with its own trade-offs and configuration quirks. With Flink 2.3, there is a better option. + +Today we're introducing `flink-s3-fs-native`, a ground-up, Hadoop-free S3 filesystem built specifically for Flink. It ships as an [experimental opt-in plugin](#availability-and-roadmap) in Flink 2.3, is already running in production at scale at major technology companies, and delivers measurable, reproducible performance gains. + + +**At a glance** + +| | | +|---|---| +| **[~2x faster checkpoints](#performance)** | 48.8 s average vs 90.1 s with the Presto plugin; up to 4.5x at small state sizes | +| **Drop-in replacement** | Swap the JAR, keep your existing `flink-conf.yaml`, restart your cluster | +| **No Hadoop dependency** | ~13 MB JAR vs ~30–93 MB; no CVE triage on Hadoop transitive dependencies | +| **[AWS SDK v2](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html)** | Async-first I/O; [AWS SDK v1 reaches end-of-support on December 31, 2025](https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/) | +| **One plugin for everything** | Exactly-once sinks and fast checkpoints — no trade-offs, no compromises | + + +## Two Plugins, One Filesystem, and No Good Answer + +If you've configured S3 for Flink before, you likely know that Flink ships two S3 filesystem plugins, and both register on the same `s3://` scheme. Only one can be active at a time. Choosing between them has been a source of confusion for years. Review Comment: ```suggestion If you've configured S3 for Flink before, you likely know that Flink ships two S3 filesystem plugins, and both register on the same `s3://` scheme. Only one can be active at a time. Choosing between them has been a source of confusion for years. Even once one has been chosen, its use still [perplexes](https://rmoff.net/2024/08/06/troubleshooting-flink-sql-s3-problems/) many end-users because of the similarly-named but different configurations required. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
