Samrat002 commented on code in PR #860: URL: https://github.com/apache/flink-web/pull/860#discussion_r3479371134
########## docs/content/posts/2026-06-14-announcing-native-s3-fs.md: ########## @@ -0,0 +1,200 @@ +--- +title: "Introducing Flink's Native S3 FileSystem: Built for Performance, Designed for Production" +date: "2026-06-14T08:00:00.000Z" +slug: "announcing-native-s3-fs" +url: "/2026/06/14/announcing-native-s3-fs/" +authors: +- gaborgsomogyi: + name: "Gabor Somogyi" +- Samrat002: + name: "Samrat Deb" +aliases: +- /news/2026/06/14/announcing-native-s3-fs.html +--- + +Apache Flink relies on the underlying filesystem for much of its work: reading and writing application data, materializing streaming sinks, and storing checkpoints and savepoints for recovery. For years, S3 support in Flink meant choosing between two Hadoop-based plugins, each with its own trade-offs and configuration quirks. With Flink 2.3, there is a better option. + +Today we're introducing `flink-s3-fs-native`, a ground-up, Hadoop-free S3 filesystem built specifically for Flink. It ships as an [experimental opt-in plugin](#availability-and-roadmap) in Flink 2.3, is already running in production at scale at major technology companies, and delivers measurable, reproducible performance gains. + + +**At a glance** + +| | | +|---|---| +| **[~2x faster checkpoints](#performance)** | 48.8 s average vs 90.1 s with the Presto plugin; up to 4.5x at small state sizes | +| **Drop-in replacement** | Swap the JAR, keep your existing `flink-conf.yaml`, restart your cluster | +| **No Hadoop dependency** | ~13 MB JAR vs ~30–93 MB; no CVE triage on Hadoop transitive dependencies | +| **[AWS SDK v2](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html)** | Async-first I/O; [AWS SDK v1 reaches end-of-support on December 31, 2025](https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/) | +| **One plugin for everything** | Exactly-once sinks and fast checkpoints — no trade-offs, no compromises | + + +## Two Plugins, One Filesystem, and No Good Answer + +If you've configured S3 for Flink before, you likely know that Flink ships two S3 filesystem plugins, and both register on the same `s3://` scheme. Only one can be active at a time. Choosing between them has been a source of confusion for years. + +The **Hadoop plugin** wraps Hadoop's S3A client. It supports `RecoverableWriter`, which enables exactly-once sinks. Unfortunately it pulls in the full `hadoop-common` dependency tree and AWS SDK v1. Configuration uses Hadoop-native keys (`fs.s3a.*`) mirrored to Flink-style keys (`s3.*`) through a compatibility layer. + +The **Presto plugin** was historically recommended for checkpointing because of its faster read path. But it does not support `RecoverableWriter`, which means exactly-once file sinks don't work with it. It carries known [bugs around directory deletion](https://github.com/prestodb/presto/issues/17416) that require Flink-side workarounds. It also depends on `hadoop-common` and AWS SDK v1 under the hood. + +Both share a common base layer that adapts a Hadoop `FileSystem` into a Flink `FileSystem`. This adaptation layer adds indirection, limits Flink-specific optimizations, and ties the implementation to Hadoop's configuration model and SDK lifecycle. + +As a result, you could have exactly-once sinks or a lighter read path, but not both. In addition, you are carrying Hadoop dependency challenges. + +**The native plugin removes the trade-off entirely.** + +--- + +## Why This Matters Beyond Engineering + +The decision to replace the S3 plugin is not just a performance choice. It has direct operational and financial consequences. + +**Security and compliance teams** have long carried the burden of triaging CVEs in `hadoop-common`'s transitive dependency tree. That tree is large, changes frequently, and generates a steady stream of vulnerability disclosures unrelated to S3 or Flink. Removing it permanently eliminates that toil. Fewer dependencies mean fewer CVEs, fewer emergency patch cycles, and fewer security review gates for new deployments. + +**Platform and infrastructure teams** running multi-tenant Flink clusters benefit from a clean, unified `s3.*` configuration namespace. The native plugin's configuration model is designed for Flink. No Hadoop-style key mirroring, no adapter translation layer, no debugging sessions caused by settings silently not propagating. + +**Risk and compliance teams** should note that the AWS SDK for Java 1.x has been in [maintenance mode since July 31, 2024](https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/) and reaches **end-of-support on December 31, 2025**, after which it receives no further updates or releases. The foundation that both existing plugins depend on is therefore reaching end-of-life, which means no new features and a winding-down stream of bug and security fixes. Continuing to operate on SDK v1 is an accumulating technical and compliance liability. The native plugin is built entirely on [AWS SDK v2](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html). + +**Operations teams** benefit from faster checkpoints in two concrete ways: +- Shorter checkpoint windows mean less CPU time spent on state serialization and more capacity for actual data processing. +- Tighter recovery windows mean less data to replay after a failure. This directly improves recovery SLAs at scale. + +## One Stop Solution: Native S3 Filesystem + +| Feature | flink-s3-fs-hadoop | flink-s3-fs-presto | flink-s3-fs-native | +|---|---|---|---| +| Exactly-once FileSink | ✓ | ✗ | ✓ | +| RecoverableWriter | ✓ | ✗ | ✓ | +| Checkpointing | ✓ | ✓ | ✓ | +| AWS SDK v2 | ✗ | ✗ | ✓ | +| No Hadoop dependency | ✗ | ✗ | ✓ | +| SSE-KMS encryption | ✓ | ✓ | ✓ | +| SSE-KMS encryption context | ✗ | ✗ | ✓ | +| Non-blocking NIO async I/O | ✗ | ✗ | ✓ | +| JAR size | ~30 MB | ~93 MB | ~13 MB | + +### Feature highlights + +**No Hadoop dependency tree.** No `hadoop-common`, no `aws-java-sdk` v1, no class-shading conflicts. The native shaded JAR weighs ~13 MB, which is less than half the size of the Hadoop plugin (30 MB) and 7x lighter than the Presto plugin (93 MB). + +**Async-first I/O.** Reads and writes use AWS SDK v2's `S3TransferManager`, backed by Netty NIO multiplexed connections that avoid the thread-per-request bottleneck of the existing plugins. Bulk state restore runs as batched concurrent transfers with connection-pool-aware concurrency control. This is the same mechanism that replaces the need for external tools like `s5cmd`. + +**Exactly-once recoverable writes.** `NativeS3RecoverableWriter` uses S3 multipart uploads to provide exactly-once semantics for Flink's sink connectors and checkpoint metadata. Uploads are resumable on failure. The writer can recover an in-progress multipart upload and continue from the last committed part. + +**Per-bucket configuration.** A single Flink cluster will be able to access multiple S3 buckets with distinct credentials, regions, endpoints, and encryption policies, configured via `s3.bucket.<bucket-name>.<property>`. This is planned for Flink 2.4. Review Comment: CRT support is on the Flink 2.4 roadmap (see the Availability and Roadmap section). The benchmark numbers here were achieved without CRT; it will push performance further. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
