Hi Flink community, Thanks for publishing the Native S3 vs. Presto S3 benchmark<https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620396/Benchmarking+Native+S3+FileSystem+flink-s3-fs-native+vs+Presto+S3+flink-s3-fs-presto>. I have been tracing the checkpoint write path to better understand the reported approximately 2x improvement.
I have a question about Section 10.2, “Architecture Differences,” which describes Native S3 uploads as using S3TransferManager with parallel, non-blocking I/O. This does not appear to match the RocksDB checkpoint path in the code I inspected. 1. RocksDB checkpoint data appears to use synchronous PutObject The benchmark uses Flink 2.1.1 with full RocksDB checkpoints. `RocksNativeFullSnapshotStrategy` uploads the snapshot files through `RocksDBStateUploader`. The subsequent path is: RocksNativeFullSnapshotStrategy → RocksDBStateUploader → FsCheckpointStateOutputStream.createStream() → EntropyInjector.createEntropyAware() → FileSystem.create() → NativeS3FileSystem.create() → new NativeS3OutputStream(clientProvider.getS3Client(), ...) → S3Client.putObject(..., RequestBody.fromFile(tmpFile)) The AWS SDK V2 implementation does not automatically turn this call into multipart upload. Files can still be uploaded concurrently by Flink’s uploader threads, but that is different from parallel multipart uploads through S3TransferManager. 2. RecoverableWriter’s use of S3TransferManager does not extend to RocksDB SST checkpoint uploads There is a TransferManager-backed upload path in the Native recoverable-writer implementation ―― NativeS3RecoverableWriter. Within this checkpoint workflow, the top-level `_metadata` file can use `RecoverableWriter`, whereas RocksDB SST files use the ordinary checkpoint output stream described above. Consequently, I do not see how the existence of this TransferManager-backed helper explains the SST upload performance in this benchmark. Could you help clarify: - Which exact commit and additional patches were used to build the benchmark artifact? - Did that build have a different checkpoint upload path that used S3TransferManager? - If the synchronous PutObject path was used, should Section 10.2 be revised, and is there profiling or request-level evidence identifying the main contributors to the improvement? My question is about the architectural explanation and performance attribution, rather than disputing the reported measurements. Please let me know if I have missed a relevant execution path. Thanks, RuiXia
