[
https://issues.apache.org/jira/browse/FLINK-41427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samrat Deb updated FLINK-41427:
-------------------------------
Affects Version/s: 2.3.0
> s3.upload.max.concurrent.uploads has no effect - multipart part uploads are
> synchronous
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-41427
> URL: https://issues.apache.org/jira/browse/FLINK-41427
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / FileSystem
> Affects Versions: 2.3.0
> Reporter: Samrat Deb
> Assignee: Samrat Deb
> Priority: Major
> Labels: pull-request-available
>
> The config option s3.upload.max.concurrent.uploads is documented as "Maximum
> concurrent part uploads per stream" (README.md, option default is the CPU
> count). It is accepted and validated (> 0), but nothing in the recoverable
> writer reads it, so changing it has no effect.
> The value goes NativeS3FileSystemFactory → NativeS3FileSystem →
> NativeS3RecoverableWriter and stops there. NativeS3RecoverableWriter#open and
> #recover don't pass it to NativeS3RecoverableFsDataOutputStream, whose
> constructors have no such parameter.
> Part uploads are fully synchronous:
> - NativeS3RecoverableFsDataOutputStream#write calls uploadCurrentPart()
> inline once currentPartSize >= minPartSize.
> - That calls NativeS3ObjectOperations#uploadPart, which uses the blocking
> S3Client#uploadPart(request, RequestBody.fromFile(file)).
> - The caller (the Flink task thread) can't accept more bytes until S3
> responds.
> Related: s3.async.enabled (S3TransferManager) is only consulted in
> NativeS3ObjectOperations#putObject. In the recoverable writer that is used
> only for the small .incomplete/ tail object written at checkpoint. Full parts
> always use the plain sync client, so enabling async doesn't speed up part
> uploads.
> #
> ## Impact
> - Users who raise the setting to improve sink throughput get no change.
> - Each part upload (5 MiB by default) stalls record processing for the S3
> round-trip, so sink throughput is bounded by S3 latency.
> #
> ## How to verify
> 1. grep -rn maxConcurrentUploadsPerStream
> flink-filesystems/flink-s3-fs-native/src/main. It is only declared, assigned
> and forwarded, never read.
> 2. Run a recoverable write (for example FileSink with S3) with
> s3.upload.max.concurrent.uploads set to 1 and then to 64. Part uploads are
> still serialized with the same timing.
> 3. The only test touching the option is
> NativeS3FileSystemFactoryTest#testInvalidMaxConcurrentUploadsThrowsException,
> which checks the > 0 validation.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
