Samrat Deb created FLINK-41427:
----------------------------------
Summary: s3.upload.max.concurrent.uploads has no effect -
multipart part uploads are synchronous
Key: FLINK-41427
URL: https://issues.apache.org/jira/browse/FLINK-41427
Project: Flink
Issue Type: Sub-task
Components: Connectors / FileSystem
Reporter: Samrat Deb
The config option s3.upload.max.concurrent.uploads is documented as "Maximum
concurrent part uploads per stream" (README.md, option default is the CPU
count). It is accepted and validated (> 0), but nothing in the recoverable
writer reads it, so changing it has no effect.
The value goes NativeS3FileSystemFactory → NativeS3FileSystem →
NativeS3RecoverableWriter and stops there. NativeS3RecoverableWriter#open and
#recover don't pass it to NativeS3RecoverableFsDataOutputStream, whose
constructors have no such parameter.
### Part uploads are fully synchronous:
- NativeS3RecoverableFsDataOutputStream#write calls uploadCurrentPart() inline
once currentPartSize >= minPartSize.
- That calls NativeS3ObjectOperations#uploadPart, which uses the blocking
S3Client#uploadPart(request, RequestBody.fromFile(file)).
- The caller (the Flink task thread) can't accept more bytes until S3 responds.
Related: s3.async.enabled (S3TransferManager) is only consulted in
NativeS3ObjectOperations#putObject. In the recoverable writer that is used only
for the small .incomplete/ tail object written at checkpoint. Full parts always
use the plain sync client, so enabling async doesn't speed up part uploads.
## Impact
- Users who raise the setting to improve sink throughput get no change.
- Each part upload (5 MiB by default) stalls record processing for the S3
round-trip, so sink throughput is bounded by S3 latency.
## How to verify
1. grep -rn maxConcurrentUploadsPerStream
flink-filesystems/flink-s3-fs-native/src/main. It is only declared, assigned
and forwarded, never read.
2. Run a recoverable write (for example FileSink with S3) with
s3.upload.max.concurrent.uploads set to 1 and then to 64. Part uploads are
still serialized with the same timing.
3. The only test touching the option is
NativeS3FileSystemFactoryTest#testInvalidMaxConcurrentUploadsThrowsException,
which checks the > 0 validation.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
