phofl opened a new issue, #40420: URL: https://github.com/apache/arrow/issues/40420
### Describe the enhancement requested Currently, the size of every part in a multipart upload is hard-coded here: https://github.com/apache/arrow/blob/d2970e1d047f1bd31c31995c35450a7e5bfce3c0/cpp/src/arrow/filesystem/s3fs.cc#L1394-L1400 We've run into issues where the request rate to s3 is too high when uploading form a bigger cluster and S3 errors. ``` OSError("When completing multiple part upload for key '***' in bucket '***': AWS Error SLOW_DOWN during CompleteMultipartUpload operation: Please reduce your request rate.") ``` We've tried alleviating this with different s3 bucket prefixes, but this didn't solve the problem completely. Would you be open to expose an option that makes the part size configurable so that we can configure the chunk size? ### Component(s) C++, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@arrow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
