klsince opened a new pull request, #10263: URL: https://github.com/apache/pinot/pull/10263
Add support to upload S3 object in multi parts but sequentially in this PR, mainly to overcome the 5GB limit of single PutObject call. This can be extended to upload parts in parallel if needed. ## Release Note ## One major config to init S3PinotFS 1. minObjectSizeForMultiPartUpload: -1 by default to disable multi part upload. One can set it to something like 2G to upload large object by multi parts; 2. multiPartUploadPartSize: 128MB by default, no less than 5MB according to S3 doc. 3. multiPartUploadMaxPartNumAllowed: must be 10000 (default) or less according to S3 doc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
