ReemaAlzaid opened a new issue, #12303: URL: https://github.com/apache/gluten/issues/12303
### Description ## Description Support multi threaded asynchronous data upload to object storage. Currently, object storage uploads can become a bottleneck when writing large files because multipart upload parts are uploaded synchronously. This limits throughput, especially for remote object stores where each part upload has network latency. This feature proposes adding support for asynchronous multipart uploads so multiple parts can be uploaded concurrently. Ref: https://github.com/apache/gluten/issues/12022, https://github.com/facebookincubator/velox/pull/14472 ### Configuration ```text spark.gluten.velox.s3UploadPartAsync spark.gluten.velox.s3MaxConcurrentUploadNum spark.gluten.velox.s3UploadThreads ``` These would map to Velox S3 configs such as: ```text hive.s3.part-upload-async hive.s3.max-concurrent-upload-num hive.s3.upload-threads ``` ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
