wcy-fdu commented on issue #5101: URL: https://github.com/apache/opendal/issues/5101#issuecomment-2357831862
Update: OpenDAL and the AWS SDK handle multipart uploads differently. - The AWS SDK operates completely in parallel - while OpenDAL processes uploads based on concurrency, utilizing an internal task queue. During the final close, it continuously polls this task queue until all tasks are completed. As a result, in extreme cases, the timeout in OpenDAL may require waiting through two rounds of I/O: the first round waits for `task.execute`, and the second waits for all tasks in the batch to finish. Currently, the timeout is not per operation; it also includes the waiting time for tasks. Therefore, when the number of tasks exceeds the concurrency, the timeout is more likely to be triggered compared to the SDK. Additionally, there may be a MinIO issue: the `CompleteMultipartUpload` operation is not necessarily idempotent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
