JunboWang created HADOOP-19992:
----------------------------------
Summary: AliyunOSS: Retry OSS multipart completion on
QpsLimitExceeded
Key: HADOOP-19992
URL: https://issues.apache.org/jira/browse/HADOOP-19992
Project: Hadoop Common
Issue Type: Improvement
Components: fs/oss
Affects Versions: 3.4.3
Reporter: JunboWang
In hadoop-aliyun, CompleteMultipartUpload is not retried by the OSS Java SDK's
default policy. A QpsLimitExceeded response can therefore fail an output stream
close or multipart copy. Increasing fs.oss.attempts.maximum does not resolve
this.
Proposed change:
- Share completion retry logic between multipart uploads and copies.
- Retry only QpsLimitExceeded, with exponential backoff and jitter capped at 10
seconds. Add fs.oss.multipart.complete.retry.limit (default: 3 retries after
the initial attempt) and fs.oss.multipart.complete.retry.interval (default: 500
ms). Propagate the failure when retries are exhausted.
- Do not treat NoSuchUpload or an existing object alone as proof of success if
a completion response is lost.
- Add regression tests in existing test classes for recovery, retry exhaustion,
non-retryable errors, and uncertain completion. No new dependencies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
