[
https://issues.apache.org/jira/browse/HADOOP-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118276#comment-18118276
]
ASF GitHub Bot commented on HADOOP-19992:
-----------------------------------------
beryllw opened a new pull request, #8750:
URL: https://github.com/apache/hadoop/pull/8750
Generated-by: OpenAI Codex
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
[HADOOP-19992](https://issues.apache.org/jira/browse/HADOOP-19992)
The OSS SDK does not retry `CompleteMultipartUpload` by default, so
`QpsLimitExceeded` can fail multipart uploads and copies even when
`fs.oss.attempts.maximum` is configured.
Share completion retry logic between uploads and copies, retrying only
`QpsLimitExceeded` with Hadoop's exponential backoff and jitter. Add
`fs.oss.multipart.complete.retry.limit` (default: 3, range: 0–10) and
`fs.oss.multipart.complete.retry.interval` (default: 500ms), with each delay
capped at 10s. Document the settings and extend the existing test class without
adding dependencies.
### How was this patch tested?
- Compiled the changed Java sources and ran `TestAliyunOSSFileSystemStore`
using a standalone JUnit launcher with OSS SDK 3.18.1 and locally cached Hadoop
dependencies: 4 unit tests passed; 3 live OSS tests were skipped because
credentials were unavailable.
- Covered retry success/exhaustion, disabled retries, backoff and delay cap,
configuration validation, non-retryable failures, interruption, and multipart
copy completion/abort behavior.
- `git diff --check` passed. Full Maven validation was not completed because
required build plugins and dependencies were unavailable locally.
- Live OSS integration tests have not been run; no endpoint was used.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)? N/A: no new
dependencies.
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files? N/A: no licensing changes.
### AI Tooling
Contains content generated by OpenAI Codex.
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> AliyunOSS: Retry OSS multipart completion on QpsLimitExceeded
> -------------------------------------------------------------
>
> Key: HADOOP-19992
> URL: https://issues.apache.org/jira/browse/HADOOP-19992
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
> Affects Versions: 3.4.3
> Reporter: JunboWang
> Priority: Minor
>
> In hadoop-aliyun, CompleteMultipartUpload is not retried by the OSS Java
> SDK's default policy. A QpsLimitExceeded response can therefore fail an
> output stream close or multipart copy. Increasing fs.oss.attempts.maximum
> does not resolve this.
> Proposed change:
> - Share completion retry logic between multipart uploads and copies.
> - Retry only QpsLimitExceeded, with exponential backoff and jitter capped at
> 10 seconds. Add fs.oss.multipart.complete.retry.limit (default: 3 retries
> after the initial attempt) and fs.oss.multipart.complete.retry.interval
> (default: 500 ms). Propagate the failure when retries are exhausted.
> - Do not treat NoSuchUpload or an existing object alone as proof of success
> if a completion response is lost.
> - Add regression tests in existing test classes for recovery, retry
> exhaustion, non-retryable errors, and uncertain completion. No new
> dependencies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
