matanper opened a new issue, #69862: URL: https://github.com/apache/doris/issues/69862
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4. master is the same: `be/src/io/fs/s3_file_reader.cpp:166-190` retries a GetObject that failed with HTTP 429 up to `max_s3_client_retry` times, and `be/src/util/s3_util.cpp:429` configures the SDK client with `S3CustomRetryStrategy(config::max_s3_client_retry)`, whose `ShouldRetry` (`common/cpp/obj_retry_strategy.cpp:54`) already retries every `IsRetryableHttpResponseCode` response, 429 included. ### Description A throttled read is retried at two levels with the same budget: - the AWS SDK retries 429 (and 5xx / SlowDown) up to `max_s3_client_retry` times with its own backoff; - when the SDK finally returns the 429, `S3FileReader::read_at_impl` loops again up to `max_s3_client_retry` times, each iteration going through the SDK's retries again. With the default `max_s3_client_retry = 10`, one read can issue up to (1 + 10) × (1 + 10) = 121 GetObject requests. Under throttling this multiplies the request rate against an endpoint that is asking clients to back off, which prolongs the throttling for every other reader, and a single query read can take minutes before it fails. ### Solution Leave 429 to the SDK's retry strategy (it already retries it), and keep the reader-level retry only for failures the SDK does not retry, such as a response with a success status whose body could not be read completely. Keep the `too_many_request` bvar/profile counters for the final 429. ### How to Reproduce? Point a BE at an S3-compatible endpoint (or a proxy) that answers every GetObject with 429 and count requests per read: each read sends about 121 requests before failing. ### Anything Else? Related but separate: transient body-transfer failures ("Failed to flush response stream") were discussed in #61636. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
