furquan39 opened a new issue, #66300: URL: https://github.com/apache/doris/issues/66300
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache Doris 4.1.0-rc03. All file:line references below are at tag `4.1.0-rc03`. ### Environment We operate Doris 4.1.0-rc03 on Kubernetes with Ceph-backed storage. Related: "[Enhancement] Compaction scheduler retries permanently-failing tablets every 5s at full I/O cost with no backoff or failure counter" (#66299). The version-bloated table mentioned below is a byproduct of that issue's only available mitigation (disabling auto compaction drives tablet version counts up); the two failure mechanisms are otherwise independent. ### What's Wrong? When BACKUP upload tasks fail, three independent defaults compound into unbounded file descriptor growth on the BE: 1. **The FE resends failed upload tasks roughly every 10 seconds until the job timeout, with no backoff and no retry cap.** On task failure the job is not failed; the comment in `fe/fe-core/src/main/java/org/apache/doris/backup/BackupJob.java:510-511` says "just retry it until timeout", and the task is re-dispatched from `BackupJob.java:457`. The default job timeout is `backup_job_default_timeout_ms = 86400000` (24 hours), so a persistently failing upload is retried on a fixed ~10s cadence for up to a day. 2. **Every resend repeats the full local I/O of the batch.** Each retried upload task re-lists the snapshot directory and recomputes md5 for every file in the batch (`be/src/runtime/snapshot_loader.cpp:826-880`). With `backup_upload_snapshot_batch_size = 10` (`fe/fe-core/src/main/java/org/apache/doris/common/Config.java:3030`) and a version-bloated table whose snapshot held roughly 16k files in our case, each 10s cycle re-opens and re-reads a large file set, on top of opening fresh S3 connections. 3. **The S3 client connection pool is effectively unbounded by default, and failed multipart uploads are never aborted on the BE.** When the repository properties do not set a connection limit, the S3 client `maxConnections` defaults to 102400 (`be/src/util/s3_util.cpp:509-513`), far above any realistic process fd ulimit. On upload failure the BE deliberately skips AbortMultipartUpload (`be/src/io/fs/s3_file_writer.cpp:76-93`, comment "We won't do S3 abort operation in BE"), so failed attempts also leave incomplete multipart state behind on the object store. Individually each of these is defensible. Together they mean: one bad repository endpoint or one transient credential/network problem turns a BACKUP into a 24-hour loop that re-reads and re-hashes thousands of files every 10 seconds while opening S3 connections against a 102400-connection ceiling, with nothing cleaning up after failed attempts. Descriptor usage on the BE climbs monotonically until the process or the node runs out. ### What You Expected? Failed upload tasks back off, and the backup job fails after a bounded number of retries instead of retrying every ~10s for 24 hours. File descriptor usage on the BE returns to baseline between retry cycles. Failed multipart uploads are aborted so their connections and file descriptors are released, rather than accumulating across cycles. ### How to Reproduce? From our dev reproduction: 1. Create an S3 repository without setting any connection-pool property, so the `maxConnections = 102400` default from `s3_util.cpp:509-513` applies. 2. Pick a table whose snapshot contains a large number of files (high version or tablet count; ours was ~16k files from a version-bloated table). 3. Start a BACKUP to that repository and induce upload failures mid-transfer, for example by revoking the credentials or pointing the repository at an endpoint that accepts connections but rejects multipart part uploads. 4. Observe in the FE logs that the failed UPLOAD tasks are re-dispatched about every 10 seconds and the job stays in UPLOADING; it will not fail on its own before `backup_job_default_timeout_ms` (24h) elapses. 5. On the BE, watch open file descriptors for the process (for example `ls /proc/<be_pid>/fd | wc -l`) across retry cycles. Each cycle re-lists and re-md5s the batch files (`snapshot_loader.cpp:826-880`) and opens new S3 connections; sockets belonging to failed multipart uploads are not released by an abort, and the count trends upward instead of returning to baseline between cycles. ### Anything Else? **Impact:** We attribute two production node outages within one week to this mechanism, based on per-process file-open telemetry that showed the BE backup upload path as the source of the descriptor growth. Exhausting file descriptors did not stop at the backup: the affected nodes went down and ingest into Doris stalled until the nodes were recovered. A failed backup, which should be an annoyance, became a node-level availability incident. **Status on master:** The reproduction is on 4.1.0-rc03, and we verified on current master as of 2026-07-30 that `be/src/io/fs/s3_file_writer.cpp` still deliberately skips the multipart abort on upload failure, so the relevant code is unchanged. **Related work:** #56628 added multipart-upload abort in the recycler, a precedent for cleaning up incomplete multipart state; #44560 introduced batch-size adaptation in the backup path; #30120 documents a prior fd-exhaustion cascade. None of them addresses the FE retry policy, the `maxConnections` default, or the BE-side abort skip reported here. **Suggested direction:** We would be happy to help test or contribute, and we may well be missing design constraints here, but from the operator side four changes would have contained this: 1. A sane default for the S3 client `maxConnections` when the repository does not set one (on the order of hundreds, not 102400), keeping the current value reachable via explicit repository configuration for users who need it. 2. Exponential backoff and a bounded retry count for failed upload tasks in `BackupJob`, failing the job after N consecutive failures of the same task instead of resending on a fixed ~10s cadence until the 24h timeout. 3. BE-side AbortMultipartUpload on upload failure in `S3FileWriter`, so failed attempts do not accumulate incomplete multipart state and held connections. 4. A metric for backup upload task retries, so operators can alert on a retry storm long before descriptor exhaustion takes the node down. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
