furquan39 commented on issue #66300:
URL: https://github.com/apache/doris/issues/66300#issuecomment-5130415210
Answers to the requested items below.
## Corrections to our report
1. **Wrong redispatch anchors.** Our report cited `BackupJob.java:509-511`
("just retry it until timeout") and `BackupJob.java:457` as the retry
mechanism. As you point out, the first is the later `UPLOAD_INFO` metadata
stage and the second is the overall timeout check. We accept the
`ReportHandler.taskReport()` + `shouldResend()` analysis (failed task left in
`AgentTaskQueue`, BE report diff every ~10 s, `UploadTask.createTime = -1`
making `shouldResend()` immediately true) as the correct description of the
resend path.
2. **We cannot currently prove the effective `max_connections` was 102400.**
The repository was created with no connection-limit property at all (verbatim,
redacted, in item 1 below), so whether the FE normalization layer injected the
bounded default of 50 toward the BE, or the BE hit the 102400 fallback, is
exactly the open question. The incident-time BE logs that would carry the
decisive `create one s3 client with ... max_connections=...` line did not
survive the post-incident pod recreation, and backups have been frozen since,
so no S3 client has been created in the retained logs. We will capture that
line in a controlled rerun (see Gaps).
3. **We overstated the held-FD evidence.** Two precision problems in our
report:
- Our FD sampler scraped a BE on a node that survived the incident. That
BE's process held-FD gauge stayed flat (332 to 422) through the incident
minute. The BE actually executing the upload tasks ran on the node that died
and was never sampled; no held-FD series from it exists.
- The ~3.3M figure in our report is a per-process file-open operation
count from node telemetry, i.e. an open-churn rate counter, not a held-FD
gauge. It supports "the BE was by far the dominant file-open source on the
dying node at the moment of death, during the backup upload phase"; it does not
by itself demonstrate monotonic held-FD growth.
The tight correlation stands: backup started 13:58:50 UTC, the hosting
node's kubelet was dead 42 seconds later at 13:59:32 UTC, and our recorder
captured the backup gateway's restart counter bumping at 13:59:16 UTC inside
that window. But the kill mechanism (held-FD exhaustion versus I/O saturation)
is not yet distinguished by our data, which matches your assessment.
## 1. CREATE REPOSITORY properties and the `create one s3 client` log line
`SHOW CREATE REPOSITORY` for the repository in use at incident time (created
2026-06-25, never recreated; our tooling only creates it if absent, so this
object served every incident-time backup):
```
CreateStmt: CREATE REPOSITORY doris_s3_backup
WITH S3
ON LOCATION "s3://<redacted-bucket>/snapshots"
PROPERTIES
("s3.access_key" = "*XXX",
"s3.region" = "us-east-1",
"use_path_style" = "true",
"s3.endpoint" = "http://<redacted-endpoint>:9000",
"s3.secret_key" = "*XXX"
)
```
No `s3.connection.maximum` and no other connection-limit property. The
creation script in our deployment history confirms the same property set at
incident time; a change adding `'s3.connection.maximum' = '64'` was only
authored the day after the incident and is not deployed (and would not alter
this pre-existing repository object anyway).
The BE log line: `grep -ih 'max_connection'` and `grep -ih 's3 client'`
across all retained `be.INFO` files on all three BEs return **zero matches**.
All three BEs were restarted ~23:12 UTC on incident day (2026-07-28), retained
logs start there, and backups have been frozen since, so the incident-time
client-creation line is gone. What is established from retained state: **no
user-specified cap existed at incident time**; the effective value (50 from FE
normalization versus the 102400 BE fallback) will be settled by the rerun log
line.
## 2. FD inventories by type
Baseline held-FD inventory of one BE, no backup running (2026-07-30 11:25
UTC, from `/proc/<pid>/fd`):
```
fd count: 34250 held FDs total
33397 regular files under /opt/apache-doris/be/storage/ (tablet/segment
data)
323 anon_inode (eventfd/epoll etc.)
261 pipe
100 socket
5 log files under /opt/apache-doris/be/log/
3 /dev/*
~20 binaries/JARs/JVM libs
```
Socket states from `/proc/<pid>/net/tcp{,6}` in the pod's namespace:
```
85 ESTABLISHED
21 TIME_WAIT
5 LISTEN
2 CLOSE_WAIT
```
So the no-backup baseline is ~34k held FDs, about 97.5% regular storage
files and only 100 sockets, against a 1,048,576 `ulimit -n`. The incident-time
per-type inventory on the uploading BE cannot be produced retroactively (that
BE was on the node that died and was not sampled); the controlled rerun will
sample `/proc/<pid>/fd` type and socket-state inventories on the uploading BE
at baseline and across several retry cycles, including remote endpoint/port via
`ss` (present in the container).
## 3. Build, limits, and backup-relevant configs
```
Build: doris-4.1.0-rc03-5960d4cea0e (all FEs and BEs
identical)
ulimit -n (BE): 1048576 (soft and hard, from /proc/<pid>/limits)
/api/show_config:
[["upload_worker_count","int32_t","1","false"]]
[["num_s3_file_upload_thread_pool_min_thread","int64_t","16","false"]]
[["num_s3_file_upload_thread_pool_max_thread","int64_t","64","false"]]
[["s3_write_buffer_size","int64_t","5242880","true"]]
[["report_task_interval_seconds","int32_t","10","true"]]
```
All stock defaults; the task-report interval driving the resend cadence is
the default 10 s. Connection/request timeouts were not overridden anywhere in
our configuration.
## 4. FE resend counter and BE upload metrics
Current master FE (note the limitation: the incident-time master FE failed
over at 14:13:05 UTC on incident day, 14 minutes after the node death, and its
counters died with it; the values below cover post-failover time only, which
includes the tail of the incident-day backup before we froze backups):
```
# TYPE doris_fe_agent_task_resend_total counter
doris_fe_agent_task_resend_total{task="PUBLISH_VERSION"} 3
doris_fe_agent_task_resend_total{task="UPLOAD"} 1
doris_fe_agent_task_resend_total{task="CLONE"} 21
doris_fe_agent_task_total{task="MAKE_SNAPSHOT"} 8
doris_fe_agent_task_total{task="RELEASE_SNAPSHOT"} 9
doris_fe_agent_task_total{task="UPLOAD"} 5
```
Non-master FEs report 0. The incident-time resend delta is therefore lost;
the rerun will record `agent_task_resend_total{task="UPLOAD"}` as a time series
alongside the retry cycles. The requested BE-side series
(`s3_file_writer_file_being_written`, `s3_file_buffer_allocated`,
`S3FileUploadThreadPool` running/queued) are likewise not in our archive for
the uploading BE and will be captured in the rerun.
## 5. Incomplete multipart uploads
New direct evidence for the confirmed cleanup defect. The backup gateway is
a single-node MinIO deployment; read-only inspection of its multipart state
area two days after the incident (all backups frozen in between):
```
$ ls /data/.minio.sys/multipart/
17 multipart hash directories total, 0 bytes of retained part data
10 directories each contain exactly one upload-id subdirectory (an incomplete
multipart upload), all with mtime Jul 28 (incident day), e.g.:
09b46d0f.../b3a08dd1-e57b-4200-a0e1-c5d748c94b73
2844799d.../fd3e6e92-deaf-4dce-8809-64e49971e826
5574ae6f.../26c8aade-05f0-4fcd-8183-67ef98c306cc
... (7 more)
7 directories are empty, mtimes Jul 29 (post-freeze housekeeping)
```
So 10 orphaned multipart upload IDs from incident day remain on the object
store two days later, with no lifecycle rule cleaning them up. This directly
supports the `S3FileWriter` no-abort defect independently of the FD question.
(A credentialed `mc ls --incomplete` before/after pair will be part of the
rerun; this pass was strictly read-only.)
## On the recommended directions
Your implementation directions (bounded upload retry state with backoff
preserving FE crash/replay semantics, best-effort BE-side multipart abort
outside the committer path, checkpointing to bound retry I/O, reviewing the
102400 fallback separately, and a backup-specific retry counter) all look sound
to us, and we would be glad to test them. We have already adopted the
operational containments you list: an explicit `s3.connection.maximum` within
the BE FD budget is prepared for the recreated repository, persistently failing
backups are cancelled promptly rather than left to the 24 h timeout, and we
alert on `agent_task_resend_total{task="UPLOAD"}`.
## Gaps and follow-ups
Only a controlled rerun can produce the following, and we commit to
providing it; a hardened re-run environment is being prepared:
1. The `create one s3 client with ... max_connections=...` line for this
repository, captured both without and with an explicit `s3.connection.maximum`,
settling the effective value each way.
2. Held-FD type and socket-state inventories (`/proc/<pid>/fd` plus `ss`) on
the BE actually executing the upload, at baseline and across several induced
retry cycles.
3. FE `agent_task_resend_total{task="UPLOAD"}` deltas and BE upload failure
logs (error names, request IDs) matched to those cycles, plus the
`s3_file_writer_*` and upload thread pool metric series.
4. Credentialed incomplete-multipart counts before and after the
reproduction.
Not recoverable from our side: the dead nodes' kernel journals from the
incident are held by the infrastructure team and are not in this pack; if they
surface the kernel-side error at death (file-table exhaustion versus something
else) we will add them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
