furquan39 commented on issue #66300:
URL: https://github.com/apache/doris/issues/66300#issuecomment-5422541127
**Status:** the committed controlled rerun is done. Mapping this comment
against your "Information needed" list: items 1 and 2 are delivered below; item
3 is confirmed with one deliberate change explained; items 4 and 5 were not
captured in this rerun, stated plainly in their sections with what we can still
offer. Headline results: a repository created via SQL with no connection
property reached the BE with `max_connections=100` (bounded; neither the 50
from `S3Properties` nor the 102400 BE fallback), and a 105-minute induced retry
storm at roughly 6x the incident file count produced no held-FD growth under
two distinct failure signatures. The confirmed defects (unbounded resend,
per-task re-list/re-hash amplification, missing multipart abort) remain the
substance of this issue; a title update is proposed at the end.
Build under test: `doris-4.1.0-rc03-5960d4cea0e` (identical to incident), 3
FE / 3 BE, S3-compatible endpoint (MinIO behind a gateway),
`use_path_style=true`.
## Item 1: effective `max_connections`, measured both ways
Method: `CREATE REPOSITORY` with the same property shape as the
incident-time repository but omitting any connection-limit property, then a
minimal `BACKUP SNAPSHOT` of one small table to force a BE-side S3 client. The
exact statement (values redacted, structure verbatim):
```sql
CREATE REPOSITORY `tmp_conn_probe` WITH S3 ON LOCATION
's3://<bucket>/conn-probe'
PROPERTIES ('s3.endpoint'='http://<endpoint>:9000','s3.access_key'='***',
's3.secret_key'='***','s3.region'='us-east-1','use_path_style'='true');
```
The BE logs the effective client configuration at creation:
```
I20260825 19:08:13.996332 1770 s3_util.cpp:537] create one s3 client with
(ak=<redacted>, token=, endpoint=http://<redacted>:9000, region=us-east-1,
bucket=<redacted>, max_connections=100, request_timeout_ms=10000,
connect_timeout_ms=10000, use_virtual_addressing=false, ...)
```
Cross-check that the property plumbs to the same field: our real backup
repository was dropped and recreated post-incident with
`'s3.connection.maximum' = '64'` (the remediation we mentioned in July), and
every BE that created a client for it logged `max_connections=64` (3 of 3 BEs).
What this settles and what it leaves open:
- With this property shape (S3-compatible endpoint, path-style, no
connection property), FE normalization injected a bounded limit of **100**. The
BE 102400 fallback was not reached via the SQL repository path, consistent with
your normalization reading.
- The measured 100 versus your line-anchored 50 in `S3Properties` is
unresolved. A plausible reconciliation is that an S3-compatible/MinIO-style
endpoint is normalized by a different property class than `S3Properties`, per
your note that other provider classes carry their own bounded defaults; we did
not trace which class handled ours, and would welcome your read. Either way the
practical conclusion holds: the SQL path emits a bounded value, and the
uncapped-default branch of our original report is withdrawn for this path.
## Item 2: held-FD and socket inventories across induced retry cycles
Rerun design:
- Backup source: a purpose-built table of 520M rows, 102,528 rowsets
(approximately 102k segment files assuming one segment per rowset; these were
small single-load rowsets so the approximation is tight), versus ~16k files in
the incident snapshot.
- Failure induction: the backup was triggered with the gateway pod healthy
and reached `UPLOADING` (snapshot phase completed in 7s). Two failure
signatures were then active, interleaved, until cancel: the gateway's backing
volume was full, so requests that reached MinIO were rejected with HTTP 507
(`XMinioStorageFull`), and the gateway pod was additionally degraded into an
OOM crash loop, so requests during its down-windows failed at connect
(`curlCode: 7`). On a representative BE the storm logged 5,333 HTTP-507
failures and 223 connection failures over 105 minutes (17:13:31 to 18:58:42
UTC); per-BE totals were 4,486 / 5,727 / 5,476 failed upload attempts, i.e.
roughly 0.7-0.9 failures/s per BE aggregated across concurrent tasks, with
individual tasks retrying on a ~25s cycle (consistent with the 10s report
interval plus the 10s connect timeout).
- One creation-time observation worth recording: a repository whose backend
refuses connections is rejected at job creation (repository stat fails, no job
starts), but a reachable backend with a **full** volume passes creation-time
validation (stat/list succeeds) and the job then enters the retry storm at the
first `put`. So "endpoint down" cannot start a storm, but "endpoint full" can.
- Observation harness: 30 samples at 3-minute cadence, per BE:
`/proc/<be_pid>/fd` counts grouped by type, socket summary states, node-wide
`file-nr`, and the BE's own `doris_be_process_fd_num_used` metric.
- Environment note on limits: the BEs now run under a deliberate
self-applied 200k `nofile` cap (post-incident hardening via a wrapper around
the identical Doris build); the incident ran at 1,048,576. The cap was never
approached.
Results over the whole window, recomputed directly from the raw log:
- Per-BE `/proc` FD totals stayed flat: be-0 4.06-4.24k, be-1 3.77-3.92k,
be-2 4.26-4.43k; each BE moved within ~5% over the entire storm, with no upward
trend. The BE's own `doris_be_process_fd_num_used` metric read 3.0-3.7k (it
tracks ~750 below the `/proc` count by construction). Peak anywhere: 4,427 FDs
= 2.2% of the 200k cap.
- By type: regular files 3.1-3.7k, anon_inode constant 323, pipe constant
261, socket-type FDs 126-202.
- Socket behavior: ESTABLISHED fluctuated between 89 and 165 in phases
(higher while the 507-answering endpoint was reachable, lower during
connect-failure windows) and ended at baseline levels; TIME_WAIT stayed at
12-13 with brief spikes that cleared within one sample. The fluctuation is
downward-releasing churn: sockets were opened and released each cycle, never
accumulated. On CLOSE_WAIT specifically: our summary capture did not bucket it
separately, but a CLOSE_WAIT accumulation necessarily holds process FDs, and
the flat 126-202 socket-FD band bounds any such accumulation to noise.
- Node-wide `file-nr` stayed in the 13.3k-20.7k band against an effectively
unlimited node ceiling.
- Baselines, with provenance: at storm start the BEs already held their
steady totals (sample 1: 4.24k/3.9k/4.43k), and the window never rose above
that, so the storm added approximately nothing to held FDs. A post-cleanup idle
measurement (after the test table was dropped) read 2.1-2.7k. Note this differs
from the 34,250 idle baseline we posted in July: that figure was dominated by
33.4k open tablet/segment data files, and the dominant table's layout was
rebuilt between the incident and this rerun as part of remediation (fewer,
larger tablets), so resident data-file FDs dropped to ~2-3k. The retry-cycle
behavior measured here is independent of that resident baseline, but the
absolute levels are not comparable to the July inventory.
- `CANCEL BACKUP` stopped all retry activity at once: the last failure line
on each BE is 18:58:36-18:58:42 UTC, and a re-check 87 minutes later (20:25
UTC) still shows zero lines after that timestamp.
<details>
<summary>Representative BE failure lines (first, 507-mode, connect-mode,
last)</summary>
```
W20260825 17:13:31.852550 task_worker_pool.cpp:1299] failed to
upload|signature=1786821864774|job_id=1786821858797|
error=[INTERNAL_ERROR]failed to put object:
s3://<bucket>/snapshots/__palo_repository_.../..._0.dat.<md5>:
XMinioStorageFull ... Storage backend has reached its minimum free drive
threshold. code=507 type=100
W20260825 17:16:04.875680 task_worker_pool.cpp:1299] failed to
upload|signature=...|job_id=1786821858797|
error=[INTERNAL_ERROR]failed to list snapshots/__palo_repository_.../:
curlCode: 7, Couldn't connect to server code=-1 type=99
W20260825 18:58:42.079645 task_worker_pool.cpp:1299] failed to
upload|signature=...|job_id=1786821858797|
error=[INTERNAL_ERROR]failed to list snapshots/__palo_repository_.../:
curlCode: 7, Couldn't connect to server code=-1 type=99
```
</details>
## Items 3-5, explicitly
- **Item 3 (build, limits, configs):** build identical to the July
disclosure (`doris-4.1.0-rc03-5960d4cea0e`); `upload_worker_count`, the S3
upload thread-pool sizes, `s3_write_buffer_size`, and the 10s report interval
are unchanged stock values as posted in July; the only deliberate change is the
200k self-applied `nofile` cap described above.
- **Item 4 (resend counter and writer-metric time series):** not captured by
this rerun's harness; the failure-log line counts above are the closest
observed proxy for the resend series. If useful, we can re-run a shorter window
scraping `agent_task_resend_total{task="UPLOAD"}` and the `s3_file_writer_*` /
upload-pool series.
- **Item 5 (incomplete-multipart counts before/after):** not captured, and
the object store was subsequently cleaned during recovery, so it cannot be
reconstructed for this run. The multipart-abort defect claim therefore
continues to rest solely on the July read-only listing of 10 incident-day
orphaned upload IDs. We can capture a proper before/after listing in a short
dedicated run if you want it on record.
## Updated causal picture
1. **Held-FD leak: not reproduced.** Across ~15k failed upload cycles in two
failure signatures (server-error responses and connection failures), held FDs
never grew. We consider a monotonic FD leak ruled out for these two signatures
on this build. Your hypothesized stuck-request path (endpoint accepts and then
stalls without responding) was NOT exercised by either signature and remains
untested; we can attempt a variant with an artificially stalling endpoint if
that would help.
2. **The amplification defect stands as the real cost, per your code
analysis.** What this rerun adds is the observed unbounded cadence (per-task
~25s, ~0.8 failures/s per BE in aggregate, for as long as the job lives). The
per-cycle re-list/re-hash amplification itself is confirmed from code (bounded
per task batch, worst when the failure lands late in a batch); with most of our
cycles failing at the remote list or first put, this particular storm exercised
the resend loop more than the re-hash tax.
3. **Node-death mechanism:** we withdraw the FD-exhaustion framing from our
original report. The incident-day node failures are under separate
infrastructure investigation on our side (a kernel-level fault is suspected but
unconfirmed); we are not asserting a causal chain here.
4. **Multipart abort defect:** unchanged status, evidenced by the July
listing as stated under item 5.
## Proposed title update
The current title overstates both the FD mechanism and the default.
Suggested replacement:
> [Bug] BACKUP upload failure handling resends failed tasks every report
interval for up to 24h with no backoff or cap, re-listing and re-hashing the
failing task's snapshot directories each cycle, and multipart uploads are never
aborted on BE
We are happy to edit the title, and remain glad to test
bounded-retry/backoff and BE-side multipart-abort patches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
