yandrey321 opened a new pull request, #11684:
URL: https://github.com/apache/ozone/pull/11684
## What changes were proposed in this pull request?
**Problem.** `XceiverClientManager.close()` closes its cached clients one
after another, under the cache lock. A
Ratis client's close waits up to 3 s for each gRPC channel to shut down
gracefully (hard-coded in Ratis
`GrpcUtil.shutdownManagedChannel`). So every datanode that doesn't end the
client's streams in time adds 3 s to the
client's exit. On a 12-datanode cluster, TestDFSIO clients spent up to 54 s
exiting, 3.0 s per stalled channel. All
29 slow exits out of 119 jobs wrote to RATIS/THREE buckets.
**Implementation idea.** During `close()`, the removal listener collects the
unreferenced clients instead of closing
them. They are then closed concurrently outside the lock, so N stalled
clients cost one grace period instead of N.
**Key decisions.**
- **One thread per client.** The pool is short-lived and uses daemon
threads; the cache size
(`scm.container.client.max.size`) bounds the number of clients. A fixed
cap would line stalled closes up again.
- **Threads, not shutdown-then-wait loop.** Ratis `RaftClient.close()`
starts the shutdown and waits in the same call,
and has no asynchronous close, so the waits can only overlap on separate
threads.
- **Only the manager's own close changes.** Normal eviction and closing on
last release stay as they are.
**Side effects.**
- A client still referenced when the manager closes is closed on its last
`releaseClient`, as before.
- The 3 s wait itself remains; removing it needs a Ratis change
(asynchronous or immediate close).
- Why datanodes sometimes don't end the streams in time is not established.
This change doesn't depend on it.
Generated-by: Claude Code (Claude Opus 5.5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15255
## How was this patch tested?
- **New integration test `TestXceiverClientManagerCloseStall`.** A MiniOzone
cluster with 4 RATIS/THREE pipelines.
The clients reach the datanodes through in-test TCP forwarders, which stop
passing traffic before `close()`. The
test asserts that each client hit the graceful-shutdown timeout and that
`close()` took less than 2 × 3 s.
- 4 stalled clients: 12,017 ms before (fails), 3,013 ms after.
- Control case with responsive datanodes: 2 ms.
- **Existing tests:** `TestXceiverClientManagerCloseContention`
(HDDS-15849), `TestXceiverClientGrpc`,
`TestXceiverClientMetrics`, and the full `TestOzoneIntegrationNonHA` suite
(340 tests, including
`TestXceiverClientManager`), all passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
