[
https://issues.apache.org/jira/browse/HDDS-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17716:
----------------------------------
Epic Link: HDDS-17864
> Avoid UserGroupInformation.getCurrentUser() per read request in
> XceiverClientManager.getPipelineCacheKey
> --------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17716
> URL: https://issues.apache.org/jira/browse/HDDS-17716
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> *Description*
> Every datanode read acquires an `XceiverClientSpi` through
> `XceiverClientManager.acquireClient`. Callers include:
> - `ChunkInputStream.acquireClient`;
> - `BlockInputStream.acquireClient`;
> - `StreamBlockInputStream.acquireGrpcClientForReadData`, called on
> *{*}every{*}* positioned read;
> - `releaseClient(..., invalidate=true)`.
> `acquireClient` builds the cache key with `getPipelineCacheKey`. In secure
> mode that appends the user:
> {code:java}
> if (isSecurityEnabled()) {
> key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
> }
> {code}
> `UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()`
> → `Subject.getSubject`, which walks the thread stack in the JVM
> (`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler
> stacks, so the walk is long. The key is also built inside
> `synchronized (clientCache)`, a monitor shared by every reader in the
> process, so the walk extends that critical section.
> Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
> | |stream read ON|stream read OFF|
> |—|—|—|
> |`getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU|2.5%|3.4%|
> |share of the EC read path CPU|~26% (18% pread acquire + 8.5%
> initialize)|~25%|
> |wall: `acquireClient`, % busy handler time|0.9%|1.3%|
> `JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of
> all RS CPU). At this load the monitor is not contended; at higher reader
> concurrency the stack walk inside `synchronized (clientCache)` becomes a
> convoy.
> *Proposed fix*
> Resolve the user *{*}once per input stream{*}*, not once per request, and
> pass it to the factory:
> 1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure
> mode only) when the stream is created
> (`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in
> the caller's security context, which is the context that should own its
> datanode clients.
> 2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean
> allowShortCircuit, String user)` (default method delegates to the existing
> one), and the matching release. `getPipelineCacheKey` uses the supplied user
> when non-null.
> 3. Compute the key *{*}before{*}* entering `synchronized (clientCache)`; only
> the cache get/refcount needs the lock.
> Write paths (`BlockOutputStream`) already acquire their client once per
> block, so they need no change.
> Rejected alternatives
> - *{*}Cache the user per `XceiverClientManager`.{*}* Wrong: the key carries
> the user precisely so different users (`doAs`) never share a client.
> - *{*}ThreadLocal of the last user.{*}* Wrong under `doAs` on pooled threads.
> Compatibility / side effects
> - *{*}Semantics change:{*}* the datanode client is chosen by the user that
> *{*}opened{*}* the stream, not the user active at each `read` call. For a
> stream shared across `doAs` contexts this differs. Today such sharing
> silently mixes users too; tokens are per request and checked on the datanode.
> - No wire change. New default method on `XceiverClientFactory`.
> Tests
> - Unit: `TestXceiverClientManager` — the same pipeline with two users gives
> two clients; the per-stream user is honored; the key is computed without
> holding the cache lock (verify with a lock-held assertion hook).
> - Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per
> read (static mock of `UserGroupInformation`).
> - Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`,
> EC read tests).
> - Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB
> reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]