[ 
https://issues.apache.org/jira/browse/HDDS-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17716:
----------------------------------
    Epic Link: HDDS-17864

> Avoid UserGroupInformation.getCurrentUser() per read request in 
> XceiverClientManager.getPipelineCacheKey
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17716
>                 URL: https://issues.apache.org/jira/browse/HDDS-17716
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> *Description*
> Every datanode read acquires an `XceiverClientSpi` through 
> `XceiverClientManager.acquireClient`. Callers include:
>  - `ChunkInputStream.acquireClient`;
>  - `BlockInputStream.acquireClient`;
>  - `StreamBlockInputStream.acquireGrpcClientForReadData`, called on 
> *{*}every{*}* positioned read;
>  - `releaseClient(..., invalidate=true)`.
> `acquireClient` builds the cache key with `getPipelineCacheKey`. In secure 
> mode that appends the user:
> {code:java}
> if (isSecurityEnabled()) {
> key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
> }
> {code}
> `UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()` 
> → `Subject.getSubject`, which walks the thread stack in the JVM
> (`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler 
> stacks, so the walk is long. The key is also built inside
> `synchronized (clientCache)`, a monitor shared by every reader in the 
> process, so the walk extends that critical section.
> Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
> | |stream read ON|stream read OFF|
> |—|—|—|
> |`getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU|2.5%|3.4%|
> |share of the EC read path CPU|~26% (18% pread acquire + 8.5% 
> initialize)|~25%|
> |wall: `acquireClient`, % busy handler time|0.9%|1.3%|
> `JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of 
> all RS CPU). At this load the monitor is not contended; at higher reader 
> concurrency the stack walk inside `synchronized (clientCache)` becomes a 
> convoy.
> *Proposed fix*
> Resolve the user *{*}once per input stream{*}*, not once per request, and 
> pass it to the factory:
> 1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure 
> mode only) when the stream is created
> (`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in 
> the caller's security context, which is the context that should own its 
> datanode clients.
> 2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean 
> allowShortCircuit, String user)` (default method delegates to the existing 
> one), and the matching release. `getPipelineCacheKey` uses the supplied user 
> when non-null.
> 3. Compute the key *{*}before{*}* entering `synchronized (clientCache)`; only 
> the cache get/refcount needs the lock.
> Write paths (`BlockOutputStream`) already acquire their client once per 
> block, so they need no change.
> Rejected alternatives
>  - *{*}Cache the user per `XceiverClientManager`.{*}* Wrong: the key carries 
> the user precisely so different users (`doAs`) never share a client.
>  - *{*}ThreadLocal of the last user.{*}* Wrong under `doAs` on pooled threads.
> Compatibility / side effects
>  - *{*}Semantics change:{*}* the datanode client is chosen by the user that 
> *{*}opened{*}* the stream, not the user active at each `read` call. For a 
> stream shared across `doAs` contexts this differs. Today such sharing 
> silently mixes users too; tokens are per request and checked on the datanode.
>  - No wire change. New default method on `XceiverClientFactory`.
> Tests
>  - Unit: `TestXceiverClientManager` — the same pipeline with two users gives 
> two clients; the per-stream user is honored; the key is computed without 
> holding the cache lock (verify with a lock-held assertion hook).
>  - Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per 
> read (static mock of `UserGroupInformation`).
>  - Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`, 
> EC read tests).
>  - Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB 
> reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to