Andrey Yarovoy created HDDS-17868:
-------------------------------------
Summary: XceiverClientShortCircuit logs every create, start, close
and idle timeout at INFO, with a stack trace on idle timeout
Key: HDDS-17868
URL: https://issues.apache.org/jira/browse/HDDS-17868
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
Assignee: Andrey Yarovoy
h3. Problem
{\{XceiverClientShortCircuit}} logs its normal lifecycle at INFO:
{noformat}
:129 LOG.info("{} is created for pipeline {}", ..., pipeline)
:146 LOG.info("{} is started", prefix)
:160 LOG.info("{} is closed for {} with {} requests sent and {} responses
received",
domainSocket.toString(), dn, requestSent, responseReceived)
:545 LOG.info("{} receiveResponseTask is closed after send {} requests and
received {} responses, due to {}",
domainSocket.toString(), requestSent, responseReceived, e.getClass().getName(),
e)
{noformat}
Short-circuit clients are created and closed often. \{{XceiverClientManager}}
keys clients per pipeline, user and short-circuit flag, and evicts them after
\{{scm.container.client.idle.threshold}} (10s). Every Ratis pipeline that
contains the local datanode gets its own short-circuit client. A busy client
such as an HBase RegionServer therefore writes INFO lines continuously in
normal operation.
Line 545 also passes the exception as the last argument, so every normal idle
close (\{{SocketTimeoutException}}, \{{EOFException}},
\{{ClosedChannelException}}) logs a full stack trace at INFO.
{\{domainSocket.toString()}} is evaluated eagerly at lines 160 and 545 even
when INFO is disabled.
h3. Evidence
async-profiler wall profile of an HBase RegionServer (Ratis, short-circuit read
on, YCSB E). The Ozone build predates HDDS-16812, so evicted clients were
closed on the RPC handler thread:
* \{{XceiverClientShortCircuit.close}} took 9.0% of busy handler time;
* inside it, 60.5% is \{{DomainSocket.toString}} (\{{String.format}}) for the
INFO line at :160, and 39.2% is \{{DomainSocket.close}}.
HDDS-16812 moves the close off the caller thread, so the latency effect is gone
on master. The log volume and the eager formatting remain.
h3. Proposed fix
* Lines 129, 146 and 160: log at DEBUG.
* Line 160: pass \{{domainSocket}} itself instead of
\{{domainSocket.toString()}}, so the string is built only when the line is
logged.
* Line 545: for the expected idle-close exceptions
(\{{SocketTimeoutException}}, \{{EOFException}}, \{{ClosedChannelException}}),
log at DEBUG with the exception class name only. Unexpected exceptions keep
their stack trace at WARN, as the \{{catch (Throwable e)}} branch below it does.
h3. Compatibility
Log output only. Operators who relied on these INFO lines to watch
short-circuit client churn can enable DEBUG for
\{{org.apache.hadoop.hdds.scm.XceiverClientShortCircuit}}.
h3. Testing
No functional change. Existing short-circuit tests
(\{{TestXceiverClientManagerSC}}, \{{TestLocalChunkInputStream}}) cover the
code paths.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]