Luis Pigueiras created HDDS-16383:
-------------------------------------
Summary: Ozone client logs full stack trace at INFO on every OM
leader failover (RetryInvocationHandler)
Key: HDDS-16383
URL: https://issues.apache.org/jira/browse/HDDS-16383
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Affects Versions: 2.2.0
Reporter: Luis Pigueiras
h3. Summary
When accessing Ozone via hdfs commands (like {{hdfs dfs -ls ofs://...}}, every
freshly-created client starts by contacting a *random* OM rather than a
cached/known leader. With N configured OMs in HA, this means there is only a
1-in-N chance of hitting the actual leader on the first try, the other (N-1)/N
of the time, the client gets redirected via an {{OMNotLeaderException}}, which
is logged at INFO with a full exception and stack trace.
As far as I understand, this logging retrial mechanism is normal with a OM HA
setup, so having it as a full INFO-level stack trace overstates its severity
and adds noise to normal client commands.
h3. Steps to reproduce
# Repeat any {{hdfs dfs}} / {{ofs://}} command against a cluster:
{code}
hdfs dfs -ls ofs://<cluster>/<existingpath>/
{code}
# Eventually you will get some message like
{code}
INFO retry.RetryInvocationHandler: org.apache...ServiceException:
org.apache.hadoop.ipc_.RemoteException(...OMNotLeaderException):
OM:<om-host> is not the leader. Suggested leader is OM:<om-host2>
at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer...
... (full stack trace)
{code}
h3. Expected behavior
This should probably not be logged as a full INFO-level stack trace? Maybe
DEBUG level, or a terse single-line INFO message without the stack trace, would
be more appropriate, this is an expected, transparently-handled event, not an
error condition. Right now we have modified the log4j.properties to add this to
reduce the noise from client commands.
{code}
log4j.logger.org.apache.hadoop.io_.retry.RetryInvocationHandler=WARN
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
