[
https://issues.apache.org/jira/browse/HDDS-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luis Pigueiras updated HDDS-16383:
----------------------------------
Description:
h3. Summary
When accessing Ozone via hdfs commands (like {{hdfs dfs -ls ofs://...}}, every
freshly-created client starts by contacting a random OM rather than a
cached/known leader. With N configured OMs in HA, this means there is only a
1-in-N chance of hitting the actual leader on the first try, the other (N-1)/N
of the time, the client gets redirected via an {{OMNotLeaderException}}, which
is logged at INFO with a full exception and stack trace.
As far as I understand, this logging retrial mechanism is normal with a OM HA
setup, so having it as a full INFO-level stack trace overstates its severity
and adds noise to normal client commands.
h3. Steps to reproduce
# Repeat any {{hdfs dfs}} / {{ofs://}} command against a cluster:
{code}
hdfs dfs -ls ofs://<cluster>/<existingpath>/
{code}
# Eventually you will get some message like
{code}
INFO retry.RetryInvocationHandler: org.apache...ServiceException:
org.apache.hadoop.ipc_.RemoteException(...OMNotLeaderException):
OM:<om-host> is not the leader. Suggested leader is OM:<om-host2>
at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer...
... (full stack trace)
{code}
h3. Expected behavior
This should probably not be logged as a full INFO-level stack trace? Maybe
DEBUG level, or a terse single-line INFO message without the stack trace, would
be more appropriate, this is an expected, transparently-handled event, not an
error condition. Right now we have modified the log4j.properties to add this to
reduce the noise from client commands.
{code}
log4j.logger.org.apache.hadoop.io_.retry.RetryInvocationHandler=WARN
{code}
was:
h3. Summary
When accessing Ozone via hdfs commands (like {{hdfs dfs -ls ofs://...}}, every
freshly-created client starts by contacting a *random* OM rather than a
cached/known leader. With N configured OMs in HA, this means there is only a
1-in-N chance of hitting the actual leader on the first try, the other (N-1)/N
of the time, the client gets redirected via an {{OMNotLeaderException}}, which
is logged at INFO with a full exception and stack trace.
As far as I understand, this logging retrial mechanism is normal with a OM HA
setup, so having it as a full INFO-level stack trace overstates its severity
and adds noise to normal client commands.
h3. Steps to reproduce
# Repeat any {{hdfs dfs}} / {{ofs://}} command against a cluster:
{code}
hdfs dfs -ls ofs://<cluster>/<existingpath>/
{code}
# Eventually you will get some message like
{code}
INFO retry.RetryInvocationHandler: org.apache...ServiceException:
org.apache.hadoop.ipc_.RemoteException(...OMNotLeaderException):
OM:<om-host> is not the leader. Suggested leader is OM:<om-host2>
at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer...
... (full stack trace)
{code}
h3. Expected behavior
This should probably not be logged as a full INFO-level stack trace? Maybe
DEBUG level, or a terse single-line INFO message without the stack trace, would
be more appropriate, this is an expected, transparently-handled event, not an
error condition. Right now we have modified the log4j.properties to add this to
reduce the noise from client commands.
{code}
log4j.logger.org.apache.hadoop.io_.retry.RetryInvocationHandler=WARN
{code}
> Ozone client logs full stack trace at INFO on every OM leader failover
> (RetryInvocationHandler)
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-16383
> URL: https://issues.apache.org/jira/browse/HDDS-16383
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Affects Versions: 2.2.0
> Reporter: Luis Pigueiras
> Priority: Minor
>
> h3. Summary
> When accessing Ozone via hdfs commands (like {{hdfs dfs -ls ofs://...}},
> every freshly-created client starts by contacting a random OM rather than a
> cached/known leader. With N configured OMs in HA, this means there is only a
> 1-in-N chance of hitting the actual leader on the first try, the other
> (N-1)/N of the time, the client gets redirected via an
> {{OMNotLeaderException}}, which is logged at INFO with a full exception and
> stack trace.
> As far as I understand, this logging retrial mechanism is normal with a OM HA
> setup, so having it as a full INFO-level stack trace overstates its severity
> and adds noise to normal client commands.
> h3. Steps to reproduce
> # Repeat any {{hdfs dfs}} / {{ofs://}} command against a cluster:
> {code}
> hdfs dfs -ls ofs://<cluster>/<existingpath>/
> {code}
> # Eventually you will get some message like
> {code}
> INFO retry.RetryInvocationHandler: org.apache...ServiceException:
> org.apache.hadoop.ipc_.RemoteException(...OMNotLeaderException):
> OM:<om-host> is not the leader. Suggested leader is OM:<om-host2>
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer...
> ... (full stack trace)
> {code}
> h3. Expected behavior
> This should probably not be logged as a full INFO-level stack trace? Maybe
> DEBUG level, or a terse single-line INFO message without the stack trace,
> would be more appropriate, this is an expected, transparently-handled event,
> not an error condition. Right now we have modified the log4j.properties to
> add this to reduce the noise from client commands.
> {code}
> log4j.logger.org.apache.hadoop.io_.retry.RetryInvocationHandler=WARN
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
