Siyao Meng created HADOOP-20047:
-----------------------------------
Summary: RetryInvocationHandler leaves the call id set when an
attempt issues no RPC
Key: HADOOP-20047
URL: https://issues.apache.org/jira/browse/HADOOP-20047
Project: Hadoop Common
Issue Type: Bug
Components: ipc, common
Affects Versions: 3.4.3, 3.5.0
Reporter: Siyao Meng
RetryInvocationHandler sets the thread local call id and retry count in Client
before every attempt on an RPC proxy, but only the creation of a Client.Call
clears the call id. When an attempt issues no RPC, the call id stays set and
the next attempt on that thread fails in Client.setCallIdAndRetryCount:
{noformat}
java.lang.IllegalStateException
at
org.apache.hadoop.util.Preconditions.checkState(Preconditions.java:254)
at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:167)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366)
{noformat}
Attempts that issue no RPC include calling toString or hashCode on the retry
proxy (noted in HDFS-8409), and a protocol translator or
ProtobufRpcEngine2.Invoker throwing before Client.call. After that, the retry
is aborted and the next call through any retry proxy on the same thread fails.
Asynchronous calls retry on the handler's own thread, so later calls on that
handler fail as well.
The code is in hadoop-common, so this affects every RetryProxy over a Hadoop
RPC proxy, including HDFS clients (HDFS-8409, HDFS-11351) and YARN (RMProxy,
ServerProxy). Apache Ozone fixed the same fault in its copy of this code in
HDDS-16745.
Proposed fix: clear the call id, retry count and external handler after every
attempt that set them, with a regression test in TestRetryProxy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
