Ivan Andika created HDDS-16382:
----------------------------------
Summary: Dedicated SCM client RPC timeout and retry for OM request
critical path
Key: HDDS-16382
URL: https://issues.apache.org/jira/browse/HDDS-16382
Project: Apache Ozone
Issue Type: Improvement
Reporter: Ivan Andika
Assignee: Ivan Andika
We need reduce the following SCM client configuration so that OM read and write
critical path is not blocked for too long when SCM is unresponsive
* hdds.scmclient.rpc.timeout
* ipc.client.connect.timeout
* ipc.client.connect.max.retries.on.timeouts
However, OzoneManager instantiates quite a lot of SCM client and each of them
share a single configuration although they have different latency and retry
requirement
* Critical paths (low latency and fail fast)
** scmContainerClient (used in read critical path)
** scmBlockClient (used in write critical path)
*** scmTopologyClient is also part of the scmBlockClient
* SCM security clients (tolerates high latency and unbounded timeout)
** scmSecurityClient
** secretKeyClient
Additionally, they are all stored in the same ClientCache entry. ClientCache is
a Map<SocketFactory, Client> and the standard implementation SocketFactory
returned in NetUtils#getDefaultSocketFactory in StandardSocketFactory that has
the following hashCode
{code:java}
@Override
public int hashCode() {
return this.getClass().hashCode();
} {code}
So this means that even if we use different two configurations, only the first
configured client will be stored in the ClientCache and the subsequent one will
reuse the cache entry. Therefore, we cannot simply create a separate
configuration with different timeout.
The ClientCache#getClient also documented this decision
{code:java}
// Construct & cache client. The configuration is only used for timeout,
// and Clients have connection pools. So we can either (a) lose some
// connection pooling and leak sockets, or (b) use the same timeout for all
// configurations. Since the IPC is usually intended globally, not
// per-job, we choose (a). {code}
Since we need two different timeout, it conflicts with the ClientCache
mechanism.
Therefore, we need another solution to split the two different clients into two
different SocketFactory.
In another note, the Hadoop client configuration seems to be suited only for
normal Hadoop client (15 minutes might make sense for a single Hadoop client).
However, Ozone reuses the Hadoop client into our critical path without
revisiting whether the timeout makes sense. So in the future, we need to check
whether OM should use Hadoop RPC client for its critical path or whether we
need to implement a high performance for OM and SCM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
