[
https://issues.apache.org/jira/browse/HDDS-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-16382:
-------------------------------
Description:
We need reduce the following SCM client configuration so that OM read and write
critical path is not blocked for too long when SCM is unresponsive
* hdds.scmclient.rpc.timeout
* ipc.client.connect.timeout
* ipc.client.connect.max.retries.on.timeouts
However, OzoneManager instantiates quite a lot of SCM client and each of them
share a single configuration although they have different latency and retry
requirement
* Critical paths (low latency and fail fast)
** scmContainerClient (used in read critical path)
** scmBlockClient (used in write critical path)
*** scmTopologyClient is also part of the scmBlockClient
* SCM security clients (tolerates high latency and unbounded timeout)
** scmSecurityClient
** secretKeyClient
Additionally, they are all stored in the same ClientCache entry. ClientCache is
a Map<SocketFactory, Client> and the standard implementation SocketFactory
returned in NetUtils#getDefaultSocketFactory in StandardSocketFactory that has
the following hashCode
{code:java}
@Override
public int hashCode() {
return this.getClass().hashCode();
} {code}
So this means that even if we use different two configurations, only the first
configured client will be stored in the ClientCache and the subsequent one will
reuse the cache entry. Therefore, we cannot simply create a separate
configuration with different timeout.
The ClientCache#getClient also documented this decision
{code:java}
// Construct & cache client. The configuration is only used for timeout,
// and Clients have connection pools. So we can either (a) lose some
// connection pooling and leak sockets, or (b) use the same timeout for all
// configurations. Since the IPC is usually intended globally, not
// per-job, we choose (a). {code}
Since we need two different timeout, it conflicts with the ClientCache
mechanism.
Therefore, we need another solution to split the two different clients into two
different SocketFactory.
In another note, the Hadoop client configuration seems to be suited only for
normal Hadoop client (15 minutes might make sense for a single Hadoop client).
However, Ozone reuses the Hadoop client into our critical path without
revisiting whether the default timeout makes sense. So in the future, we need
to check whether OM should use Hadoop RPC client for its critical path or
whether we need to implement a high performance for OM and SCM.
was:
We need reduce the following SCM client configuration so that OM read and write
critical path is not blocked for too long when SCM is unresponsive
* hdds.scmclient.rpc.timeout
* ipc.client.connect.timeout
* ipc.client.connect.max.retries.on.timeouts
However, OzoneManager instantiates quite a lot of SCM client and each of them
share a single configuration although they have different latency and retry
requirement
* Critical paths (low latency and fail fast)
** scmContainerClient (used in read critical path)
** scmBlockClient (used in write critical path)
*** scmTopologyClient is also part of the scmBlockClient
* SCM security clients (tolerates high latency and unbounded timeout)
** scmSecurityClient
** secretKeyClient
Additionally, they are all stored in the same ClientCache entry. ClientCache is
a Map<SocketFactory, Client> and the standard implementation SocketFactory
returned in NetUtils#getDefaultSocketFactory in StandardSocketFactory that has
the following hashCode
{code:java}
@Override
public int hashCode() {
return this.getClass().hashCode();
} {code}
So this means that even if we use different two configurations, only the first
configured client will be stored in the ClientCache and the subsequent one will
reuse the cache entry. Therefore, we cannot simply create a separate
configuration with different timeout.
The ClientCache#getClient also documented this decision
{code:java}
// Construct & cache client. The configuration is only used for timeout,
// and Clients have connection pools. So we can either (a) lose some
// connection pooling and leak sockets, or (b) use the same timeout for all
// configurations. Since the IPC is usually intended globally, not
// per-job, we choose (a). {code}
Since we need two different timeout, it conflicts with the ClientCache
mechanism.
Therefore, we need another solution to split the two different clients into two
different SocketFactory.
In another note, the Hadoop client configuration seems to be suited only for
normal Hadoop client (15 minutes might make sense for a single Hadoop client).
However, Ozone reuses the Hadoop client into our critical path without
revisiting whether the timeout makes sense. So in the future, we need to check
whether OM should use Hadoop RPC client for its critical path or whether we
need to implement a high performance for OM and SCM.
> Dedicated SCM client RPC timeout and retry for OM request critical path
> -----------------------------------------------------------------------
>
> Key: HDDS-16382
> URL: https://issues.apache.org/jira/browse/HDDS-16382
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We need reduce the following SCM client configuration so that OM read and
> write critical path is not blocked for too long when SCM is unresponsive
> * hdds.scmclient.rpc.timeout
> * ipc.client.connect.timeout
> * ipc.client.connect.max.retries.on.timeouts
> However, OzoneManager instantiates quite a lot of SCM client and each of them
> share a single configuration although they have different latency and retry
> requirement
> * Critical paths (low latency and fail fast)
> ** scmContainerClient (used in read critical path)
> ** scmBlockClient (used in write critical path)
> *** scmTopologyClient is also part of the scmBlockClient
> * SCM security clients (tolerates high latency and unbounded timeout)
> ** scmSecurityClient
> ** secretKeyClient
> Additionally, they are all stored in the same ClientCache entry. ClientCache
> is a Map<SocketFactory, Client> and the standard implementation SocketFactory
> returned in NetUtils#getDefaultSocketFactory in StandardSocketFactory that
> has the following hashCode
> {code:java}
> @Override
> public int hashCode() {
> return this.getClass().hashCode();
> } {code}
> So this means that even if we use different two configurations, only the
> first configured client will be stored in the ClientCache and the subsequent
> one will reuse the cache entry. Therefore, we cannot simply create a separate
> configuration with different timeout.
> The ClientCache#getClient also documented this decision
> {code:java}
> // Construct & cache client. The configuration is only used for timeout,
> // and Clients have connection pools. So we can either (a) lose some
> // connection pooling and leak sockets, or (b) use the same timeout for all
> // configurations. Since the IPC is usually intended globally, not
> // per-job, we choose (a). {code}
> Since we need two different timeout, it conflicts with the ClientCache
> mechanism.
> Therefore, we need another solution to split the two different clients into
> two different SocketFactory.
> In another note, the Hadoop client configuration seems to be suited only for
> normal Hadoop client (15 minutes might make sense for a single Hadoop
> client). However, Ozone reuses the Hadoop client into our critical path
> without revisiting whether the default timeout makes sense. So in the future,
> we need to check whether OM should use Hadoop RPC client for its critical
> path or whether we need to implement a high performance for OM and SCM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
