Andrey Yarovoy created HDDS-17780:
-------------------------------------
Summary: Ozone client starts at a random OM, so a new client's
first call goes to a non-leader two times in three
Key: HDDS-17780
URL: https://issues.apache.org/jira/browse/HDDS-17780
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
Assignee: Andrey Yarovoy
h3. Problem
Since HDDS-9279 (OM follower reads), \{{HadoopRpcOMFailoverProxyProvider}} and
\{{GrpcOMFailoverProxyProvider}} shuffle the configured OMs
(\{{Collections.shuffle(omProxies)}} in \{{initOmProxiesFromConfigs}}). Each
new provider starts at index 0 of its own shuffled list. The shuffle spreads
follower reads, but it also applies to the leader-only provider, so every new
OM client picks its first OM at random.
With 3 OMs, a new client's first leader-bound call therefore goes to a follower
two times in three. The follower answers \{{OMNotLeaderException}} with the
suggested leader, and the client retries there. Each such retry costs:
* an extra RPC round trip to the follower;
* a new connection to the follower, which on a secure cluster means TCP, SASL
and a Kerberos service ticket for that OM;
* a \{{RetryInvocationHandler}} INFO line with a full stack trace (HDDS-16383).
Nothing is remembered between providers. A JVM that creates several OM clients
pays this again for each one, even though the first one already found the
leader. That happens in MapReduce AppMasters (\{{FileContext}} creates a new
file system each time), Spark and Hive drivers, the S3 Gateway, and CLIs.
h3. Measured
*Compose (\{{ozonesecure-ha}}, 3 OMs, leadership moved between rounds, one new
client per probe):*
* 14 of 30 probes hit one not-leader retry, with om1 and with om3 as leader
alike. A uniform shuffle predicts about 20
of 30; the difference is unexplained.
* Those probes took *19–26 ms longer* in \{{FileSystem.get}}, about 23 ms on
the median.
* That's on one host. On a real network, the extra connection and Kerberos
handshake cost more.
*Cluster (12 workers, 3 OMs, Ozone 2.3.0, 64 TestDFSIO job clients, one JVM
each):*
||not-leader retries per job-client JVM||0||1||2||3||4||5||6||8||
|JVMs|4|6|9|19|13|5|5|3|
* 212 retries in total, *3.3 per JVM*.
* Split evenly between the two followers (102 and 110), as a random start
predicts.
* Every retry reached the leader with one failover.
The YARN containers of each job (localizers, AppMaster, tasks) start their own
clients too, so the per-job total is higher.
h3. Proposed change
Remember the last known leader per OM service in the JVM, and start new
leader-bound providers there.
# In \{{OMFailoverProxyProviderBase}}, add a static \{{ConcurrentMap<String,
String>}} from \{{omServiceId}} to the last known leader node id.
# *Write it* when a leader-bound call succeeds on a node, or when
\{{OMNotLeaderException}} names a suggested leader that is in the configured
OMs.
# *Clear it* when the hinted node itself answers \{{OMNotLeaderException}}.
Writes are idempotent and a stale hint is harmless, so plain \{{put}} and
\{{remove(key, value)}} are enough.
# *Use it* in the constructor of the leader-bound providers
(\{{HadoopRpcOMFailoverProxyProvider}}, \{{GrpcOMFailoverProxyProvider}}): keep
the shuffle, then move the hinted node to the front if it is configured.
# *Leave follower reads alone:* the read path of
\{{HadoopRpcOMFollowerReadFailoverProxyProvider}} keeps starting at a random
OM. Only its inner leader proxy uses the hint.
*What doesn't change:*
* the \{{OMNotLeaderException}} handling, the retry policy and the
suggested-leader logic;
* correctness: the hint is only a starting point. A stale hint costs one retry,
as every client pays today, and the hint then corrects itself;
* config, wire and proto: none.
*Out of scope:* sharing the hint across processes. A single-client JVM, such as
a YARN localizer or a map task, still starts at random. It could be carried,
for example, in the job configuration or in a node-local file, but each has its
own trade-offs. That needs a separate discussion.
h3. Expected result
* In a JVM that creates many OM clients, at most the first one pays a
not-leader retry. In the cluster job clients that's 3.3 → at most 1 per JVM.
* Fewer connections from new clients to followers, and fewer stack-trace INFO
lines.
* No change for a JVM's first client.
h3. Testing
* *Unit*, with 3 configured OMs and a fake transport:
** a second provider for the same service starts at the leader that the first
one found;
** a stale hint (the hinted node answers not-leader) costs one retry and
replaces the hint;
** a hint naming a node that isn't configured is ignored;
** hints for different \{{omServiceId}}s are independent;
** concurrent providers updating the hint leave a valid entry;
** the follower-read path still spreads reads.
* *Integration* (MiniOzoneHAClusterImpl, 3 OMs):
** create several clients one after another, and check that only the first one
hits \{{OMNotLeaderException}}, by log capture or a client counter;
** transfer leadership, then check that the next client recovers with one retry.
* *Cluster or compose:* count "is not the leader" per job-client log before and
after.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]