Andrey Yarovoy created HDDS-17780:
-------------------------------------

             Summary: Ozone client starts at a random OM, so a new client's 
first call goes to a non-leader two times in three
                 Key: HDDS-17780
                 URL: https://issues.apache.org/jira/browse/HDDS-17780
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Client
            Reporter: Andrey Yarovoy
            Assignee: Andrey Yarovoy


h3. Problem

Since HDDS-9279 (OM follower reads), \{{HadoopRpcOMFailoverProxyProvider}} and 
\{{GrpcOMFailoverProxyProvider}} shuffle the configured OMs 
(\{{Collections.shuffle(omProxies)}} in \{{initOmProxiesFromConfigs}}). Each 
new provider starts at index 0 of its own shuffled list. The shuffle spreads 
follower reads, but it also applies to the leader-only provider, so every new 
OM client picks its first OM at random.

With 3 OMs, a new client's first leader-bound call therefore goes to a follower 
two times in three. The follower answers \{{OMNotLeaderException}} with the 
suggested leader, and the client retries there. Each such retry costs:
* an extra RPC round trip to the follower;
* a new connection to the follower, which on a secure cluster means TCP, SASL 
and a Kerberos service ticket for that OM;
* a \{{RetryInvocationHandler}} INFO line with a full stack trace (HDDS-16383).

Nothing is remembered between providers. A JVM that creates several OM clients 
pays this again for each one, even though the first one already found the 
leader. That happens in MapReduce AppMasters (\{{FileContext}} creates a new 
file system each time), Spark and Hive drivers, the S3 Gateway, and CLIs.

h3. Measured

*Compose (\{{ozonesecure-ha}}, 3 OMs, leadership moved between rounds, one new 
client per probe):*
* 14 of 30 probes hit one not-leader retry, with om1 and with om3 as leader 
alike. A uniform shuffle predicts about 20
of 30; the difference is unexplained.
* Those probes took *19–26 ms longer* in \{{FileSystem.get}}, about 23 ms on 
the median.
* That's on one host. On a real network, the extra connection and Kerberos 
handshake cost more.

*Cluster (12 workers, 3 OMs, Ozone 2.3.0, 64 TestDFSIO job clients, one JVM 
each):*

||not-leader retries per job-client JVM||0||1||2||3||4||5||6||8||
|JVMs|4|6|9|19|13|5|5|3|

* 212 retries in total, *3.3 per JVM*.
* Split evenly between the two followers (102 and 110), as a random start 
predicts.
* Every retry reached the leader with one failover.

The YARN containers of each job (localizers, AppMaster, tasks) start their own 
clients too, so the per-job total is higher.

h3. Proposed change

Remember the last known leader per OM service in the JVM, and start new 
leader-bound providers there.
# In \{{OMFailoverProxyProviderBase}}, add a static \{{ConcurrentMap<String, 
String>}} from \{{omServiceId}} to the last known leader node id.
# *Write it* when a leader-bound call succeeds on a node, or when 
\{{OMNotLeaderException}} names a suggested leader that is in the configured 
OMs.
# *Clear it* when the hinted node itself answers \{{OMNotLeaderException}}. 
Writes are idempotent and a stale hint is harmless, so plain \{{put}} and 
\{{remove(key, value)}} are enough.
# *Use it* in the constructor of the leader-bound providers 
(\{{HadoopRpcOMFailoverProxyProvider}}, \{{GrpcOMFailoverProxyProvider}}): keep 
the shuffle, then move the hinted node to the front if it is configured.
# *Leave follower reads alone:* the read path of 
\{{HadoopRpcOMFollowerReadFailoverProxyProvider}} keeps starting at a random 
OM. Only its inner leader proxy uses the hint.

*What doesn't change:*
* the \{{OMNotLeaderException}} handling, the retry policy and the 
suggested-leader logic;
* correctness: the hint is only a starting point. A stale hint costs one retry, 
as every client pays today, and the hint then corrects itself;
* config, wire and proto: none.

*Out of scope:* sharing the hint across processes. A single-client JVM, such as 
a YARN localizer or a map task, still starts at random. It could be carried, 
for example, in the job configuration or in a node-local file, but each has its 
own trade-offs. That needs a separate discussion.

h3. Expected result

* In a JVM that creates many OM clients, at most the first one pays a 
not-leader retry. In the cluster job clients that's 3.3 → at most 1 per JVM.
* Fewer connections from new clients to followers, and fewer stack-trace INFO 
lines.
* No change for a JVM's first client.

h3. Testing

* *Unit*, with 3 configured OMs and a fake transport:
** a second provider for the same service starts at the leader that the first 
one found;
** a stale hint (the hinted node answers not-leader) costs one retry and 
replaces the hint;
** a hint naming a node that isn't configured is ignored;
** hints for different \{{omServiceId}}s are independent;
** concurrent providers updating the hint leave a valid entry;
** the follower-read path still spreads reads.
* *Integration* (MiniOzoneHAClusterImpl, 3 OMs):
** create several clients one after another, and check that only the first one 
hits \{{OMNotLeaderException}}, by log capture or a client counter;
** transfer leadership, then check that the next client recovers with one retry.
* *Cluster or compose:* count "is not the leader" per job-client log before and 
after.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to