Siyao Meng created HDDS-17326:
---------------------------------
Summary: Client keeps waiting for ALL_COMMITTED on a pipeline with
a failed follower
Key: HDDS-17326
URL: https://issues.apache.org/jira/browse/HDDS-17326
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Siyao Meng
Assignee: Siyao Meng
With the default {{hdds.ratis.client.request.watch.type}}=ALL_COMMITTED,
{{XceiverClientRatis.watchForCommit}} waits for all replicas first. Only after
that watch fails (up to {{hdds.ratis.raft.client.rpc.watch.request.timeout}},
30s) does it fall back to MAJORITY_COMMITTED and drop the failed server from
its commit info map. The fallback is not remembered: the next watch on the same
pipeline asks for ALL_COMMITTED again, unless the commit index already cached
for the remaining replicas covers it. A new {{XceiverClientRatis}} (e.g. after
the client cache entry expires, or in a new client) starts without this
information.
A watch is sent on every putBlock (every
{{ozone.client.stream.buffer.flush.size}}, 16MB), hsync and close, and the
writer waits on it when the stream buffer is full. So until SCM closes the
pipeline (e.g. after {{hdds.ratis.raft.server.rpc.slowness.timeout}}, 300s, or
when the node becomes stale), writes to a pipeline with a dead follower can
wait up to the watch timeout again on later watches. Short lived writers and
writers that call hsync often (e.g. HBase WAL) pay at least one such wait per
key.
Proposed fix: after a fallback, remember for a while (per pipeline) that a
server failed, and watch with MAJORITY_COMMITTED during that time.
Related: HDDS-10978
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
