[
https://issues.apache.org/jira/browse/HDDS-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17326:
------------------------------
Epic Link: HDDS-17321
> Client keeps waiting for ALL_COMMITTED on a pipeline with a failed follower
> ---------------------------------------------------------------------------
>
> Key: HDDS-17326
> URL: https://issues.apache.org/jira/browse/HDDS-17326
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> With the default {{hdds.ratis.client.request.watch.type}}=ALL_COMMITTED,
> {{XceiverClientRatis.watchForCommit}} waits for all replicas first. Only
> after that watch fails (up to
> {{hdds.ratis.raft.client.rpc.watch.request.timeout}}, 30s) does it fall back
> to MAJORITY_COMMITTED and drop the failed server from its commit info map.
> The fallback is not remembered: the next watch on the same pipeline asks for
> ALL_COMMITTED again, unless the commit index already cached for the remaining
> replicas covers it. A new {{XceiverClientRatis}} (e.g. after the client cache
> entry expires, or in a new client) starts without this information.
> A watch is sent on every putBlock (every
> {{ozone.client.stream.buffer.flush.size}}, 16MB), hsync and close, and the
> writer waits on it when the stream buffer is full. So until SCM closes the
> pipeline (e.g. after {{hdds.ratis.raft.server.rpc.slowness.timeout}}, 300s,
> or when the node becomes stale), writes to a pipeline with a dead follower
> can wait up to the watch timeout again on later watches. Short lived writers
> and writers that call hsync often (e.g. HBase WAL) pay at least one such wait
> per key.
> Proposed fix: after a fallback, remember for a while (per pipeline) that a
> server failed, and watch with MAJORITY_COMMITTED during that time.
> Related: HDDS-10978
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
