[
https://issues.apache.org/jira/browse/HDDS-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085861#comment-18085861
]
Tsz-wo Sze commented on HDDS-15443:
-----------------------------------
(Original description by [~sumitagrawl])
When leader performs write () and it fails, ratis server do not respond
immediately as it wait for re-election, and other server can operate over this
request in quorum. But since leader is present, re-election do not happen or
its random to get success.
But since reply is not returned by the server, client hangs till timeout occurs
OR pipeline gets close by SCM on this error.
Since the state machine is not usable as no other request is allowed to be
processed. So its better to close, so that having below behavior:
If Leader write() fails and state machine closes,
* leader reply with ServerNotReadyException immediately
* Client will retry as per policy, till either new leader or raft group removal
* leader election will happen if leader is closed within few seconds
* Once new leader is choosen and client retry, it will return success with
majority commit
If One follower write() fails and state machine closes, Still leader will
process client request with majority node success with commit.
SCM on failure of any node,
* will close containers with cool down time (2.5 minute default)
* stop allocating any new blocks
* close pipeline after 5 min
This ensures in-progress write can finish with 2-node running if any.
Impact:
* Do not handle graceful shutdown to finish apply transaction, impact:
** If leader closes, it return failure to client waiting for reply and can
retry
** If one follower closes, majority nodes are present to process and container
closes before pipeline close
** 2-node follower failure - case have only one node having data as expected.
Below issue to be handled with separate JIRA
# 2-node failure case
# client configuration for long wait for commit-all / majority-commit and
other config
> close statemachine immediately on write failure
> -----------------------------------------------
>
> Key: HDDS-15443
> URL: https://issues.apache.org/jira/browse/HDDS-15443
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Sumit Agrawal
> Assignee: Sumit Agrawal
> Priority: Major
> Labels: pull-request-available
>
> When leader performs write () and it fails, ratis server do not respond
> immediately as it wait for re-election, and other server can operate over
> this request in quorum. But since leader is present, re-election do not
> happen or its random to get success.
>
> But since reply is not returned by the server, client hangs till timeout
> occurs OR pipeline gets close by SCM on this error.
>
> Since the state machine is not usable as no other request is allowed to be
> processed. So its better to close, so that having below behavior:
> If Leader write() fails and state machine closes,
> * leader reply with ServerNotReadyException immediately
> * Client will retry as per policy, till either new leader or raft group
> removal
> * leader election will happen if leader is closed within few seconds
> * Once new leader is choosen and client retry, it will return success with
> majority commit
>
> If One follower write() fails and state machine closes, Still leader will
> process client request with majority node success with commit.
>
> SCM on failure of any node,
> * will close containers with cool down time (2.5 minute default)
> * stop allocating any new blocks
> * close pipeline after 5 min
> This ensures in-progress write can finish with 2-node running if any.
>
> Impact:
> * Do not handle graceful shutdown to finish apply transaction, impact:
> ** If leader closes, it return failure to client waiting for reply and can
> retry
> ** If one follower closes, majority nodes are present to process and
> container closes before pipeline close
> ** 2-node follower failure - case have only one node having data as expected.
>
> Below issue to be handled with separate JIRA
> # 2-node failure case
> # client configuration for long wait for commit-all / majority-commit and
> other config
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
