[
https://issues.apache.org/jira/browse/HDDS-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136390#comment-18136390
]
Chia-Chuan Ho commented on HDDS-17125:
--------------------------------------
Hi [~smeng]
Thank you for the detailed analysis and the proposed approaches.
1. Return success when the target is already the leader
* Directly addresses the reported retry to omNode-2 without starting another
priority-setting / transfer / priority-reset cycle.
* Keeps the change small and preserves existing client failover behavior.
* Does not cover retries before the initial transfer completes, or -r requests
that may select a different target.
2. Stop retrying on another OM after a timeout
* Prevents an automatic replay when the outcome of the first request is
unknown, covering both explicit-target and -r requests.
* May report a timeout to the client even when the transfer eventually
succeeds.
* Requires distinguishing ambiguous failures from explicit NotLeader /
LeaderNotReady responses, for which retries should remain allowed.
My initial preference would be to start with the server-side guard and a
regression test for the reported scenario. Would that be a reasonable first
step, or to include timeout retry handling in this Jira as well?
> OM transferLeadership is not idempotent under client failover retry
> -------------------------------------------------------------------
>
> Key: HDDS-17125
> URL: https://issues.apache.org/jira/browse/HDDS-17125
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: OM HA
> Reporter: Siyao Meng
> Assignee: Chia-Chuan Ho
> Priority: Major
>
> If the {{transferLeadership}} admin RPC takes longer than
> {{ozone.om.client.rpc.timeout}} (see HDDS-17124 for why it can take 4-13 s),
> the OM client fails over and sends the same request to the next OM. After a
> successful transfer that is the *new* leader, which then runs a second
> set-priority / transfer / reset-priority cycle while the first one is still
> resetting priorities.
> Observed in a 5-node test (5 s client RPC timeout):
> * 17.52 omNode-1 starts transfer to omNode-2; 22.52 client
> SocketTimeoutException and failover.
> * 24.21 transfer to omNode-2 succeeds; 24.55 the retried request arrives at
> omNode-2: "Start transferring leadership to omNode-2" (to itself).
> * 29.83 / 30.10 both handlers reset priorities; 31.60 omNode-2 steps down and
> omNode-4 is elected. That is an extra leader change nobody asked for.
> * Client fails with "Could not determine or connect to OM Leader" after 10
> failovers.
> *Proposal:* treat "target is already the leader" as success without touching
> priorities, and/or do not retry {{transferLeadership}} on a different OM
> after a timeout.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
