[
https://issues.apache.org/jira/browse/HDDS-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sadanand Shenoy resolved HDDS-16057.
------------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
> OM bootstrap fails when checkpoint install is rejected during BOOTSTRAPPING
> ---------------------------------------------------------------------------
>
> Key: HDDS-16057
> URL: https://issues.apache.org/jira/browse/HDDS-16057
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sadanand Shenoy
> Assignee: Sadanand Shenoy
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> HDDS-15103 added state checks before triggering installSnapshot which caused
> this
> OM bootstrap fails to add a new node to the Ratis ring when catch-up requires
> an OM checkpoint install (leader logs purged).
> {{installSnapshotFromLeader()}} aborts because {{{}omState ==
> BOOTSTRAPPING{}}}, causing a deadlock with Ratis {{setConfiguration}} staging.
> h2. Root cause
> During {{{}--bootstrap{}}}, the main thread blocks in {{bootstrap()}} while
> the leader runs {{setConfiguration}} staging. Staging waits for the new peer
> to catch up. The leader triggers OM checkpoint install, but
> {{installSnapshotFromLeader()}} rejects the request with {{!isRunning()}}
> while {{{}omState == BOOTSTRAPPING{}}}. Catch-up never completes, so the
> 3-peer config is never committed to the raft log.
> The HDDS-15103 guard is correct for normal restart; it unintentionally blocks
> the bootstrap catch-up path.
> h2. Timeline
> ||Time||Node||Event||
> |T1|New OM|{{{}ozone om --bootstrap{}}}, {{{}omState=BOOTSTRAPPING{}}}, Ratis
> starts with empty peer group|
> |T2|New OM|{{start()}} finishes (secret manager, RPC, Ratis up), calls
> {{bootstrap()}} — main thread blocks|
> |T3|New OM → Leader|Bootstrap RPC sent|
> |T4|Leader|{{startSetConfiguration}} (3 peers) — staging begins, conf NOT
> written to raft log|
> |T5|Leader → New OM|Install snapshot notify (logs purged; e.g. first
> available index 213, leader at 542)|
> |T6|New OM|{{installSnapshotFromLeader()}} → ABORT: state BOOTSTRAPPING →
> {{SNAPSHOT_UNAVAILABLE}}|
> |T7|Leader ↔ New OM|Staging loop: leader retries install notify, catch-up
> never completes|
> |T8|Cluster|Committed Ratis membership stays at 2 peers — new OM never joins
> the ring|
> h2. Symptoms
> * Bootstrap hangs / never completes
> * {{ozone admin om roles}} shows 2 OMs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
