[
https://issues.apache.org/jira/browse/HDDS-17066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17066:
------------------------------
Attachment:
MC-7-recon-take-dead-node-op-state-from-scm-before-replica-removal.patch
> Recon removes the replicas of a datanode stopped for maintenance when it has
> not yet learned IN_MAINTENANCE from SCM
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17066
> URL: https://issues.apache.org/jira/browse/HDDS-17066
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-7-recon-take-dead-node-op-state-from-scm-before-replica-removal.patch,
> TestBugMC7ReconDropsMaintenanceReplica.java
>
>
> h3. Mechanism
> {{ReconDeadNodeHandler.onMessage}} first calls {{DeadNodeHandler.onMessage}},
> which keeps the replicas of a dead node only if the node is IN_MAINTENANCE,
> and reads that state from Recon's own node manager. Only after that does it
> fetch the node list from SCM and copy the node's operational state with
> {{ReconNodeManager.updateNodeOperationalStateFromScm}}. HDDS-5277 added that
> lookup to fix the displayed state; it was placed after the replica decision.
> Recon learns the operational state of a live datanode only from the
> datanode's heartbeats ({{ReconNodeManager.updateDatanodeOpState}});
> {{PipelineSyncTask}} copies it from SCM only for nodes that are already DEAD.
> The datanode gets IN_MAINTENANCE in a heartbeat reply from SCM and reports it
> to Recon in its next heartbeat. The datanode sends heartbeats to Recon on its
> SCM cycle ({{hdds.heartbeat.interval}}, 30s), so Recon can lag SCM by up to
> about 60s.
> h3. Trigger
> The documented maintenance workflow: {{ozone admin datanode maintenance}} for
> a node, wait until SCM shows it IN_MAINTENANCE, stop the node. If the node is
> stopped within about a minute, Recon still has ENTERING_MAINTENANCE (or
> IN_SERVICE) when the node goes DEAD, and removes its replicas. A script that
> stops the node as soon as SCM shows the state hits this. By code reading (not
> run), the same happens if the node is stopped while SCM still shows
> ENTERING_MAINTENANCE and SCM completes the transition before the node is
> DEAD, since the admin workflow only aborts for a DEAD node.
> h3. Impact
> For the whole maintenance, Recon reports the node's containers
> UNDER_REPLICATED while SCM, whose {{DeadNodeHandler}} keeps the replicas of
> an IN_MAINTENANCE node, reports them healthy. In the attached run with 5
> datanodes, after the stopped node was DEAD, SCM listed 3 replicas including
> the stopped node and its {{ReplicationManager}} report had no
> UNDER_REPLICATED container, while Recon listed 2 replicas and its unhealthy
> containers endpoint returned one UNDER_REPLICATED container. Recon showed
> ENTERING_MAINTENANCE for the node at the moment it was stopped. The replicas
> come back only when the node returns and reports. Monitoring only; SCM and
> the data are not affected.
> h3. Reproduction
> [^TestBugMC7ReconDropsMaintenanceReplica.java] goes in
> {{hadoop-ozone/integration-test-recon}} (target path and Maven command in the
> class comment) and passes on unmodified source when the bug is present. It
> starts a mini cluster with 5 datanodes and Recon, writes a key, puts one
> replica holder in maintenance through the SCM admin API
> ({{ScmClient.startMaintenanceNodes}}), waits until SCM shows IN_MAINTENANCE,
> stops the datanode at once and waits until it is DEAD. It uses a 1s heartbeat
> and short stale and dead intervals to keep the run short, and no hooks; Recon
> still had ENTERING_MAINTENANCE when the node was stopped.
> h3. Patch
> [^MC-7-recon-take-dead-node-op-state-from-scm-before-replica-removal.patch],
> against 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master
> at 14e0611fdd5.
> {{ReconDeadNodeHandler.onMessage}} now fetches the state from SCM first, then
> calls {{DeadNodeHandler.onMessage}}, then runs the container health and
> pipeline sync tasks. Recon then keeps the replicas of a node that SCM has
> IN_MAINTENANCE, like SCM does. If the SCM call fails, the error is logged as
> before, Recon decides with its own state, which is the current behaviour, and
> the container health and pipeline sync tasks are skipped as before.
> Covered by the new parameterized
> {{testDeadNodeHandlerUsesOperationalStateFromScm}} in
> {{TestReconNodeManager}}: SCM IN_MAINTENANCE keeps the replica, SCM
> IN_SERVICE and a failing SCM call remove it, and the pipeline sync is skipped
> when the SCM call fails. Without the change the first case fails with a
> replica removal that should not happen. With it the unit tests in the Recon
> {{scm}} and {{fsck}} packages and {{TestReconTasks}} pass, checkstyle is
> clean, and the attached reproduction shows 3 replicas and no UNDER_REPLICATED
> container in Recon.
> Found by TLA+ model checking and code review of the datanode container report
> and Recon container tracking paths under HDDS-15926, on commit
> 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-5277 (resolved, added the SCM state lookup to this handler
> for the displayed state), HDDS-2607 (resolved, SCM keeps the replicas of a
> dead node in maintenance), HDDS-9645 (resolved, Recon counting of out of
> service replicas). The attached patch is a proposal for review. Generated
> with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
