[
https://issues.apache.org/jira/browse/HDDS-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17064:
------------------------------
Attachment: TestBugMC2LostFcrNotResent.java
> Full container report carried by a failed heartbeat is not resent until the
> next periodic report, hiding volume failure losses for up to two hours
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17064
> URL: https://issues.apache.org/jira/browse/HDDS-17064
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-resend-container-report-after-failed-heartbeat.patch,
> TestBugMC2LostFcrNotResent.java
>
>
> h3. Mechanism
> {{StateContext.getFullReports}} clears the per endpoint "ready to send" flag
> of the container report as soon as it puts the report into a heartbeat
> request (HDDS-5111). If that heartbeat fails with an {{IOException}},
> {{HeartbeatEndpointTask.putBackIncrementalReports}} puts back only
> incremental and command status reports; its comment says the full reports are
> kept because only the latest is needed, but nothing sets the flag again. The
> container report is therefore not sent until the next
> {{StateContext.refreshFullReport}}, which happens only at the next periodic
> container report or the next volume failure.
> Since HDDS-8448, {{ContainerSet.handleVolumeFailures}} removes the containers
> of a failed volume without sending ICRs and relies on that full container
> report (FCR) to tell SCM and Recon about the loss. If the one heartbeat
> carrying it fails, the loss is not reported.
> h3. Trigger
> # A volume fails its health check; {{ContainerSet.handleVolumeFailures}}
> removes its containers, refreshes the FCR and triggers a heartbeat.
> # That heartbeat fails while SCM stays up, for example an RPC timeout
> ({{ozone.scm.heartbeat.rpc-timeout}}, 5s by default) or a reset of an open
> connection. A refused connection is first retried by the RPC client (15
> times, 1s apart, by default), and after an SCM restart the datanode
> re-registers and sends a fresh report.
> # Later heartbeats succeed, without the FCR.
> The same happens to a periodic FCR, delaying the receiver's reconciliation by
> one more period.
> h3. Impact
> * SCM keeps the replicas of every container on the failed volume until the
> next periodic FCR from that datanode, up to one report period later
> ({{hdds.container.report.interval}} plus a random delay of up to the same
> interval, chosen when the datanode starts, so a fixed period between 60 and
> 120 minutes by default). In that time ReplicationManager does not see these
> containers as under replicated, so their re-replication is delayed by up to
> that long (argued from the code, the SCM side was not run). SCM only asks a
> datanode to re-register when it is not registered (as after an SCM restart,
> which therefore does not hit this), so a reachable datanode is not asked to
> send a fresh report. In SCM HA each SCM has its own flag, so only the SCMs
> whose heartbeat failed miss the report, and ReplicationManager is affected
> when the leader is one of them.
> * Recon shows the same lost replicas in its replica and container health
> views for the same time (run). Recon asks a datanode to re-register when its
> heartbeats stop for 3 heartbeat intervals (180s by default), so at Recon only
> failures shorter than that are affected.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugMC2LostFcrNotResent.java]
> ({{hadoop-ozone/recon}}) runs the real datanode {{StateContext}},
> {{ContainerSet.handleVolumeFailures}} and {{HeartbeatEndpointTask}} against a
> stub RPC proxy that hands each heartbeat's FCR to a real
> {{ReconContainerReportHandler}} over a real {{ReconContainerManager}} and
> {{SCMNodeManager}}. The volume is a mock whose {{isFailed}} is switched to
> true, and the stub fails one heartbeat with a {{ConnectException}}, as the
> RPC client throws once its connection retries are used up. After the failed
> heartbeat and 10 successful ones, Recon has received only the first FCR and
> still lists the replica the datanode removed; a control run without the
> failed heartbeat removes it at once. Heartbeats do not go through
> {{ReconNodeManager.processHeartbeat}}, so the 180s re-register check was
> read, not run. A passing test means the defect is present.
> h3. Patch
> [^MC-2-resend-container-report-after-failed-heartbeat.patch], against
> 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
> 62d982ff2d9.
> A new {{StateContext.putBackFullContainerReport}} sets the container report
> flag of the endpoint again, and {{putBackIncrementalReports}} calls it when
> the failed request carried a container report. The next heartbeat then sends
> the stored container report, which is the one that failed or a newer one.
> Node and pipeline reports are left as they are, since their publishers
> refresh them every 60s by default. The container report now rides on every
> following heartbeat until one succeeds. If the heartbeat failed after SCM
> processed it, SCM receives the same report twice, which it handles as an
> ordinary repeated report. A heartbeat that fails because it is too large (the
> situation of HDDS-16723) keeps carrying the report and can keep failing
> longer than before, when only the ICR backlog was retried.
> Covered by {{testFailedHeartbeatResendsContainerReport}} in the existing
> {{TestHeartbeatEndpointTask}}: after a failed heartbeat, the next one must
> carry the container report again and the one after must not, and a later
> failed heartbeat that carried no container report must not cause one to be
> sent. Without the change it fails because the second heartbeat has no
> container report, and without the check that the failed request carried the
> report it fails because the fourth heartbeat carries one. With it
> {{TestHeartbeatEndpointTask}}, {{TestStateContext}}, {{TestContainerSet}},
> {{TestReportPublisher}}, {{TestVolumeSetDiskChecks}}, {{TestOzoneContainer}},
> {{TestDatanodeStateMachine}} and {{TestRunningDatanodeState}} (108 tests) and
> {{TestEndPoint}} in server-scm (18 tests) pass, checkstyle is clean. With the
> patch the reproduction fails, because Recon receives the FCR again with the
> next heartbeat and removes the replica.
> Found by TLA+ model checking and code review of the datanode container report
> and Recon container tracking paths under HDDS-15926, on commit
> 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
