[
https://issues.apache.org/jira/browse/HDDS-17061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17061:
------------------------------
Attachment: TestBugMC11StaleRowAfterFailedRebuild.java
> Recon drops the replicas of a DELETED container when it moves it back to
> SCM's live state
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-17061
> URL: https://issues.apache.org/jira/browse/HDDS-17061
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-10-keep-replicas-when-recon-reopens-deleted-container.patch,
> TestBugMC10RebuildDropsReplicas.java,
> TestBugMC11StaleRowAfterFailedRebuild.java
>
>
> h3. Mechanism
> When Recon has a container DELETED but SCM lists it as CLOSED or
> QUASI_CLOSED, {{ReconStorageContainerSyncHelper.rebuildContainerFromScm}}
> calls {{deleteContainer}}, which removes the whole entry including its
> replica set, and then {{addNewContainer}} with SCM's {{ContainerInfo}}. Two
> symptoms follow:
> # The re-added container has no replicas. The next health sweep
> ({{ReconReplicationManager.processAll}}) records it MISSING although
> datanodes still hold it, at the latest until each datanode's next full
> container report ({{hdds.container.report.interval}}, default 60m) adds its
> replica back.
> # If {{addNewContainer}} throws, the failure is only logged and the container
> is no longer in Recon's map. {{storeHealthStatesToDatabase}} rewrites
> UNHEALTHY_CONTAINERS rows only for containers in the map, so a row the
> container had before stays. {{ContainerEndpoint.toUnhealthyMetadata}} then
> gets {{ContainerNotFoundException}} for it and
> {{/api/v1/containers/unhealthy}} and {{/api/v1/containers/unhealthy/MISSING}}
> fail with HTTP 500 for every page that includes the row, until a later sync
> adds the container back.
> SCM's own report handler moves a container back from DELETED in place with
> {{ContainerManager.transitionDeletingOrDeletedToTargetState}}, keeping the
> record and its replicas.
> h3. Trigger
> The rebuild runs only when Recon has the container DELETED while SCM lists it
> as live, and Recon has not moved it back itself. Recon's report handler moves
> it back in place when it gets the same datanode report that made SCM do so,
> so the state needs Recon to miss or misjudge that report. The route that was
> run: Recon's DELETED pass reads a page of SCM's DELETED IDs, a datanode
> reports a non-empty CLOSING replica that makes SCM move the container to
> QUASI_CLOSED (HDDS-14684) before the page is processed, Recon retires the
> container from the stale page while keeping the replica, and the datanode
> then closes its replica (bcsId not above the container's), which does not
> move a DELETED container back. The next sync cycle runs the rebuild. Another
> route, argued and not run: Recon is down while the datanode reports the
> replica that makes SCM move the container back.
> Symptom 2 also needs {{addNewContainer}} to fail and a MISSING row from
> before the container was retired, which survives only if no health sweep ran
> between the two syncs. At defaults the sweep runs every 5 minutes
> ({{ozone.recon.task.missingcontainer.interval}}) and the sync every 6 hours,
> so this needs a second sync within minutes of the one that retired the
> container: a sync interval configured below the sweep interval, or a Recon
> restart, where the first sync runs 1 minute after startup (argued, not run).
> For a CLOSED or QUASI_CLOSED container {{addNewContainer}} can fail only on a
> container table write error or if Recon's pipeline sync removes the
> container's pipeline between the pipeline check and the pipeline update
> inside {{ContainerStateManagerImpl.addContainer}}. Neither was driven; the
> reproduction injects the exception.
> h3. Impact
> * Symptom 1: a false MISSING container in Recon's unhealthy containers API
> and UI for up to one full report interval. No data is lost; the replica
> history in Recon's DB is kept. The same code is in 2.2.0 and 2.2.1.
> * Symptom 2: the unhealthy containers API returns HTTP 500 until the next
> successful sync ({{ozone.recon.scm.container.sync.task.interval.delay}},
> default 6h).
> h3. Reproduction
> Both PASS, deterministic, unmodified source, in {{hadoop-ozone/recon}}. A
> passing test means the defect is present.
> * [^TestBugMC10RebuildDropsReplicas.java]: real {{ReconContainerManager}},
> {{ReconContainerReportHandler}} fed with full reports, {{SCMNodeManager}},
> sync helper and health sweep (its database writes captured by a mock); only
> the SCM client is a fake backed by a state table, and the SCM state change
> and datanode report are placed right after the DELETED page RPC returns. For
> SCM QUASI_CLOSED and for SCM CLOSED, the rebuild leaves 0 replicas and the
> sweep writes MISSING; the next full report restores the replica and the row
> goes away.
> * [^TestBugMC11StaleRowAfterFailedRebuild.java]: real container manager, sync
> helper, health sweep, UNHEALTHY_CONTAINERS table (Derby) and
> {{ContainerEndpoint}}; the SCM client is a mock and one {{IOException}} is
> injected into {{addNewContainer}}. The container leaves the map, its MISSING
> row survives three sweeps, and {{getUnhealthyContainers}} throws a
> {{WebApplicationException}} with status 500 caused by
> {{ContainerNotFoundException}}. The next sync re-adds the container and the
> endpoint returns 200. A control test with a sweep between the two syncs shows
> no failure.
> h3. Patch
> [^MC-10-keep-replicas-when-recon-reopens-deleted-container.patch], against
> 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
> 62d982ff2d9.
> {{rebuildContainerFromScm}} still checks that SCM lists the container in that
> state, then moves Recon's record back in place with
> {{transitionDeletingOrDeletedToTargetState}}, as SCM's report handler does,
> instead of deleting and re-adding it. The replicas are kept, and if the
> update fails the container stays in Recon's map with its replicas, so the
> next sweep rewrites its rows and the next sync tries again. This fixes both
> symptoms. Recon keeps its own copy of the container metadata (sequence ID,
> key count, used bytes) instead of taking SCM's; the report handler keeps
> those up to date from replica reports.
> Covered by {{containerDeletedInReconButClosedInScmKeepsReplicas}} and
> {{failedCorrectionFromDeletedKeepsContainerInRecon}} in
> {{TestReconSCMContainerSyncIntegration}}. Without the change they fail (0
> replicas; container gone from the map); with it the Recon {{scm}} package
> tests pass and checkstyle is clean. With the patch the reproductions fail at
> their defect assertions: the replica is kept, and {{addNewContainer}} is no
> longer called so the container stays in the map. The control test of the
> second reproduction passes with and without the patch; it only checks that
> the endpoint answers 200 when a sweep ran between the two syncs.
> Not changed: the stale DELETED page itself. Recon still retires such a
> container and shows it DELETED until the next sync moves it back. A tolerant
> {{ContainerEndpoint.toUnhealthyMetadata}} (skip and log a row whose container
> is not in the map) would be separate hardening.
> Found by TLA+ model checking and code review of the datanode container report
> and Recon container tracking paths under HDDS-15926, on commit
> 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
