[
https://issues.apache.org/jira/browse/HDDS-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang updated HDDS-16455:
-----------------------------------
Status: Patch Available (was: In Progress)
> Stale scrub task demotes a CLOSED reconstructed EC replica to UNHEALTHY,
> forcing SCM re reconstruction
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16455
> URL: https://issues.apache.org/jira/browse/HDDS-16455
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Chi-Hsuan Huang
> Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-16455.001.patch
>
>
> h3. Finding
> A stale scrub task silently demotes a successfully-CLOSED reconstructed EC
> replica to UNHEALTHY with no ICR; on the next full container report SCM
> deletes and re-reconstructs a valid replica. Reachable via the normal scrub
> background thread racing the reconstruction/close path; harm is persistent on
> the datanode but bounded (SCM re-replicates), so High rather than Critical.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-ec-reconstruction
> Guidance:
> campaigns/ozone-9fbf9ee/targets/022-dn-ec-reconstruction/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-ec-reconstruction|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. An EC reconstruction target is created RECOVERING; its
> {{recoveringContainerMap}} entry carries a deadline (create time plus
> recovering timeout).
> 2. Reconstruction stalls past the deadline, so
> {{StaleRecoveringContainerScrubbingService.getTasks}} enqueues a
> {{RecoveringContainerScrubbingTask}} and removes the map entry, all while the
> container is still RECOVERING.
> 3. Reconstruction then completes: the replica is moved RECOVERING to CLOSING
> to CLOSED through the normal close path.
> 4. The queued scrub task fires on the background pool. It re-checks only the
> deadline (now null, so it proceeds) and calls {{markContainerUnhealthy()}},
> which has no state guard, so the successfully CLOSED replica is silently
> demoted to UNHEALTHY. No ICR is sent, so SCM only learns of the UNHEALTHY
> replica on the next full container report and then deletes and
> re-reconstructs a valid replica.
> h3. Root cause
> {{StaleRecoveringContainerScrubbingService}} splits its "decide stale" step
> ({{getTasks}}) from its "act" step
> ({{RecoveringContainerScrubbingTask.call}}) across two threads with no state
> re-check between them. {{call()}} re-reads only the deadline (added earlier
> for the deadline-extension race) and never re-checks the container state
> before calling {{Container.markContainerUnhealthy}}.
> {{KeyValueContainer.markContainerUnhealthy}} unconditionally sets state to
> UNHEALTHY, so a container that reached a terminal CLOSED (or CLOSING) state
> between scheduling and execution is degraded anyway.
> h3. Suggested fix
> Attached patch [^HDDS-16455.001.patch] re-checks the container state under
> the container write lock inside {{RecoveringContainerScrubbingTask.call}} and
> marks the container UNHEALTHY only if it is still RECOVERING, mirroring the
> write-lock discipline that {{KeyValueHandler.markContainerForClose}} already
> uses. A container that has moved out of RECOVERING (closed or deleted) is
> left untouched. A unit test in
> {{TestStaleRecoveringContainerScrubbingService}} drives the schedule then
> close then execute interleaving and asserts the replica stays CLOSED.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
