Siyao Meng created HDDS-16170:
---------------------------------
Summary: Intermittent failure in
TestSnapshotDiffManager#testLoadJobsOnStartUp (DONE vs IN_PROGRESS)
Key: HDDS-16170
URL: https://issues.apache.org/jira/browse/HDDS-16170
Project: Apache Ozone
Issue Type: Sub-task
Components: Snapshot, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR] Tests run: 60, Failures: 1, Errors: 0, Skipped: 0 <<< FAILURE! -- in
org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager
org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager.testLoadJobsOnStartUp
-- Time elapsed: 0.080 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <DONE> but was: <IN_PROGRESS>
at
org.apache.hadoop.ozone.om.snapshot.TestSnapshotDiffManager.testLoadJobsOnStartUp(TestSnapshotDiffManager.java:1150)
{code}
h3. Root cause
The test submits diff jobs, then stubs {{generateSnapshotDiffReport}} with a
{{doAnswer}} that writes {{DONE}} to the DB inside the answer body, calls
{{loadJobsOnStartUp()}}, and waits with {{attempt(() -> verify(spy,
atLeast(1)).generateSnapshotDiffReport(...))}} before reading the job back from
the DB and asserting {{DONE}}.
Mockito records an invocation before it runs the stubbed answer. So
{{verify(spy, atLeast(1))}} can pass as soon as the worker thread enters
{{generateSnapshotDiffReport}}, while the answer is still executing and has not
yet persisted {{DONE}}. The main thread then reads the job and observes the
original {{IN_PROGRESS}}, so {{assertEquals(DONE, ...)}} fails intermittently.
This is a recurrence of the symptom HDDS-9434 tried to address: that fix only
replaced the fixed sleep with a wait on {{verify(...)}}, which still does not
wait for the answer's DB write to complete.
h3. Fix
Wait on the actual observable end state the test asserts: poll
{{getSnapshotDiffJobFromDb(...).getStatus()}} until it is {{DONE}} instead of
waiting on the Mockito invocation. This is deterministic by construction, since
the read that the assertion depends on is exactly what is waited on.
- https://github.com/apache/ozone/actions/runs/31673201699/job/94364422373
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
