[
https://issues.apache.org/jira/browse/HDDS-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17050:
------------------------------
Attachment: TestBugMC7FailedBatchLostAcrossRestart.java
> Recon task cursor advances past a batch that failed both attempts, so a
> restart soon after the failure can lose that batch
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17050
> URL: https://issues.apache.org/jira/browse/HDDS-17050
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-7-skip-batches-behind-failed-batch-until-reinit.patch,
> TestBugMC7FailedBatchLostAcrossRestart.java
>
>
> h3. Mechanism
> When a delta batch fails both attempts in
> {{ReconTaskControllerImpl.processOMUpdateBatch}}, the controller only sets
> the in memory {{tasksFailed}} flag; the next sync iteration turns it into a
> reinitialization. Batches that are already in the event buffer behind the
> failed one are still processed, and for each task that succeeds
> {{processTasks}} saves {{lastUpdatedSeqNumber}} to the task status table. The
> cursor of the failed task therefore reaches the cursor of {{OmDeltaRequest}}.
> If Recon restarts before the reinitialization has finished and before the
> sync has fetched any new OM update, the flag and the queued reinitialization
> are gone, and the startup check in {{OzoneManagerServiceProviderImpl.start}},
> which rebuilds only tasks whose cursor is below that of {{OmDeltaRequest}},
> finds nothing to do. The failed batch is never applied.
> The same happens when a manual rebuild drains a buffered batch and the
> rebuild's reprocess step then fails, because that failure also only sets
> {{tasksFailed}} (code reading, not run). A rebuild that fails in its DB swap
> sets no flag at all, because {{reInitializeTasks}} still reports success;
> HDDS-17047 makes it report the failure.
> h3. Trigger
> # The sync buffers two batches before the event thread gets to the first, as
> when it fetches several deltas in one run.
> # A task fails the first batch on both attempts and succeeds on the second
> batch. The failure has to be tied to the first batch; a Recon RocksDB I/O
> error closes the DB, so the second batch would fail too. One such failure is
> HDDS-17048: {{NSSummaryTask}} fails with a NullPointerException, on both
> attempts, on a key event whose bucket was deleted later in the same delta,
> while a later batch about other buckets succeeds (code reading).
> # Recon restarts after the event thread has processed the buffered batches
> and before the sync fetches any new OM update. A non empty fetch after the
> failure moves the {{OmDeltaRequest}} cursor past the task cursors, because
> the batch is refused or drained, and the startup check then rebuilds. On a
> cluster with ongoing writes this leaves about one sync interval (5s by
> default). Without new writes, it lasts until the queued reinitialization has
> completed.
> h3. Impact
> The changes of the failed batch are missing from that task's derived data for
> good: for NSSummary the namespace summary and du APIs report too few files
> and bytes. Later syncs and restarts do not notice it; a manual rebuild
> repairs it. Without a restart the reinitialization repairs it as designed.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC7FailedBatchLostAcrossRestart.java] ({{hadoop-ozone/recon}}) uses
> a real OM RocksDB that serves its write ahead log through a mocked
> {{getDBUpdates}}, and the real {{OzoneManagerServiceProviderImpl}},
> {{ReconTaskControllerImpl}}, {{NSSummaryTask}} and task status table.
> {{NSSummaryTask.process}} returns a failed result for one batch on both
> attempts, and its first attempt is held until the next batch is buffered. The
> restart stops the provider and controller and starts new ones on the same
> databases. Output before the restart: {{tasksFailed}} true, NSSummaryTask
> cursor 5, OmDeltaRequest cursor 5, OM has 3 files under the bucket and
> NSSummary 2. After the restart and a sync NSSummary still reports 2; a
> rebuild gives 3. A passing test means the defect is present.
> h3. Patch
> [^MC-7-skip-batches-behind-failed-batch-until-reinit.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> {{processOMUpdateBatch}} does not run the tasks while {{tasksFailed}} is set.
> In that state {{consumeOMEvents}} already refuses new batches, and queueing
> the reinitialization drains the buffer, clears the flag and takes the
> checkpoint the rebuild uses, which contains every batch applied to the Recon
> OM DB, so the skipped batches would be discarded by it anyway. Not processing
> them keeps the failed task's cursor behind, and the existing startup check
> rebuilds after a restart. Without a restart nothing changes, except that
> until the reinitialization is queued (normally the next sync iteration) the
> tasks that did not fail also stop applying buffered batches, which the
> reinitialization rebuilds.
> Covered by {{testBatchesBehindFailedBatchDoNotAdvanceTaskCursor}} in the
> existing {{TestReconTaskControllerImpl}}, which fails without the change
> ({{expected: 0L but was: 200L}}). With the change the Recon tasks suites and
> {{TestOzoneManagerServiceProviderImpl}} (155 tests) pass and checkstyle is
> clean. With the patch the reproduction fails at its undercount assertion: the
> NSSummaryTask cursor stays at 3, the restart rebuilds the tasks and NSSummary
> reports 3.
> A related case, a task whose {{process}} throws instead of returning a failed
> result and so never sets {{tasksFailed}}, is HDDS-17040.
> Found by TLA+ model checking and code review of the Recon namespace summary
> and task sync paths under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-5146 (a failure between the Recon OM DB update and the task
> hand off loses the batch) and HDDS-16463 (sync the Recon DB before the cursor
> advances). The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
