[
https://issues.apache.org/jira/browse/HDDS-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17732:
------------------------------
Attachment: TestBugCR13CloseAfterRocksDbFailure.java
> SnapDiffJobStore.close stops at the first RocksDB error and leaves the job's
> temporary column families in the snapshot diff DB until OM restart
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17732
> URL: https://issues.apache.org/jira/browse/HDDS-17732
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-13-best-effort-snap-diff-job-store-close.patch,
> TestBugCR13CloseAfterRocksDbFailure.java
>
>
> h3. Mechanism
> {{SnapDiffJobStore.close}} runs its steps in sequence without try/finally:
> {{flushWrites}}, close the write batch and write options, drop every tracked
> temporary column family through {{dropAndClose}}, close the column family
> options. Any exception ends it:
> * A failed {{flushWrites}} leaves the batch as it was, so after a stage flush
> failed, the flush in {{close}} repeats the same write, fails again, and no
> column family is dropped and no native object is closed.
> * {{dropAndClose}} removes the handle from the tracking list before
> {{SnapshotUtils.dropColumnFamilyHandle}}, which turns a {{RocksDBException}}
> into a {{RuntimeException}}. That handle is then never closed and the
> remaining column families are never dropped.
> Nothing in a running OM drops them later; only
> {{OmSnapshotManager.dropUnknownColumnFamilies}} at OM startup does. A dropped
> column family keeps its files on disk while a handle to it is open, so the
> unclosed handle also keeps that space until the process exits.
> h3. Trigger
> A RocksDB write or column family drop fails on the snapshot diff DB while an
> optimized snapshot diff job runs, which in practice means a full or failing
> disk under {{ozone.om.snapshot.diff.db.dir}}. Only the optimized snapshot
> diff path ({{ozone.om.snapshot.diff.optimized.enabled}}, default false) uses
> {{SnapDiffJobStore}}, and the class is not in any release yet. The job store
> is closed by the try with resources in
> {{SnapshotDiffManager.generateSnapshotDiffReport}}. Each job creates nine
> temporary column families (seven for OBS buckets), plus up to four more when
> its in memory sets spill, all sized by the diff, so the job that fills the
> volume is the one whose data stays.
> If the error happens before the report is complete, the job is marked FAILED
> as usual. The store is closed after {{updateJobStatusToDone}}, and some
> families are still open then (on an OBS bucket the four classified families
> and the dependency node family). If one of those drops fails, the job stays
> DONE, which is correct because its report is complete; the existing caller
> then calls {{updateJobStatusToFailed}}, which throws an
> {{IllegalStateException}} for the DONE job before the close error is logged.
> That caller behaviour is not changed here.
> h3. Impact
> * The temporary column families of the failed job, with their data, stay in
> the snapshot diff DB until the OM restarts. When the snapshot diff DB is on
> its own volume, the OM keeps running and the space is not released, so later
> diff jobs can keep failing for lack of space.
> * One {{ColumnFamilyHandle}} (drop path), or the write batch, write options,
> column family options and all handles (flush path) are never closed (native
> memory).
> * Job state is not corrupted: a FAILED job can be resubmitted under a new job
> ID, and a DONE job keeps its complete report.
> The legacy path, {{SnapshotDiffManager.generateLegacySnapshotDiffReport}},
> drops its three temporary column families with the same sequential pattern,
> which is in every release from 1.4.0 to 2.2.2; it is handled separately.
> h3. Reproduction
> [^TestBugCR13CloseAfterRocksDbFailure.java] ({{hadoop-ozone/ozone-manager}}):
> a real RocksDB and the store's public calls; the RocksDB error is injected
> with a Mockito spy on the {{RocksDB}} object (write fails; or the first drop
> is applied and then fails, as a full disk does when RocksDB cannot persist
> its OPTIONS file). On d3b75c75bf9ad475551202477c3a21a3f4b968b7 both tests
> fail: after the failed flush all nine temporary column families of the job
> remain, after the failed drop the other eight remain and the handle of the
> failed drop is still open. With the patch both pass. A real out of space
> error was reproduced only by the original investigation (on a 16 MB disk
> image), not with the attached test.
> h3. Patch
> [^CR-13-best-effort-snap-diff-job-store-close.patch], against
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
> bbc8505e00d.
> {{close}} now attempts every step even if an earlier one fails: the flush
> failure no longer skips the rest, every tracked column family is attempted
> (the first failure is thrown with later ones added as suppressed), and the
> write batch, write options and column family options are closed in finally
> blocks. {{dropAndClose}} closes the handle in a finally block. A store that
> closes without error behaves as before.
> Covered by two tests in the existing {{TestMergeJoinSnapDiffWriter}}, which
> already drives {{SnapDiffJobStore}} on a real RocksDB: a failed flush and a
> failed drop, each asserting that {{close}} reports the error and no column
> family of the job is left. Without the change both fail on the column family
> check: the flush test finds all nine families of the job, the drop test the
> other eight. With it {{TestMergeJoinSnapDiffWriter}},
> {{TestDagDiffSequentialReader}}, {{TestFullDiffSequentialReader}},
> {{TestSnapDiffPathResolver}}, {{TestSnapDiffDependencyGraph}} and
> {{TestSnapshotDiffManager}} (139 tests) pass and checkstyle is clean.
> Found by code review of the OM optimized snapshot diff paths, as part of the
> TLA+ verification effort under HDDS-15926, on commit
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
