[
https://issues.apache.org/jira/browse/HDDS-16880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16880:
------------------------------
Attachment: TestBugCR16TruncatedSstReusedOnReconRetry.java
> Recon reuses the truncated file left by a failed OM DB checkpoint untar, so
> its retry from the same OM leader builds a DB with a truncated SST
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16880
> URL: https://issues.apache.org/jira/browse/HDDS-16880
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-16-recon-clear-candidate-dir-after-failed-fetch.patch,
> TestBugCR16TruncatedSstReusedOnReconRetry.java
>
>
> h3. Mechanism
> When Recon fetches the OM DB,
> {{RDBSnapshotProvider.downloadDBSnapshotFromLeader}} untars the downloaded
> part into {{om.snapshot.db.candidate}} with {{FileUtil.unTar}}, which runs
> GNU tar on Linux. If the archive ends in the middle of an entry, or
> extraction runs out of space, tar exits with code 2 and leaves the partly
> written file in the candidate dir. Since HDDS-15766,
> {{OzoneManagerServiceProviderImpl.getOzoneManagerDBSnapshot}} deliberately
> keeps the candidate dir after a failure, and
> {{RDBSnapshotProvider.checkLeaderConsistency}} keeps it for the same leader.
> On the next fetch {{ReconRDBSnapshotProvider.downloadSnapshot}} sends every
> file name in the candidate dir ({{HAUtils.getExistingFiles}}, names only) as
> already received. {{OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir}}
> skips files whose {{<inode>-<mtime>}} id is in that list, which still matches
> as long as the SST is live on the leader, but still lists them in the
> hardLinkFile. {{InodeMetadataRocksDBCheckpoint}} then links the truncated
> file as {{om.db/<n>.sst}}, and
> {{ReconRDBSnapshotProvider.getCheckpointFromUntarredDb}} promotes it to
> {{om.snapshot.db_<ts>}}. Nothing on the Recon side compares file sizes or
> checksums.
> The comment that keeps the dir says this "matches how the OM follower handles
> a failed download", but it does not:
> {{OzoneManager.installSnapshotFromLeader}} deletes the candidate dir after
> any failed download ({{cleanupCheckpoint(null)}}). The OM follower and
> {{ozone repair om}} are not affected.
> h3. Trigger
> # Recon fetches a full OM DB snapshot (first start, or fallback after delta
> updates fail).
> # The untar of the response fails part way. One way is the leader side:
> {{OMDBCheckpointServletInodeBasedXfer.processMetadataSnapshotRequest}}
> catches an {{IOException}} from {{OMDBArchiver.writeToArchive}} after the
> response is committed, logs it and returns normally, so Recon gets HTTP 200
> and a tar cut in the middle of an entry (see HDDS-16864). A dropped
> connection does not trigger this, because
> {{ReconRDBSnapshotProvider.downloadSnapshot}} deletes the partial tar on a
> read error. The other way, more likely on Recon, is the Recon OM snapshot
> volume filling up during extraction: the tar stays on the volume until the
> untar succeeds, so extraction needs about twice the checkpoint size. Neither
> was run end to end; the reproduction uses a response cut in the middle of an
> SST entry.
> # The next fetch goes to the same OM leader while that SST is still live
> there.
> h3. Impact
> * The retry builds an OM DB with a truncated SST. RocksDB refuses to open it
> ({{Corruption: Sst file size mismatch between expected 24743 and file system
> 12288 sstable: .../om.snapshot.db_<ts>/000010.sst}}), so Recon does not read
> the truncated data.
> * {{ReconOmMetadataManagerImpl.initializeNewRdbStore}} only logs the open
> failure. {{updateReconOmDBWithNewSnapshot}} returns true, logs "Successfully
> updated Recon OM DB with new snapshot." and keeps the previous OM DB handle
> (none on first start). {{getOzoneManagerDBSnapshot}} already deleted that
> directory before the first fetch, so a later flush on it fails. In the
> scheduled sync the run is then marked failed and
> {{GET_OM_DB_SNAPSHOT_FAILED}} is set again, but only because the task
> reinitialization that follows cannot checkpoint that deleted DB (or the
> missing one on first start).
> * The retry that should have repaired Recon is wasted, and the corrupt
> {{om.snapshot.db_<ts>}} stays on disk. Recon recovers at the next full fetch,
> which deletes that dir and starts from an empty candidate dir, or after a
> restart (the next full fetch was run; the restart was argued, not run).
> * No effect on OM. Affects Version: 2.3.0. The affected code is only on
> master (HDDS-15766 is in no release; 2.2.1 and earlier extract the OM DB into
> a fresh dir without an exclusion list).
> h3. Reproduction
> Deterministic, unmodified source.
> [^TestBugCR16TruncatedSstReusedOnReconRetry.java] ({{hadoop-ozone/recon}})
> runs Recon's own
> {{OzoneManagerServiceProviderImpl.updateReconOmDBWithNewSnapshot}} and
> {{ReconRDBSnapshotProvider}}, with the real request form data and shell
> untar, against a real OM RocksDB. Only the HTTP transport, the OM RPC client
> and Recon's task wiring are mocked; the snapshot provider, the form data, the
> shell untar, the hard link install and the RocksDB open are the real code.
> The leader side is a stand-in for {{OMDBCheckpointServletInodeBasedXfer}}
> with {{includeSnapshotData=false}}: a fresh checkpoint per request, entries
> named by inode and mtime, files in the request's exclusion list skipped, then
> the hardLinkFile and the completion flag. The first response ends in the
> middle of the largest SST. The test asserts the correct behaviour, so it
> fails while the defect is present. On the unfixed code the second request
> excludes the truncated file, and the promoted DB has {{000010.sst}} at 12288
> bytes instead of 24743. The same run logs the RocksDB size mismatch, then
> "Successfully updated Recon OM DB with new snapshot.". With the patch it
> passes.
> h3. Patch
> [^CR-16-recon-clear-candidate-dir-after-failed-fetch.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> 6818707698c.
> After a failed fetch, {{getOzoneManagerDBSnapshot}} now calls
> {{reconSnapshotProvider.init()}}, which empties the candidate dir, as the OM
> follower does. A {{RuntimeException}} from {{init()}} (for example
> {{HddsUtils.createDir}} failing) is logged, so the method still returns null.
> This puts back the reset that an earlier revision of HDDS-15766
> (apache/ozone#10696) had in the same place, with the same guard. It was
> dropped on review so that a future batched transfer could resume the parts
> already received. Keeping the dir is only safe if every file in it is
> complete, and a failed untar breaks that. Resuming a batched transfer would
> need each part extracted atomically (to a staging dir, moved in only on
> success) or verified (HDDS-14651). The OM follower, which does use batches,
> also wipes the dir after a failure. Today Recon requests the checkpoint with
> {{includeSnapshotData=false}}, which the leader sends in a single part, so
> after a failure the dir only holds the output of a failed part and nothing
> that is safe to resume is lost. This holds for the v1 endpoint too
> ({{ozone.om.db.checkpoint.use.inode.based.transfer=false}}), where Recon
> sends SST paths and the leader also excludes by name (argued, not run). A
> leader side fix would need sizes or checksums on the wire (see HDDS-14651).
> That {{initializeNewRdbStore}} reports a failed open as success is not
> changed here; it is filed as a separate issue.
> Covered by {{testPartiallyUntarredFileNotReportedOnRetry}} in the existing
> {{TestOzoneManagerServiceProviderImpl}}: after an untar failure on a
> truncated SST, the next request must list no files and the candidate dir must
> be empty. Without the change it fails with {{[[], ["000010.sst"]]}}.
> {{testGetOzoneManagerDBSnapshotReturnsNullWhenCandidateDirResetFails}} covers
> the guard. With the change, {{TestOzoneManagerServiceProviderImpl}},
> {{TestReconRDBSnapshotProvider}} and {{TestTriggerDBSyncEndpoint}} (23 tests)
> and the mini-cluster {{TestReconWithOzoneManagerHA}} pass, and checkstyle is
> clean.
> Related: HDDS-15766 (introduced the retention), HDDS-16864 (the same failed
> untar also leaves the downloaded tarball behind), HDDS-14651 (manifest and
> checksum validation would detect the short file).
> Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
