[
https://issues.apache.org/jira/browse/HDDS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16864:
------------------------------
Attachment: delete-checkpoint-tarball-when-untar-fails.patch
> OM follower leaves the downloaded checkpoint tarball in the ratis snapshot
> dir when untar fails
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-16864
> URL: https://issues.apache.org/jira/browse/HDDS-16864
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: TestUntarFailureLeaksTarball.java,
> delete-checkpoint-tarball-when-untar-fails.patch
>
>
> h3. Mechanism
> {{RDBSnapshotProvider.untarContentsOfTarball}} calls {{FileUtil.unTar}} and
> deletes the downloaded tarball only after it returns. If untar throws, the
> {{om.db-<leader>-<millis>.tar}} file stays in the OM ratis snapshot dir. No
> other path removes it: {{OmRatisSnapshotProvider.downloadSnapshot}} deletes
> the file only when the HTTP download itself fails,
> {{OzoneManager.cleanupCheckpoint}} deletes only the candidate dir,
> {{RDBSnapshotProvider.init}} (OM startup, leader change,
> {{notifySnapshotInstalled}}) empties only the candidate dir, and a later
> successful install deletes only its own tarball. Every retry uses a new file
> name, so each failed attempt leaves one more file. The code is unchanged
> since HDDS-7995 (1.4.0) and is present in master.
> Related: HDDS-12820, whose description assumes {{init()}} removes partial
> downloads; it removes only the candidate dir.
> h3. Trigger
> # A follower OM downloads a checkpoint part from the leader.
> # The download completes but the tarball cannot be extracted. Two ways, both
> argued from the code and not run end to end:
> {{OMDBCheckpointServletInodeBasedXfer.processMetadataSnapshotRequest}}
> catches an {{IOException}} from {{OMDBArchiver.writeToArchive}} after the
> response is committed, logs it and returns normally (its {{setStatus(500)}}
> has no effect at that point), so the follower gets HTTP 200 with a truncated
> tar; or extraction runs out of space on the snapshot volume.
> # {{unTar}} throws, {{installSnapshotFromLeader}} cleans the candidate dir
> and returns null, and the leader retries the install.
> h3. Impact
> * Each failed untar permanently consumes the size of the failing part on the
> OM ratis snapshot volume. Restart does not remove it.
> * Since HDDS-15171 (2.3.0, and master), on a tight disk the leftover file
> makes the next attempts fail the bootstrap disk space check before
> downloading, so the follower cannot install a checkpoint until an operator
> deletes the file by hand. The error points at disk space but not at the
> leftover tarball. Earlier releases have no such check, so the next download
> runs into the full disk while writing instead (argued from the code, not run).
> * If the disk check still passes, every failed retry adds another tarball.
> * Recon uses the same {{RDBSnapshotProvider}} code (argued, not run for
> Recon).
> h3. Reproduction
> Deterministic, unmodified source. [^TestUntarFailureLeaksTarball.java]
> ({{hadoop-ozone/ozone-manager}}) calls
> {{OmRatisSnapshotProvider.downloadDBSnapshotFromLeader}} over its real
> download path with only {{HttpURLConnection}} mocked, as in
> {{TestOmRatisSnapshotProvider}}: the leader answers HTTP 200 with a tar that
> ends in the middle of an entry. After each of three failed attempts the test
> performs the follower's own cleanup (delete the candidate dir, then
> {{init()}}). The test asserts that no tarball is left, so it fails while the
> defect is present: on the unfixed code three {{om.db-om1-*.tar}} files
> remain, one per attempt. It passes with the patch.
> h3. Patch
> [^delete-checkpoint-tarball-when-untar-fails.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> The existing {{FileUtil.fullyDelete(snapshot)}} moves into a {{finally}}
> around {{FileUtil.unTar}}, so the tarball is deleted whether or not
> extraction succeeds and the exception still propagates. The success path is
> unchanged. Tarballs left by earlier failures or by a process crash between
> download and untar are not removed by the patch.
> Covered by {{testDownloadedTarballDeletedWhenUntarFails}} in the existing
> {{TestRDBSnapshotProvider}}: a truncated first part must fail without leaving
> a tarball, and the retry with a complete part must return the checkpoint
> without leaving one. Without the change it fails with the tarball still
> present. With it {{TestRDBSnapshotProvider}}, {{TestRDBStore}},
> {{TestOmRatisSnapshotProvider}} and {{TestReconRDBSnapshotProvider}} (34
> tests) pass and checkstyle is clean.
> Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
