ArafatKhan2198 opened a new pull request, #11563: URL: https://github.com/apache/ozone/pull/11563
## **What changes were proposed in this pull request?** ### **Problem** Recon fetches the OM DB by downloading a tar file from the OM leader and unpacking it into a candidate dir (`om.snapshot.db.candidate`). If the unpacking fails halfway, for example because the disk fills up or the leader sends a tar that is cut short (HDDS-16864), tar leaves a partially written SST file in the candidate dir. Since HDDS-15766, Recon keeps the candidate dir after a failed fetch, so the retry can resume: it tells the leader which files it already has, and the leader skips them. That list is built from file names only, so the truncated SST is on it too, and the leader does not send it again. Recon then builds the new OM DB with the truncated SST, and RocksDB refuses to open it. The retry is wasted, and a corrupt `om.snapshot.db_<ts>` is left on disk. ### **Fix** The fix keeps the resume behaviour of HDDS-15766 and makes sure only complete files are ever kept in the candidate dir. - `ReconRDBSnapshotProvider` now overrides `untarContentsOfTarball`. It extracts each file to `<name>.tmp` and renames it to its real name only after all its bytes are written. If the untar fails partway, the `.tmp` file is deleted. It uses the existing `Archiver` helpers, which Recon already uses to unpack the SCM DB. - On the retry, the candidate dir holds only fully extracted files. The leader skips those, and sends the file that failed in full along with the remaining files. - `RDBSnapshotProvider.untarContentsOfTarball` is changed from `private` to `protected` so that Recon can override it. The OM follower is unchanged. - The comment in `OzoneManagerServiceProviderImpl.getOzoneManagerDBSnapshot` said keeping the dir "matches how the OM follower handles a failed download". That is wrong: `OzoneManager.installSnapshotFromLeader` deletes the candidate dir after a failed download. The comment now says that the dir only holds fully extracted files. Only master is affected: HDDS-15766 is not in any release. OM is not affected. Related: HDDS-15766 (introduced resuming from the candidate dir), HDDS-16882 (Recon reports the failed open of such a DB as success), HDDS-16864 (a failed untar also leaves the downloaded tarball behind, which is not changed here). ## **What is the link to the Apache JIRA** https://issues.apache.org/jira/browse/HDDS-16880 ## **How was this patch tested?** `testPartiallyUntarredFileNotReportedOnRetry` was added to `TestOzoneManagerServiceProviderImpl`. The download returns a tar that holds one complete SST (`000009.sst`) and ends in the middle of a second one (`000010.sst`), so the untar fails. The test checks that the retry lists only `000009.sst` as already received, and that the candidate dir holds only that file, at its full size. - On the original code it fails with `[[], ["000010.sst", "000009.sst"]]`: the truncated file is listed. - If the candidate dir is wiped instead, it fails with `[[], []]`: nothing is resumed. `TestOzoneManagerServiceProviderImpl`, `TestReconRDBSnapshotProvider`, `TestTriggerDBSyncEndpoint` and `TestRDBSnapshotProvider` pass (24 tests), and checkstyle is clean. The reproduction test attached to the Jira fails without the fix and passes with it: the retry builds a complete OM DB that Recon opens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
