ArafatKhan2198 opened a new pull request, #11560: URL: https://github.com/apache/ozone/pull/11560
## What changes were proposed in this pull request? When Recon does a full refresh of its OM DB copy, it first deletes the folder of the current copy (to save disk space) and then downloads a new one. If RocksDB cannot open the new copy, for example because an SST file is truncated or the disk is full, `ReconOmMetadataManagerImpl.initializeNewRdbStore` catches the error, only logs it, and returns normally. As a result: - Recon logs `Successfully updated Recon OM DB with new snapshot.` and counts the refresh as a success in its metrics, even though nothing was installed. - Recon keeps using its previous OM DB, whose folder has already been deleted. - Because Recon believes the refresh worked, it queues a task reinitialization. That tries to checkpoint the previous DB, fails with `NotFound … Failed to createCheckpoint`, and logs a message that blames the previous DB (`the on-disk OM DB may be incomplete or corrupt`). This is the only failure that gets reported, and it points at the wrong cause. - While the reinitialization keeps failing, the Recon tasks stop receiving delta updates. After 6 failed attempts (about 30 seconds) Recon falls back to a full fetch, so it does recover, but only by accident. OM is not affected. A truncated SST can come from a retry after a failed untar (HDDS-16880). ### **Fix** All changes are in `ReconOmMetadataManagerImpl`: 1. `initializeNewRdbStore` no longer catches the exception, so a failed open is reported to the caller. 2. `start()` now has that catch, so startup behaves as before: if the last known snapshot cannot be opened, Recon starts without an OM DB and fetches a new one. 3. `updateOmDB` deletes the snapshot folder that failed to open and rethrows the error. The previous store stays open. Without the deletion, a restart could load that leftover snapshot, which would be behind the Recon tasks, so delta updates would reach the tasks a second time. `updateReconOmDBWithNewSnapshot` already had a catch for this case that never ran. It now logs `Unable to refresh Recon OM DB Snapshot.` with the real RocksDB error, counts a failure, and returns `false`. `executeFullSnapshot` then marks the run as failed and does not queue the reinitialization. This is exactly how a failed download is already handled, so no new code path is added. ### **Note for reviewers** Today, the failing reinitialization forces a new full fetch within about 6 sync cycles. With this change, if the full fetch was triggered by a temporary delta failure, Recon keeps applying delta updates to the previous DB until the next delta failure, the next failed flush of that DB, or a restart. During that time health shows `GET_OM_DB_SNAPSHOT_FAILED`. This is the same state Recon is already in after a failed download. The underlying issue, that the previous DB folder is deleted before the new snapshot is confirmed, is not changed here. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16882 ## How was this patch tested? Three unit tests were added to existing test classes. Each one uses a real RocksDB snapshot that cannot be opened: - `TestReconOmMetadataManagerImpl#testUnreadableSnapshot`: `updateOmDB` throws, the broken folder is deleted, the previous store stays open, and `start()` still skips an unreadable last known snapshot. The broken snapshot has a `CURRENT` file that points to a missing MANIFEST. - `TestOzoneManagerServiceProviderImpl#testUpdateReconOmDBWithUnreadableSnapshot`: `updateReconOmDBWithNewSnapshot()` returns `false`, Recon keeps the previous store, and no snapshot folder is left on disk. - `TestOzoneManagerServiceProviderImpl#testSyncDataFromOMWithUnreadableSnapshot`: a full sync through `syncDataFromOM()` with a truncated SST. Delta updates are rejected, so the sync falls back to a full fetch. The test checks that the failure is counted, health shows `GET_OM_DB_SNAPSHOT_FAILED`, the task reinitialization is not queued, and the previous store is kept. All three tests fail without the fix and pass with it. The reproduction test attached to the Jira also passes with the fix. Checkstyle is clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
