SaketaChalamchala opened a new pull request, #8906: URL: https://github.com/apache/ozone/pull/8906
## What changes were proposed in this pull request? Log all the asynchronous Garbage Collection logs to a separate rolling log file `om-gc.log` on OMs to improve debuggability of deletion services. The size threshold for rolling the log is 256MB and the max no. of om-gc.log files retained is 10. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-8203 ## How was this patch tested? Manually tested in docker environment. ``` sh-4.2$ tail -f /var/log/hadoop/om-gc.log 2025-08-06T01:45:58,524 [om1-OMStateMachineApplyTransactionThread - 0] INFO OMGCLogger: Successfully executed snapshotSetPropertyRequest: {[snapshotKey: "/vol1/bucket1/snapshot-1" deepCleanedDeletedDir: true ]}. 2025-08-06T01:46:58,470 [om1-DirectoryDeletingService#1] INFO OMGCLogger: Snapshot 18fea150-674c-48cf-ac80-b2527fb1440d has already been deep cleaned directory. Skipping the snapshot in this iteration. 2025-08-06T01:46:58,473 [om1-SnapshotDeletingService#0] INFO OMGCLogger: Started Snapshot Deletion Processing for snapshot : /vol1/bucket1/snapshot-1 2025-08-06T01:46:58,473 [om1-SnapshotDeletingService#0] INFO OMGCLogger: Snapshot: /vol1/bucket1/snapshot-1 entries will be moved to next active snapshot: /vol1/bucket1/snapshot-2 2025-08-06T01:46:58,530 [om1-KeyDeletingService#4] INFO OMGCLogger: Send 0 key(s) to SCM, first 0 keys: [] 2025-08-06T01:46:58,543 [om1-KeyDeletingService#4] INFO OMGCLogger: 0 BlockGroup deletion are acked by SCM in 13 ms 2025-08-06T01:46:58,560 [om1-KeyDeletingService#4] INFO OMGCLogger: Blocks for (0,true) (out of 0) keys are deleted from DB in 16 ms. Limit per task is 50000. 2025-08-06T01:46:58,561 [pool-35-thread-1] ERROR OMGCLogger: Error while running delete directories for store : /vol1/bucket1/snapshot-2 and files background task. Will retry at next run. org.apache.hadoop.ozone.om.exceptions.OMException: Unable to load snapshot. Snapshot with table key '/vol1/bucket1/snapshot-1' is no longer active at org.apache.hadoop.ozone.om.snapshot.SnapshotUtils.checkSnapshotActive(SnapshotUtils.java:150) at org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshot(OmSnapshotManager.java:725) at org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshot(OmSnapshotManager.java:717) at org.apache.hadoop.ozone.om.OmSnapshotManager.getActiveSnapshot(OmSnapshotManager.java:694) at org.apache.hadoop.ozone.om.snapshot.filter.ReclaimableFilter.initializePreviousSnapshotsFromChain(ReclaimableFilter.java:164) at org.apache.hadoop.ozone.om.snapshot.filter.ReclaimableFilter.apply(ReclaimableFilter.java:199) at org.apache.hadoop.ozone.om.service.DirectoryDeletingService$DirDeletingTask.processDeletedDirectories(DirectoryDeletingService.java:618) at org.apache.hadoop.ozone.om.service.DirectoryDeletingService$DirDeletingTask.lambda$0(DirectoryDeletingService.java:545) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2025-08-06T01:46:58,565 [om1-OMStateMachineApplyTransactionThread - 0] INFO OMGCLogger: Successfully executed snapshotPurgeRequest: {snapshotDBKeys: "/vol1/bucket1/snapshot-1" } along with updating snapshots:{/vol1/bucket1/snapshot-2=SnapshotInfo{snapshotId: '3f0abb78-1351-470d-bb9e-be167b0319e5', name: 'snapshot-2', volumeName: 'vol1', bucketName: 'bucket1', snapshotStatus: 'SNAPSHOT_ACTIVE', creationTime: '1754444778673', deletionTime: '-1', pathPreviousSnapshotId: 'null', globalPreviousSnapshotId: 'null', snapshotPath: 'vol1/bucket1', checkpointDir: '-3f0abb78-1351-470d-bb9e-be167b0319e5', dbTxSequenceNumber: '138', deepClean: 'false', sstFiltered: 'false', referencedSize: '54461', referencedReplicatedSize: '54461', exclusiveSize: '0', exclusiveReplicatedSize: '0', exclusiveSizeDeltaFromDirDeepCleaning: '0', exclusiveReplicatedSizeDeltaFromDirDeepCleaning: '0', deepCleanedDeletedDir: 'false', lastTransactionInfo: '<ByteString@3dc0ad60 size=4>'}}. 2025-08-06T01:47:58,482 [pool-35-thread-1] INFO OMGCLogger: Number of dirs deleted: 1, Number of sub-dir deleted: 0, Number of sub-files moved: 1 to DeletedTable, Number of sub-dirs moved 0 to DeletedDirectoryTable, iteration elapsed: 10ms, totalRunCount: 7 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
