Hernan Gelaf-Romer created HBASE-29448: ------------------------------------------
Summary: Modern backup failures can cause backup system to lock up Key: HBASE-29448 URL: https://issues.apache.org/jira/browse/HBASE-29448 Project: HBase Issue Type: Bug Components: backup&restore Reporter: Hernan Gelaf-Romer Assignee: Hernan Gelaf-Romer Prior to any backup operation, a snapshot of the backup:system table will be taken. If the backup operation fails, we attempt to restore the backup system table from the snapshot. This is done as a way to revert to a previously successful state. In order to restore, we run different [procedures|https://github.com/apache/hbase/blob/8ddf925daac7af48a5b624c6192bd2cdc45f7955/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java#L1400] in sequence. The problem is that these procedures aren't guaranteed to run back to back without interference; there are no atomicity guarantees that prevent other {{backup:system}} operations from being interleaved here. This can cause the backup system to get into a stuck state, Where it is unable to proceed until it receives manual intervention. For example, we may fail a backup for whatever reason, and go to restore from the snapshot. However, an EnableTableProcedure might sneak through and run between the DisableTableProcedure and the restore snapshot. A concrete example is the BackupHFileCleaner running and enabling the {{backup:system}} table when it creates a [BackupSystemTable|https://github.com/apache/hbase/blob/8ddf925daac7af48a5b624c6192bd2cdc45f7955/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/BackupHFileCleaner.java#L69] object {code:java} 2025-07-06T11:39:23,061 [hfile_cleaner-dir-scan-pool-285] INFO org.apache.hadoop.hbase.client.HBaseAdmin: Started enable of backup:system{code} Now, subsequent backups cannot run, b/c they cannot snapshot the table due to an existing snapshot that wasn't correctly cleaned up {code:java} 2025-07-06 11:41:48.004 [pool-115-thread-1] ERROR o.a.h.h.b.impl.TableBackupClient - Unexpected Exception : org.apache.hadoop.hbase.snapshot.SnapshotExistsException: Snapshot 'snapshot_backup_system' already stored on the filesystem. at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.sanityCheckBeforeSnapshot(SnapshotManager.java:804) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.access$000(SnapshotManager.java:127) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager$1.run(SnapshotManager.java:725) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.submitSnapshotProcedure(SnapshotManager.java:722) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:713) at org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1723) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:443) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:105) at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:85) {code} I can think of two possible solutions, but wanted to poll the community to see if anybody had other ideas # Create a specific procedure that restores the {{backup:system}} table from a snapshot that takes a table-level lock against the backup system table. This would ensure the restore process runs without interference # Implement a different checkpointing system, that doesn't require snapshotting the backup system table I'm err'ing towards the first option, as it would be easier to implement, and wouldn't require a massive re-work -- This message was sent by Atlassian Jira (v8.20.10#820010)