Hernan Gelaf-Romer created HBASE-29448:
------------------------------------------
Summary: Modern backup failures can cause backup system to lock up
Key: HBASE-29448
URL: https://issues.apache.org/jira/browse/HBASE-29448
Project: HBase
Issue Type: Bug
Components: backup&restore
Reporter: Hernan Gelaf-Romer
Assignee: Hernan Gelaf-Romer
Prior to any backup operation, a snapshot of the backup:system table will be
taken. If the backup operation fails, we attempt to restore the backup system
table from the snapshot. This is done as a way to revert to a previously
successful state.
In order to restore, we run different
[procedures|https://github.com/apache/hbase/blob/8ddf925daac7af48a5b624c6192bd2cdc45f7955/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java#L1400]
in sequence. The problem is that these procedures aren't guaranteed to run
back to back without interference; there are no atomicity guarantees that
prevent other {{backup:system}} operations from being interleaved here. This
can cause the backup system to get into a stuck state, Where it is unable to
proceed until it receives manual intervention.
For example, we may fail a backup for whatever reason, and go to restore from
the snapshot. However, an EnableTableProcedure might sneak through and run
between the DisableTableProcedure and the restore snapshot. A concrete example
is the BackupHFileCleaner running and enabling the {{backup:system}} table when
it creates a
[BackupSystemTable|https://github.com/apache/hbase/blob/8ddf925daac7af48a5b624c6192bd2cdc45f7955/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/BackupHFileCleaner.java#L69]
object
{code:java}
2025-07-06T11:39:23,061 [hfile_cleaner-dir-scan-pool-285] INFO
org.apache.hadoop.hbase.client.HBaseAdmin: Started enable of backup:system{code}
Now, subsequent backups cannot run, b/c they cannot snapshot the table due to
an existing snapshot that wasn't correctly cleaned up
{code:java}
2025-07-06 11:41:48.004 [pool-115-thread-1] ERROR
o.a.h.h.b.impl.TableBackupClient - Unexpected Exception :
org.apache.hadoop.hbase.snapshot.SnapshotExistsException: Snapshot
'snapshot_backup_system' already stored on the filesystem. at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.sanityCheckBeforeSnapshot(SnapshotManager.java:804)
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.access$000(SnapshotManager.java:127)
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager$1.run(SnapshotManager.java:725)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132)
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.submitSnapshotProcedure(SnapshotManager.java:722)
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:713)
at
org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1723)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:443) at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:105) at
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:85)
{code}
I can think of two possible solutions, but wanted to poll the community to see
if anybody had other ideas # Create a specific procedure that restores the
{{backup:system}} table from a snapshot that takes a table-level lock against
the backup system table. This would ensure the restore process runs without
interference
# Implement a different checkpointing system, that doesn't require
snapshotting the backup system table
I'm err'ing towards the first option, as it would be easier to implement, and
wouldn't require a massive re-work
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
