SpookyJumpyBeans opened a new pull request, #8921: URL: https://github.com/apache/hbase/pull/8921
https://issues.apache.org/jira/browse/HBASE-18886 A backup takes the backup exclusive lock (`activesession:` in the backup system table) in `TableBackupClient.init`, and writes its RUNNING record later in `beginBackup`. If it fails in between, the lock stays `yes` with no record, and `hbase backup repair` returns 0 without releasing it, so every later backup create, delete and merge fails to take the lock. Two ways to get there: - A lost response to taking the lock: `checkAndMutate` is not idempotent under client retries (HBASE-15588), so the retried conditional put finds `yes` and fails, and the backup waits for its own lock until `hbase.backup.exclusive.op.timeout.seconds` and fails. `BackupAdminImpl.deleteBackups` returns -1 holding the lock on the same lost response. - A client that dies before recording the operation, as in the original report. ### Changes - `BackupSystemTable.isBackupExclusiveOperationActive()`: reads whether the lock is held. - `RepairCommand`: when there is no RUNNING backup, after repairing failed deletes and merges, release the lock if it is still held, and say so: ```java private void releaseExclusiveOperationLockIfHeld(BackupSystemTable sysTable) throws IOException { if (sysTable.isBackupExclusiveOperationActive()) { System.out.println("REPAIR status: the backup exclusive lock is held, but no backup," + " delete or merge operation is recorded as running; releasing it. Run repair only" + " when no backup, delete or merge command is running."); sysTable.finishBackupExclusiveOperation(); } } ``` Repair already assumes no backup operation is running: when it finds a RUNNING backup it marks it failed and releases the lock without checking whether it is alive, and it deletes the backup system table snapshot unconditionally. This change covers the case where the operation never recorded itself, under the same assumption. If a delete or merge repair runs first, it leaves the lock released, so the new check does nothing. ### Not changed - The lock still doesn't record its owner, which would avoid the problem at the source but changes the stored value. - `checkAndMutate` retry idempotency (HBASE-15588). - The failed-delete repair path calling `finishBackupExclusiveOperation` when the lock is already released (HBASE-28103). ### Tests `TestRepairAfterLostBackupLockResponse` makes the backup system table region fail the first successful update of the lock after applying it, which simulates a lost response. It checks that the backup fails without recording a session, that the lock is held, that `repair` returns 0 and releases it, and that the next backup succeeds. Without this change it fails at "Backup should succeed after repair" (the backup times out waiting for the lock). I ran it with `TestBackupRepair`, `TestRepairAfterFailedDelete`, `TestBackupDeleteWithFailures`, `TestIncrementalBackupMergeWithFailures` and `TestBackupSystemTable` on Linux (WSL) with JDK 21; all pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
