SpookyJumpyBeans opened a new pull request, #8921:
URL: https://github.com/apache/hbase/pull/8921

   https://issues.apache.org/jira/browse/HBASE-18886
   
   A backup takes the backup exclusive lock (`activesession:` in the backup 
system table) in `TableBackupClient.init`, and writes its RUNNING record later 
in `beginBackup`. If it fails in between, the lock stays `yes` with no record, 
and `hbase backup repair` returns 0 without releasing it, so every later backup 
create, delete and merge fails to take the lock.
   
   Two ways to get there:
   
   - A lost response to taking the lock: `checkAndMutate` is not idempotent 
under client retries (HBASE-15588), so the retried conditional put finds `yes` 
and fails, and the backup waits for its own lock until 
`hbase.backup.exclusive.op.timeout.seconds` and fails. 
`BackupAdminImpl.deleteBackups` returns -1 holding the lock on the same lost 
response.
   - A client that dies before recording the operation, as in the original 
report.
   
   ### Changes
   
   - `BackupSystemTable.isBackupExclusiveOperationActive()`: reads whether the 
lock is held.
   - `RepairCommand`: when there is no RUNNING backup, after repairing failed 
deletes and merges, release the lock if it is still held, and say so:
   
   ```java
   private void releaseExclusiveOperationLockIfHeld(BackupSystemTable sysTable)
     throws IOException {
     if (sysTable.isBackupExclusiveOperationActive()) {
       System.out.println("REPAIR status: the backup exclusive lock is held, 
but no backup,"
         + " delete or merge operation is recorded as running; releasing it. 
Run repair only"
         + " when no backup, delete or merge command is running.");
       sysTable.finishBackupExclusiveOperation();
     }
   }
   ```
   
   Repair already assumes no backup operation is running: when it finds a 
RUNNING backup it marks it failed and releases the lock without checking 
whether it is alive, and it deletes the backup system table snapshot 
unconditionally. This change covers the case where the operation never recorded 
itself, under the same assumption. If a delete or merge repair runs first, it 
leaves the lock released, so the new check does nothing.
   
   ### Not changed
   
   - The lock still doesn't record its owner, which would avoid the problem at 
the source but changes the stored value.
   - `checkAndMutate` retry idempotency (HBASE-15588).
   - The failed-delete repair path calling `finishBackupExclusiveOperation` 
when the lock is already released (HBASE-28103).
   
   ### Tests
   
   `TestRepairAfterLostBackupLockResponse` makes the backup system table region 
fail the first successful update of the lock after applying it, which simulates 
a lost response. It checks that the backup fails without recording a session, 
that the lock is held, that `repair` returns 0 and releases it, and that the 
next backup succeeds. Without this change it fails at "Backup should succeed 
after repair" (the backup times out waiting for the lock).
   
   I ran it with `TestBackupRepair`, `TestRepairAfterFailedDelete`, 
`TestBackupDeleteWithFailures`, `TestIncrementalBackupMergeWithFailures` and 
`TestBackupSystemTable` on Linux (WSL) with JDK 21; all pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to