Viraj Jasani created HBASE-28042: ------------------------------------ Summary: Snapshot corruptions due to non-atomic rename within same filesystem Key: HBASE-28042 URL: https://issues.apache.org/jira/browse/HBASE-28042 Project: HBase Issue Type: Bug Reporter: Viraj Jasani Assignee: Viraj Jasani Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1
Sequence of events that can lead to snapshot corruptions: # Create snapshot using admin command # Active master triggers async snapshot creation # If the snapshot operation doesn't complete within 5 min, client gets exception {code:java} org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Snapshot 'T1_1691888405683_1691888440827_1' wasn't completed in expectedTime:600000 ms {code} # Client initiates snapshot deletion after this error # In the snapshot completion/commit phase, the files are moved from tmp to final dir. # Snapshot delete and snapshot commit operations can cause corruption by leaving incomplete metadata: * [Snapshot commit] create "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo" * [Snapshot delete from client] delete "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo" * [Snapshot commit] create "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/data-manifest" The changes introduced by HBASE-21098 performs atomic rename for hbase 1 but not for hbase 2 {code:java} public static void completeSnapshot(Path snapshotDir, Path workingDir, FileSystem fs, FileSystem workingDirFs, final Configuration conf) throws SnapshotCreationException, IOException { LOG.debug( "Sentinel is done, just moving the snapshot from " + workingDir + " to " + snapshotDir); URI workingURI = workingDirFs.getUri(); URI rootURI = fs.getUri(); if ( (!workingURI.getScheme().equals(rootURI.getScheme()) || workingURI.getAuthority() == null || !workingURI.getAuthority().equals(rootURI.getAuthority()) || workingURI.getUserInfo() == null //always true for hdfs://{cluster} || !workingURI.getUserInfo().equals(rootURI.getUserInfo()) || !fs.rename(workingDir, snapshotDir)) //this condition isn't even evaluated due to short circuit above && !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true, conf) // non-atomic rename operation ) { throw new SnapshotCreationException("Failed to copy working directory(" + workingDir + ") to completed directory(" + snapshotDir + ")."); } } {code} whereas for hbase 1 {code:java} // check UGI/userInfo if (workingURI.getUserInfo() == null && rootURI.getUserInfo() != null) { return true; } if (workingURI.getUserInfo() != null && !workingURI.getUserInfo().equals(rootURI.getUserInfo())) { return true; } {code} this causes shouldSkipRenameSnapshotDirectories() to return false if workingURI and rootURI share the same filesystem, which would always lead to atomic rename: {code:java} if ((shouldSkipRenameSnapshotDirectories(workingURI, rootURI) || !fs.rename(workingDir, snapshotDir)) && !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true, conf)) { throw new SnapshotCreationException("Failed to copy working directory(" + workingDir + ") to completed directory(" + snapshotDir + ")."); } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)