[ https://issues.apache.org/jira/browse/HBASE-28042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Jasani resolved HBASE-28042. ---------------------------------- Hadoop Flags: Reviewed Resolution: Fixed > Snapshot corruptions due to non-atomic rename within same filesystem > -------------------------------------------------------------------- > > Key: HBASE-28042 > URL: https://issues.apache.org/jira/browse/HBASE-28042 > Project: HBase > Issue Type: Bug > Reporter: Viraj Jasani > Assignee: Viraj Jasani > Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1 > > > Sequence of events that can lead to snapshot corruptions: > # Create snapshot using admin command > # Active master triggers async snapshot creation > # If the snapshot operation doesn't complete within 5 min, client gets > exception > {code:java} > org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Snapshot > 'T1_1691888405683_1691888440827_1' wasn't completed in expectedTime:600000 ms > {code} > # Client initiates snapshot deletion after this error > # In the snapshot completion/commit phase, the files are moved from tmp to > final dir. > # Snapshot delete and snapshot commit operations can cause corruption by > leaving incomplete metadata: > * [Snapshot commit] create > "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo" > * [Snapshot delete from client] delete > "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo" > * [Snapshot commit] create > "/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/data-manifest" > > The changes introduced by HBASE-21098 performs atomic rename for hbase 1 but > not for hbase 2 > {code:java} > public static void completeSnapshot(Path snapshotDir, Path workingDir, > FileSystem fs, > FileSystem workingDirFs, final Configuration conf) > throws SnapshotCreationException, IOException { > LOG.debug( > "Sentinel is done, just moving the snapshot from " + workingDir + " to > " + snapshotDir); > URI workingURI = workingDirFs.getUri(); > URI rootURI = fs.getUri(); > if ( > (!workingURI.getScheme().equals(rootURI.getScheme()) || > workingURI.getAuthority() == null > || !workingURI.getAuthority().equals(rootURI.getAuthority()) > || workingURI.getUserInfo() == null //always true for hdfs://{cluster} > || !workingURI.getUserInfo().equals(rootURI.getUserInfo()) > || !fs.rename(workingDir, snapshotDir)) //this condition isn't even > evaluated due to short circuit above > && !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, > true, conf) // non-atomic rename operation > ) { > throw new SnapshotCreationException("Failed to copy working directory(" > + workingDir > + ") to completed directory(" + snapshotDir + ")."); > } > } {code} > whereas for hbase 1 > {code:java} > // check UGI/userInfo > if (workingURI.getUserInfo() == null && rootURI.getUserInfo() != null) { > return true; > } > if (workingURI.getUserInfo() != null && > !workingURI.getUserInfo().equals(rootURI.getUserInfo())) { > return true; > } > {code} > this causes shouldSkipRenameSnapshotDirectories() to return false if > workingURI and rootURI share the same filesystem, which would always lead to > atomic rename: > {code:java} > if ((shouldSkipRenameSnapshotDirectories(workingURI, rootURI) > || !fs.rename(workingDir, snapshotDir)) > && !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true, > conf)) { > throw new SnapshotCreationException("Failed to copy working directory(" + > workingDir > + ") to completed directory(" + snapshotDir + ")."); > } {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)