[
https://issues.apache.org/jira/browse/HBASE-21755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748774#comment-16748774
]
Nihal Jain commented on HBASE-21755:
------------------------------------
bq. if we correct the jira information for HBASE-21688 I think that'd be fine.
Do I need to re-open HBASE-21688 ?
bq. This should impact any release line that includes HBASE-17437, right?
Correct, it is going to impact all those releases.
> RS aborts while performing replication with wal dir on s3, root dir on hdfs
> ---------------------------------------------------------------------------
>
> Key: HBASE-21755
> URL: https://issues.apache.org/jira/browse/HBASE-21755
> Project: HBase
> Issue Type: Bug
> Components: Filesystem Integration, Replication, wal
> Affects Versions: 1.5.0, 2.1.3
> Reporter: Nihal Jain
> Assignee: Nihal Jain
> Priority: Critical
> Labels: s3
>
> *Environment/Configuration*
> - _hbase.wal.dir_ : Configured to be on s3
> - _hbase.rootdir_ : Configured to be on hdfs
> In replication scenario, while trying to get archived log dir (using method
> [WALEntryStream.java#L314|https://github.com/apache/hbase/blob/da92b3e0061a7c67aa9a3e403d68f3b56bf59370/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L314])
> we get the following exception:
> {code:java}
> 2019-01-21 17:43:55,440 ERROR
> [RS_REFRESH_PEER-regionserver/host2:22222-1.replicationSource,2.replicationSource.wal-reader.host2%2C22222%2C1548063439555.host2%2C22222%2C1548063439555.regiongroup-1,2]
> regionserver.ReplicationSource: Unexpected exception in
> RS_REFRESH_PEER-regionserver/host2:22222-1.replicationSource,2.replicationSource.wal-reader.host2%2C22222%2C1548063439555.host2%2C22222%2C1548063439555.regiongroup-1,2
>
> currentPath=hdfs://dummy_path/hbase/WALs/host2,22222,1548063439555/host2%2C22222%2C1548063439555.host2%2C22222%2C1548063439555.regiongroup-1.1548063492594
> java.lang.IllegalArgumentException: Wrong FS:
> s3a://xxxxxx/hbase128/oldWALs/host2%2C22222%2C1548063439555.host2%2C22222%2C1548063439555.regiongroup-1.1548063492594,
> expected: hdfs://dummy_path
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:246)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1622)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1619)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1634)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:465)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1742)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.getArchivedLog(WALEntryStream.java:319)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.resetReader(WALEntryStream.java:404)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.reset(WALEntryStream.java:161)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:148)
> 2019-01-21 17:43:55,444 ERROR
> [RS_REFRESH_PEER-regionserver/host2:22222-1.replicationSource,2.replicationSource.wal-reader.host2%2C22222%2C1548063439555.host2%2C22222%2C1548063439555.regiongroup-1,2]
> regionserver.HRegionServer: ***** ABORTING region server
> host2,22222,1548063439555: Unexpected exception in
> RS_REFRESH_PEER-regionserver/host2:22222-1.replicationSource,2.replicationSource.wal-reader.host2%2C22222%2C1548063439555.host2%2C22222%2C1548063439555.regiongroup-1,2
> *****
> java.lang.IllegalArgumentException: Wrong FS:
> s3a://xxxxxx/hbase128/oldWALs/host2%2C22222%2C1548063439555.host2%2C22222%2C1548063439555.regiongroup-1.1548063492594,
> expected: hdfs://dummy_path
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:246)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1622)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1619)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1634)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:465)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1742)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.getArchivedLog(WALEntryStream.java:319)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.resetReader(WALEntryStream.java:404)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.reset(WALEntryStream.java:161)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:148)
> {code}
>
> Current code is:
> {code:java}
> private Path getArchivedLog(Path path) throws IOException {
> Path rootDir = FSUtils.getRootDir(conf);
> // Try found the log in old dir
> Path oldLogDir = new Path(rootDir, HConstants.HREGION_OLDLOGDIR_NAME);
> Path archivedLogLocation = new Path(oldLogDir, path.getName());
> if (fs.exists(archivedLogLocation)) {
> LOG.info("Log " + path + " was moved to " + archivedLogLocation);
> return archivedLogLocation;
> }
> .
> .
> .
> return path;
> }
> {code}
> It considers root dir while we should use wal dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
