[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926862#comment-16926862 ]
Rushabh S Shah commented on HBASE-23000: ---------------------------------------- > I had different test results when porting HBASE-22627. Not doubting the > results here, but wondering if more is going on. I am certain that HBASE-22627 caused these tests to fail. I checked out the commit before and after this commit and the tests were succeeding before and failing just after. > Ok, well HBASE-22627 is a critical bug fix and cannot be reverted, Agree with this statement. I think I understand why these tests are failing. The stack trace from logs: {noformat} WARNING! HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http:/ /hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'. 2019-09-10 12:03:30,031 FATAL [rushabhshah-ltm:58520.activeMasterManager] master.HMaster$2(1891): Failed to become active master org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'. at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:684) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:499) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:171) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:141) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:719) at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:202) at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1887) at java.lang.Thread.run(Thread.java:748) {noformat} It is expecting to find hbase.version file under wal dir but it is unable to find that file. Before HBASE-22627, the code use to go into this method [FSUtils.java#L667|https://github.com/apache/hbase/blob/branch-1.3/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L667] since metaRegiondir didn't exist. But after the change, metaRegionDir *does* exist because we write sequence file to this directory while instantiating the HRegion. Refer to this [FSUtils.java#L1227|https://github.com/apache/hbase/blob/branch-1.3/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L1227] In short, the root cause the tests are failing {code:title=FsUtils.java|borderStyle=solid} public static Path getWALTableDir(final Configuration conf, final TableName tableName) throws IOException { Path baseDir = new Path(getWALRootDir(conf), HConstants.BASE_NAMESPACE_DIR); --- This line was introduce in HBASE-22627 that is causing the issue. return new Path(new Path(baseDir, tableName.getNamespaceAsString()), tableName.getQualifierAsString()); {code} Hope this explains the issue. > Fix all consistently failing tests in branch-1.3 > ------------------------------------------------ > > Key: HBASE-23000 > URL: https://issues.apache.org/jira/browse/HBASE-23000 > Project: HBase > Issue Type: Bug > Components: test > Affects Versions: 1.3.6 > Reporter: Rushabh S Shah > Assignee: Rushabh S Shah > Priority: Major > > Flaky test report: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.3/Flaky_20Test_20Report/dashboard.html#job_2 > In last 30 builds this test failed all 30 times. > Here is the stack trace: > {noformat} > Stacktrace > java.io.IOException: Shutting down > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > Caused by: java.lang.RuntimeException: Master not initialized after 200000ms > seconds > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > {noformat} > Link to latest jenkins build: > https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/9351/testReport/org.apache.hadoop.hbase.fs/TestBlockReorder/testBlockLocation/ -- This message was sent by Atlassian Jira (v8.3.2#803003)