[jira] [Commented] (HBASE-23000) Fix all consistently failing tests in branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927935#comment-16927935 ] HBase QA commented on HBASE-23000: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 48s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange} 0m 0s{color} | {color:orange} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} branch-1.3 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 37s{color} | {color:green} branch-1.3 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} branch-1.3 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 36s{color} | {color:green} branch-1.3 passed {color} | | {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m 12s{color} | {color:red} branch has 10 errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green} branch-1.3 passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m 32s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 28s{color} | {color:green} branch-1.3 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m 12s{color} | {color:red} patch has 10 errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 10m 6s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 2.5.2 2.6.5 2.7.7. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 46s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}135m 7s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.master.cleaner.TestSnapshotFromMaster | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-608/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/608 | | JIRA Issue | HBASE-23000 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux f6f7512ed469 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-608/out/precommit/personality/provid
[jira] [Commented] (HBASE-23000) Fix all consistently failing tests in branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927394#comment-16927394 ] HBase QA commented on HBASE-23000: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange} 0m 0s{color} | {color:orange} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} branch-1.3 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 12s{color} | {color:green} branch-1.3 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} branch-1.3 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 43s{color} | {color:green} branch-1.3 passed {color} | | {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m 13s{color} | {color:red} branch has 10 errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green} branch-1.3 passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m 20s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 18s{color} | {color:green} branch-1.3 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 40s{color} | {color:red} hbase-server: The patch generated 2 new + 26 unchanged - 0 fixed = 28 total (was 26) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m 14s{color} | {color:red} patch has 10 errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 9m 43s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 2.5.2 2.6.5 2.7.7. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}113m 24s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}146m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-608/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/608 | | JIRA Issue | HBASE-23000 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8f8d4326274b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-608/out/precommit/personality/provided.sh | | git revision | branch-1.
[jira] [Commented] (HBASE-23000) Fix all consistently failing tests in branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927245#comment-16927245 ] Rushabh S Shah commented on HBASE-23000: We need to backport HBASE-18150 to fix these TFs. Created a PR to fix this. https://github.com/apache/hbase/pull/608 > Fix all consistently failing tests in branch-1.3 > > > Key: HBASE-23000 > URL: https://issues.apache.org/jira/browse/HBASE-23000 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.3.6 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Flaky test report: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.3/Flaky_20Test_20Report/dashboard.html#job_2 > In last 30 builds this test failed all 30 times. > Here is the stack trace: > {noformat} > Stacktrace > java.io.IOException: Shutting down > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > Caused by: java.lang.RuntimeException: Master not initialized after 20ms > seconds > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > {noformat} > Link to latest jenkins build: > https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/9351/testReport/org.apache.hadoop.hbase.fs/TestBlockReorder/testBlockLocation/ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-23000) Fix all consistently failing tests in branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927102#comment-16927102 ] Andrew Purtell commented on HBASE-23000: Thanks. Let’s patch branch-1.3 to fix this. > Fix all consistently failing tests in branch-1.3 > > > Key: HBASE-23000 > URL: https://issues.apache.org/jira/browse/HBASE-23000 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.3.6 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Flaky test report: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.3/Flaky_20Test_20Report/dashboard.html#job_2 > In last 30 builds this test failed all 30 times. > Here is the stack trace: > {noformat} > Stacktrace > java.io.IOException: Shutting down > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > Caused by: java.lang.RuntimeException: Master not initialized after 20ms > seconds > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > {noformat} > Link to latest jenkins build: > https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/9351/testReport/org.apache.hadoop.hbase.fs/TestBlockReorder/testBlockLocation/ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-23000) Fix all consistently failing tests in branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926862#comment-16926862 ] Rushabh S Shah commented on HBASE-23000: > I had different test results when porting HBASE-22627. Not doubting the > results here, but wondering if more is going on. I am certain that HBASE-22627 caused these tests to fail. I checked out the commit before and after this commit and the tests were succeeding before and failing just after. > Ok, well HBASE-22627 is a critical bug fix and cannot be reverted, Agree with this statement. I think I understand why these tests are failing. The stack trace from logs: {noformat} WARNING! HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http:/ /hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'. 2019-09-10 12:03:30,031 FATAL [rushabhshah-ltm:58520.activeMasterManager] master.HMaster$2(1891): Failed to become active master org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'. at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:684) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:499) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:171) at org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:141) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:719) at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:202) at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1887) at java.lang.Thread.run(Thread.java:748) {noformat} It is expecting to find hbase.version file under wal dir but it is unable to find that file. Before HBASE-22627, the code use to go into this method [FSUtils.java#L667|https://github.com/apache/hbase/blob/branch-1.3/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L667] since metaRegiondir didn't exist. But after the change, metaRegionDir *does* exist because we write sequence file to this directory while instantiating the HRegion. Refer to this [FSUtils.java#L1227|https://github.com/apache/hbase/blob/branch-1.3/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L1227] In short, the root cause the tests are failing {code:title=FsUtils.java|borderStyle=solid} public static Path getWALTableDir(final Configuration conf, final TableName tableName) throws IOException { Path baseDir = new Path(getWALRootDir(conf), HConstants.BASE_NAMESPACE_DIR); --- This line was introduce in HBASE-22627 that is causing the issue. return new Path(new Path(baseDir, tableName.getNamespaceAsString()), tableName.getQualifierAsString()); {code} Hope this explains the issue. > Fix all consistently failing tests in branch-1.3 > > > Key: HBASE-23000 > URL: https://issues.apache.org/jira/browse/HBASE-23000 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.3.6 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Flaky test report: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.3/Flaky_20Test_20Report/dashboard.html#job_2 > In last 30 builds this test failed all 30 times. > Here is the stack trace: > {noformat} > Stacktrace > java.io.IOException: Shutting down > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > Caused by: java.lang.RuntimeException: Master not initialized after 20ms > seconds > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > {noformat} > Link to latest jenkins build: > https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/9351/testReport/org.apache.hadoop.hbase.fs/TestBlockReorder/testBlockLocation/ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-23000) Fix all consistently failing tests in branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926808#comment-16926808 ] Andrew Purtell commented on HBASE-23000: Ok, well HBASE-22627 is a critical bug fix and cannot be reverted, so we shall need to fix what it broke. I had different test results when porting HBASE-22627. Not doubting the results here, but wondering if more is going on. > Fix all consistently failing tests in branch-1.3 > > > Key: HBASE-23000 > URL: https://issues.apache.org/jira/browse/HBASE-23000 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.3.6 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Flaky test report: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.3/Flaky_20Test_20Report/dashboard.html#job_2 > In last 30 builds this test failed all 30 times. > Here is the stack trace: > {noformat} > Stacktrace > java.io.IOException: Shutting down > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > Caused by: java.lang.RuntimeException: Master not initialized after 20ms > seconds > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > {noformat} > Link to latest jenkins build: > https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/9351/testReport/org.apache.hadoop.hbase.fs/TestBlockReorder/testBlockLocation/ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-23000) Fix all consistently failing tests in branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926798#comment-16926798 ] Rushabh S Shah commented on HBASE-23000: I did some analysis yesterday. Found the commit that caused these tests to fail. This is the jira: https://issues.apache.org/jira/browse/HBASE-22627 I reverted just 1 commit from branch-1.3 head and ran all the failing tests and it succeeded. See the results below. {noformat} [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 112.481 s - in org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad [INFO] Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.036 s - in org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort [INFO] Running org.apache.hadoop.hbase.master.TestMasterFileSystemWithWALDir [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.381 s - in org.apache.hadoop.hbase.master.TestMasterFileSystemWithWALDir [INFO] Running org.apache.hadoop.hbase.fs.TestBlockReorder [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.484 s - in org.apache.hadoop.hbase.fs.TestBlockReorder [INFO] [INFO] Results: [INFO] [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (secondPartTestsExecution) @ hbase-server --- [INFO] Tests are skipped. [INFO] {noformat} git log output: {noformat} commit e55a9054dacfacdef1dcb2d8d18e41dfd107f78e (HEAD -> HBASE-23000) Author: Rushabh Date: Tue Sep 10 09:42:26 2019 -0700 Revert "HBASE-22627 Port HBASE-22617 (Recovered WAL directories not getting cleaned up) to branch-1 (#339)" This reverts commit 13d1c231ba010854e3a3d2d5b9e028b40c28e3e0. commit 64c0ac5dd4e2dac664c58cbe309a9d8b4a69c4c9 (origin/branch-1.3, branch-1.3) Author: ravowlga123 Date: Thu Aug 29 17:29:24 2019 +0200 HBASE-22900 Multiple httpcore and httpclient versions included in binary package (branch-1.3 and branch-1.4) closes #563 Signed-off-by: Jan Hentschel Signed-off-by: Sean Busbey {noformat} I will post how the test fails in next post. > Fix all consistently failing tests in branch-1.3 > > > Key: HBASE-23000 > URL: https://issues.apache.org/jira/browse/HBASE-23000 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.3.6 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > > Flaky test report: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.3/Flaky_20Test_20Report/dashboard.html#job_2 > In last 30 builds this test failed all 30 times. > Here is the stack trace: > {noformat} > Stacktrace > java.io.IOException: Shutting down > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > Caused by: java.lang.RuntimeException: Master not initialized after 20ms > seconds > at > org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428) > {noformat} > Link to latest jenkins build: > https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/9351/testReport/org.apache.hadoop.hbase.fs/TestBlockReorder/testBlockLocation/ -- This message was sent by Atlassian Jira (v8.3.2#803003)