[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333791#comment-16333791 ] Duo Zhang commented on HBASE-19794: --- The test still hangs... https://builds.apache.org/job/HBASE-Flaky-Tests/24995/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestZooKeeper-output.txt/*view*/ When shutting down after testLogSplittingAfterMasterRecoveryDueToZKExpiry. The test itself is a bit strange that we abort without confirming that the RS we abort is the one which carries the region. But anyway, the problem is that when arriving the shutdown point, there is no active master yet and our backup master is trying to become the active master. Let me open an issue to modify the test to see if it helps. > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.003.patch, > HBASE-19794.master.003.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333581#comment-16333581 ] Hudson commented on HBASE-19794: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4442 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4442/]) HBASE-19794 TestZooKeeper hangs (zhangduo: rev 27d00f5861336cb9f8f01174be33f8c3c431980f) * (edit) hbase-common/src/main/resources/hbase-default.xml * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ZKNamespaceManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestZKAsyncRegistry.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchemaServiceImpl.java > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.003.patch, > HBASE-19794.master.003.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333500#comment-16333500 ] Duo Zhang commented on HBASE-19794: --- Let me commit after fixing the checkstyle issues. > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.003.patch, > HBASE-19794.master.003.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333498#comment-16333498
]
Hadoop QA commented on HBASE-19794:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
6s{color} | {color:red} hbase-server: The patch generated 2 new + 205 unchanged
- 0 fixed = 207 total (was 205) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 12s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
15s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}100m
14s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19794 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906993/HBASE-19794.master.003.patch
|
| Optional Tests | asflicense javac javadoc unit xml findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 128a0399b128 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7fe4aa6fe4 |
| maven | version: Apache Maven 3.5.2
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333452#comment-16333452 ] Duo Zhang commented on HBASE-19794: --- Do not know when will YETUS-598 be online so delete the old patches and try again... > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.003.patch, > HBASE-19794.master.003.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333433#comment-16333433
]
Hadoop QA commented on HBASE-19794:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
7s{color} | {color:red} hbase-server: The patch generated 1 new + 205 unchanged
- 0 fixed = 206 total (was 205) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}124m
48s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19794 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906974/HBASE-19794.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c6e490663a34 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7fe4aa6fe4 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11142/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11142/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11142/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> TestZooKeeper hangs
>
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333417#comment-16333417 ] stack commented on HBASE-19794: --- .003 sets TableNamespaceManager general timeout at 30 seconds for all tests (as per Duo suggestion). Also includes a bit of debug for a related failure, TestZKAsyncRegistry > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.001.patch, > HBASE-19794.master.002.patch, HBASE-19794.master.003.patch, > org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333415#comment-16333415
]
stack commented on HBASE-19794:
---
{quote}Maybe we can do this in HBaseTestingUtility?
{quote}
I was thinking of doing this in general. Let me do it. Thanks for review
[~Apache9]
> TestZooKeeper hangs
> ---
>
> Key: HBASE-19794
> URL: https://issues.apache.org/jira/browse/HBASE-19794
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19794.master.001.patch,
> HBASE-19794.master.002.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt
>
>
> Seems like the TestZKAsyncRegistry that hangs in shutdown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1699#comment-1699
]
Duo Zhang commented on HBASE-19794:
---
{code}
// Default is 5 minutes which is too long for test given we are going to end up
blocked here
// in one of the tests below; after starting up backup master, we exit w/o
waiting on master
// to come up.
conf.setInt(HMaster.HBASE_MASTER_WAIT_ON_SERVICE_IN_SECONDS, 3);
{code}
Maybe we can do this in HBaseTestingUtility?
Anyway +1 on the patch. Can fix the checkstyle issue on committing.
Thanks.
> TestZooKeeper hangs
> ---
>
> Key: HBASE-19794
> URL: https://issues.apache.org/jira/browse/HBASE-19794
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19794.master.001.patch,
> HBASE-19794.master.002.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt
>
>
> Seems like the TestZKAsyncRegistry that hangs in shutdown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1690#comment-1690
]
Hadoop QA commented on HBASE-19794:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
46s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
7s{color} | {color:red} hbase-server: The patch generated 1 new + 205 unchanged
- 0 fixed = 206 total (was 205) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}100m
20s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19794 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906974/HBASE-19794.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 12ae0c5676ca 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7fe4aa6fe4 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11140/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11140/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11140/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> TestZooKeeper hangs
>
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1677#comment-1677 ] stack commented on HBASE-19794: --- .002 fixes checkstyle > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.001.patch, > HBASE-19794.master.002.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1676#comment-1676 ] stack commented on HBASE-19794: --- TestStochasticLoadBalancer timed out. Passes locally. Retrying (with checkstyle fix) > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.001.patch, > org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1665#comment-1665
]
Hadoop QA commented on HBASE-19794:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
57s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
9s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
29s{color} | {color:red} hbase-server: The patch generated 1 new + 205
unchanged - 0 fixed = 206 total (was 205) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 29s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 6s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19794 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906974/HBASE-19794.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cc118fc7be95 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7fe4aa6fe4 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11139/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/11139/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11139/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11139/console |
| Powered by | Apache
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1609#comment-1609 ] stack commented on HBASE-19794: --- .001 Duo patch w/ some cleanup of TableNamespaceManager and shortened wait on TableNamespaceManager to come up. We're not 'fixing' the root issue here. These are workarounds. See HBASE-19834 for what has to happen to make this work properly; i.e. make it so we never block the Master main thread so it is available to process cluster shutdown, etc., and is never blocked retrying client ops against regions that will never show up. > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19794.master.001.patch, > org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333295#comment-16333295
]
stack commented on HBASE-19794:
---
While Jira was down I spent some time on this last night. The backup Master
tries to become active during cluster shutdown but only gets this far:
{code:java}
78612 Thread 1542 (M:1;asf903:32967):
78613 State: TIMED_WAITING
78614 Blocked count: 178
78615 Waited count: 389
78616 Stack:
78617 java.lang.Object.wait(Native Method)
78618
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:168)
78619 org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
78620 org.apache.hadoop.hbase.client.HTable.get(HTable.java:362)
78621
org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1117)
78622
org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:1960)
78623
org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.getTableState(ConnectionUtils.java:131)
78624
org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:573)
78625
org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.isTableDisabled(ConnectionUtils.java:131)
78626
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:219)
78627
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105)
78628 org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
78629 org.apache.hadoop.hbase.client.HTable.get(HTable.java:362)
78630
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:139)
78631
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:276)
78632
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:101)
78633
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
78634
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
78635
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1059)
78636
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:921){code}
The backup Master will just be stuck here until all retries have been
exhausted. This is a variant on a issue seen elsewhere where client hosted in
server is trying to contact a server or region that is not going to show up,
usually because cluster is going down. We need means of signaling the client it
should give up because its host is going away. We probably also need to move
client communication off the main thread so the main thread remains available
and can react to shutdown.
Concurrent w/ my digging [~Apache9] was digging too and arrived at same place
(offline because Jira was down). He came up w/ a better workaround for now than
my cutting down on retries. He suggested minihbasecluster should put down
backup master's first, before we do the active Master (Thinking on it, it may
not work... damage may already have been done before we get to the shutdown
sequence... The backup master may have already started in on the shutdown
sequence).
Let me work up a patch based on Duo's
[https://github.com/Apache9/hbase/commit/97e030584504cc6019ef06462f6d44ca40125c45]
Let me add timeout, Duo's suggestion, and some other cleanup I came across
digging last night. Will also file issue to deal better w/ the root problem of
clients stuck in retry though cluster has been asked go down.
> TestZooKeeper hangs
> ---
>
> Key: HBASE-19794
> URL: https://issues.apache.org/jira/browse/HBASE-19794
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt
>
>
> Seems like the TestZKAsyncRegistry that hangs in shutdown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329649#comment-16329649
]
stack commented on HBASE-19794:
---
{quote}but I do not think set the thread daemon can solve all the problems...
{quote}
Agreed. We have a bunch of shutdown issues at mo. Making daemon seems to solve
at least TestRegionsOnServer. Will see what is left over. I wish I could get
this to fail locally (smile).
> TestZooKeeper hangs
> ---
>
> Key: HBASE-19794
> URL: https://issues.apache.org/jira/browse/HBASE-19794
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt
>
>
> Seems like the TestZKAsyncRegistry that hangs in shutdown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329619#comment-16329619 ] Duo Zhang commented on HBASE-19794: --- +1 on making it daemon, but I do not think set the thread daemon can solve all the problems. I saw it blocked in shutdownMiniCluster, not in the process exit... > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329492#comment-16329492 ] stack commented on HBASE-19794: --- I can't make this hang locally or on a test machine. I see it failing 16% of time according to [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html] Its a timeout. Log has loads of threads hanging out. Some Proc workers blocked: Thread 2268 (RS_CLOSE_REGION-asf903:58756-1): State: BLOCKED Blocked count: 12 Waited count: 17 Blocked on org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode@1c0991d8 Blocked by 2083 (ProcExecWrkr-6) Stack: org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportTransition(AssignmentManager.java:869) org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:857) org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:801) org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1561) org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2263) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:121) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 2267 (RS_CLOSE_REGION-asf903:58756-0): State: BLOCKED Blocked count: 14 Waited count: 17 Blocked on org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode@75cdbae3 Blocked by 2086 (ProcExecWrkr-9) Stack: org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportTransition(AssignmentManager.java:869) org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:857) org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:801) org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1561) org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2263) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:121) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) The Proc Workers are not daemon threads. Let me change that so at least we stop timing out. > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328329#comment-16328329 ] stack commented on HBASE-19794: --- Still need to look at this. Hopefully ** HBASE-19598 is related. > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325696#comment-16325696 ] stack commented on HBASE-19794: --- Assigned to myself. Will take a look over next day or so. > TestZooKeeper hangs > --- > > Key: HBASE-19794 > URL: https://issues.apache.org/jira/browse/HBASE-19794 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt > > > Seems like the TestZKAsyncRegistry that hangs in shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
