[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207824#comment-15207824 ] Phil Yang commented on HBASE-14678: --- Sounds good to me, [~stack] what do you think? > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207817#comment-15207817 ] Sean Busbey commented on HBASE-14678: - bump > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203785#comment-15203785 ] Sean Busbey commented on HBASE-14678: - several commits for this issue were in 1.2.0. any objections to closing with an appropriate fix version and moving the sub-task to a top-level task? > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202462#comment-15202462 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.3-IT #564 (See [https://builds.apache.org/job/HBase-1.3-IT/564/]) HBASE-15302 Reenable the other tests disabled by HBASE-14678 (Phil Yang) (tedyu: rev d942d4e5a1fe5644a51b5775f3a3e27dd219ad1d) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202506#comment-15202506 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.4 #31 (See [https://builds.apache.org/job/HBase-1.4/31/]) HBASE-15302 Reenable the other tests disabled by HBASE-14678 (Phil Yang) (tedyu: rev d942d4e5a1fe5644a51b5775f3a3e27dd219ad1d) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163599#comment-15163599 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-Trunk_matrix #734 (See [https://builds.apache.org/job/HBase-Trunk_matrix/734/]) HBASE-15302 Reenable the other tests disabled by HBASE-14678 (stack: rev 30cec72f9ade972d7e9ce4bba527b0e6074cae60) * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMobSnapshotCloneIndependence.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobFlushSnapshotFromClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Phil Yang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156431#comment-15156431 ] Phil Yang commented on HBASE-14678: --- I have created a subtask to add them back. Working on it :) > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155089#comment-15155089 ] stack commented on HBASE-14678: --- Thanks [~yangzhe1991] Do they pass locally for you if you run them in a loop? The disabled ones? Want to file a subtask to add them back if they pass? We could try it. A bunch has changed since these were disabled. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154131#comment-15154131 ] Phil Yang commented on HBASE-14678: --- Hi [~stack], it seems that some test cases are deleted by this issue and only two of them added back at HBASE-15023 . Any plan to add the others back? Thanks. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1 > > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056265#comment-15056265 ] stack commented on HBASE-14678: --- Ok. We seem to have figured the test killer (it was self-inflicted). The long-time flakies and hangers are being addressed slowly. It will be time to start reenabling these disabled tests soon. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984591#comment-14984591 ] stack commented on HBASE-14678: --- So, random tests show as hung, killed, with this on the end. ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? This is even after going back to 2.18.1 surefire. So, while this experiment has stabilized tests some, I've not been able to finger a particular test as the killer, the culprit. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980019#comment-14980019 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.2 #321 (See [https://builds.apache.org/job/HBase-1.2/321/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 6b5939c4b8ea526bf21a51031b32f6c5a355f6e5) * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979976#comment-14979976 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-TRUNK #6977 (See [https://builds.apache.org/job/HBase-TRUNK/6977/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 2288742c10e04d46212dbf70b931e460214992bf) * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980098#comment-14980098 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.3 #322 (See [https://builds.apache.org/job/HBase-1.3/322/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 45faf380284236ddc19b89231ecbd8a6c3693985) * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980107#comment-14980107 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.2-IT #251 (See [https://builds.apache.org/job/HBase-1.2-IT/251/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others (stack: rev 6b5939c4b8ea526bf21a51031b32f6c5a355f6e5) * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980167#comment-14980167 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.3-IT #280 (See [https://builds.apache.org/job/HBase-1.3-IT/280/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 45faf380284236ddc19b89231ecbd8a6c3693985) * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978728#comment-14978728 ] stack commented on HBASE-14678: --- In both cases, we exited with this: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (secondPartTestsExecution) on project hbase-server: ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /home/jenkins/jenkins-slave/workspace/HBase-1.3/jdk/latest1.8/label/Hadoop/hbase-server && /home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/jre/bin/java -enableassertions -Dhbase.test -Xmx2800m -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar /home/jenkins/jenkins-slave/workspace/HBase-1.3/jdk/latest1.8/label/Hadoop/hbase-server/target/surefire/surefirebooter4897692710194210921.jar /home/jenkins/jenkins-slave/workspace/HBase-1.3/jdk/latest1.8/label/Hadoop/hbase-server/target/surefire/surefire9037972699328837547tmp /home/jenkins/jenkins-slave/workspace/HBase-1.3/jdk/latest1.8/label/Hadoop/hbase-server/target/surefire/surefire_1158195690304051651854tmp [~chenheng] Thanks, yeah, trying to get your MR patch backported let me do that now. [~eclark] Let me ask. Something is up though still that we are exiting a JVM without report a few of the tests need fixing/removing still. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[
https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978723#comment-14978723
]
stack commented on HBASE-14678:
---
This morning on trunk... two builds in a row did this:
{code}
kalashnikov:hbase.git.commit stack$
kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/317/jdk=latest1.8,label=Hadoop/consoleText
Fetching
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/317/jdk=latest1.8,label=Hadoop/consoleText
Building remotely on H5 (Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs) in
workspace
/home/jenkins/jenkins-slave/workspace/HBase-1.3/jdk/latest1.8/label/Hadoop
Printing hanging tests
Hanging test :
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithOperationAttributes
Hanging test : org.apache.hadoop.hbase.mapreduce.TestRowCounter
Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat
Hanging test :
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
Hanging test : org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
Hanging test : org.apache.hadoop.hbase.util.TestHBaseFsck
Hanging test : org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
Hanging test : org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
Hanging test : org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded
Hanging test : org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Printing Failing tests
Failing test : org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/318/jdk=latest1.8,label=Hadoop/consoleText
Fetching
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/318/jdk=latest1.8,label=Hadoop/consoleText
Building remotely on H5 (Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs) in
workspace
/home/jenkins/jenkins-slave/workspace/HBase-1.3/jdk/latest1.8/label/Hadoop
Printing hanging tests
Hanging test :
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithOperationAttributes
Hanging test : org.apache.hadoop.hbase.mapreduce.TestRowCounter
Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat
Hanging test :
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
Hanging test : org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
Hanging test : org.apache.hadoop.hbase.util.TestHBaseFsck
Hanging test : org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
Hanging test : org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
Hanging test : org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded
Hanging test : org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Printing Failing tests
Failing test : org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
{code}
> Experiment: Temporarily disable balancer and a few others to see if root of
> crashed/timedout JVMs
> -
>
> Key: HBASE-14678
> URL: https://issues.apache.org/jira/browse/HBASE-14678
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Looking at recent builds of 1.2, I see a few of the runs finishing with kills
> and notice that a JVM exited without reporting back state. Running the
> hanging test finder, I can see at least that in one case that the balancer
> tests seem to be outstanding; looking in test output, seems to be still going
> on A few others are reported as hung but they look like they have just
> started running and are just killed by surefire.
> This issue is about trying to disable a few of the problematics like balancer
> tests to see if our overall stability improves. If so, I can concentrate on
> stabilizing these few tests. Else will just undo the experiment and put the
> tests back on line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978762#comment-14978762 ] stack commented on HBASE-14678: --- [~chenheng] But then I see how it hangs on trunk just now. Printing hanging tests Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Hanging test : org.apache.hadoop.hbase.mapreduce.TestSyncTable Hanging test : org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Printing Failing tests Failing test : org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient Failing test : org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient Ok. I'm going to do more aggressive pruning. I want to find tests that cause surefire exit. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979817#comment-14979817 ] stack commented on HBASE-14678: --- Removed TestWALSplitCompressed and TestWALSplit from branch-1.2+. Put back when experiement done. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979813#comment-14979813 ] stack commented on HBASE-14678: --- Let me find it [~chenheng] 1.2 builds just failed with this... its the surefire-killed issue: $ python ./dev-support/findHangingTests.py https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/318/jdk=latest1.7,label=Hadoop/consoleText Hanging test : org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint Hanging test : org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpointNoMaster Hanging test : org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager Hanging test : org.apache.hadoop.hbase.wal.TestWALSplit and this: kalashnikov:hbase.git.commit2 stack$ python ./dev-support/findHangingTests.py https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/319/jdk=latest1.8,label=Hadoop/consoleText Fetching https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/319/jdk=latest1.8,label=Hadoop/consoleText Building remotely on H9 (Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs) in workspace /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop Printing hanging tests Hanging test : org.apache.hadoop.hbase.wal.TestWALSplitCompressed Hanging test : org.apache.hadoop.hbase.replication.multiwal.TestReplicationSyncUpToolWithMultipleWAL Hanging test : org.apache.hadoop.hbase.wal.TestWALSplit Printing Failing tests Failing test : org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL Let me disable TestWALSplit. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[
https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979456#comment-14979456
]
Heng Chen commented on HBASE-14678:
---
{quote}
But then I see how it hangs on trunk just now.
{quote}
OK, Let me dig them today. [~stack]
> Experiment: Temporarily disable balancer and a few others to see if root of
> crashed/timedout JVMs
> -
>
> Key: HBASE-14678
> URL: https://issues.apache.org/jira/browse/HBASE-14678
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Looking at recent builds of 1.2, I see a few of the runs finishing with kills
> and notice that a JVM exited without reporting back state. Running the
> hanging test finder, I can see at least that in one case that the balancer
> tests seem to be outstanding; looking in test output, seems to be still going
> on A few others are reported as hung but they look like they have just
> started running and are just killed by surefire.
> This issue is about trying to disable a few of the problematics like balancer
> tests to see if our overall stability improves. If so, I can concentrate on
> stabilizing these few tests. Else will just undo the experiment and put the
> tests back on line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[
https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979466#comment-14979466
]
Heng Chen commented on HBASE-14678:
---
{quote}
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Hanging test : org.apache.hadoop.hbase.mapreduce.TestSyncTable
Hanging test : org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Printing Failing tests
Failing test : org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient
Failing test : org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient
{quote}
Could you post the console output URL? Thanks [~stack]
> Experiment: Temporarily disable balancer and a few others to see if root of
> crashed/timedout JVMs
> -
>
> Key: HBASE-14678
> URL: https://issues.apache.org/jira/browse/HBASE-14678
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Looking at recent builds of 1.2, I see a few of the runs finishing with kills
> and notice that a JVM exited without reporting back state. Running the
> hanging test finder, I can see at least that in one case that the balancer
> tests seem to be outstanding; looking in test output, seems to be still going
> on A few others are reported as hung but they look like they have just
> started running and are just killed by surefire.
> This issue is about trying to disable a few of the problematics like balancer
> tests to see if our overall stability improves. If so, I can concentrate on
> stabilizing these few tests. Else will just undo the experiment and put the
> tests back on line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975831#comment-14975831 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-TRUNK #6965 (See [https://builds.apache.org/job/HBase-TRUNK/6965/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 496d20cfca5a30bc72a29e4ef893424964f9fa91) * hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobFlushSnapshotFromClient.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977116#comment-14977116 ] stack commented on HBASE-14678: --- And this Hanging test : org.apache.hadoop.hbase.client.TestAdmin1 Hanging test : org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters Hanging test : org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd Hanging test : org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot Hanging test : org.apache.hadoop.hbase.util.TestHBaseFsck Hanging test : org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL Hanging test : org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Hanging test : org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient Hanging test : org.apache.hadoop.hbase.snapshot.TestExportSnapshot Hanging test : org.apache.hadoop.hbase.io.encoding.TestChangingEncoding in here... https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/313/jdk=latest1.8,label=Hadoop/consoleText Failed with this: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (secondPartTestsExecution) on project hbase-server: ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? .. a bunch of killing going on: Running org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters Killed Killed Killed Killed Killed > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977112#comment-14977112 ] stack commented on HBASE-14678: --- In this https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/312/jdk=latest1.7,label=Hadoop/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat-output.txt ... saw this: Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977127#comment-14977127 ] Elliott Clark commented on HBASE-14678: --- These seem to be running much better on non-apache hardware. At what point do we stop disabling things and start pushing hard on Apache ? > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[
https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977618#comment-14977618
]
Heng Chen commented on HBASE-14678:
---
{quote}
Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
{quote}
I notice this failed is after revert HBASE-14684 in branch-1.
IMO we could remove MiniMRCluster in {{TestHFileOutputFormat}} at least.
> Experiment: Temporarily disable balancer and a few others to see if root of
> crashed/timedout JVMs
> -
>
> Key: HBASE-14678
> URL: https://issues.apache.org/jira/browse/HBASE-14678
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Looking at recent builds of 1.2, I see a few of the runs finishing with kills
> and notice that a JVM exited without reporting back state. Running the
> hanging test finder, I can see at least that in one case that the balancer
> tests seem to be outstanding; looking in test output, seems to be still going
> on A few others are reported as hung but they look like they have just
> started running and are just killed by surefire.
> This issue is about trying to disable a few of the problematics like balancer
> tests to see if our overall stability improves. If so, I can concentrate on
> stabilizing these few tests. Else will just undo the experiment and put the
> tests back on line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975132#comment-14975132 ] stack commented on HBASE-14678: --- On 1.2, I saw this just now: Hanging test : org.apache.hadoop.hbase.mapreduce.TestImportTSVWithOperationAttributes Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat Hanging test : org.apache.hadoop.hbase.mapreduce.TestImportExport Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableInputFormat Hanging test : org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels There are kills in the test listing 5 kills. Let me try converting over these MR jobs to run without cluster... backporting HBASE-14684 rather than disabling the above. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975611#comment-14975611 ] stack commented on HBASE-14678: --- HBASE-14791 is about reenabling the TestMobFlushSnapshotFromClient test. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975605#comment-14975605 ] stack commented on HBASE-14678: --- I removed hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobFlushSnapshotFromClient.java from master branch. It hung failed: https://builds.apache.org/job/HBase-TRUNK/6962/consoleText for second time in 24 hours Let me open issue to dig in on it. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972734#comment-14972734 ] stack commented on HBASE-14678: --- [~chenheng] I think that the TestExportSnapshot and TestMobExportSnapshot fails have to do with our application of HBASE-14684... lets discuss over there on how to address (did revert for now). > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972692#comment-14972692 ] stack commented on HBASE-14678: --- Ok. Let me disable that one too as part of this experiment [~chenheng] Also saw this just now kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py https://builds.apache.org/job/HBase-1.3/jdk=latest1.8,label=Hadoop/300/consoleText Fetching https://builds.apache.org/job/HBase-1.3/jdk=latest1.8,label=Hadoop/300/consoleText Building remotely on H6 (Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs) in workspace /home/jenkins/jenkins-slave/workspace/HBase-1.3/jdk/latest1.8/label/Hadoop Printing hanging tests Hanging test : org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay Printing Failing tests https://builds.apache.org/job/HBase-1.3/jdk=latest1.8,label=Hadoop/300/consoleText And then this over in https://builds.apache.org/job/HBase-1.3/jdk=latest1.8,label=Hadoop/300/consoleText kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py https://builds.apache.org/job/HBase-1.2/jdk=latest1.7,label=Hadoop/300/consoleText Fetching https://builds.apache.org/job/HBase-1.2/jdk=latest1.7,label=Hadoop/300/consoleText Building remotely on H7 (Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs) in workspace /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop Printing hanging tests Hanging test : org.apache.hadoop.hbase.io.TestFileLink Hanging test : org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite Hanging test : org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Printing Failing tests Failing test : org.apache.hadoop.hbase.regionserver.TestAtomicOperation > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972892#comment-14972892 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-TRUNK #6954 (See [https://builds.apache.org/job/HBase-TRUNK/6954/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev df36aef23c5a4593a3160eb3937c54baf27991d1) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972933#comment-14972933 ] stack commented on HBASE-14678: --- Remove hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMobSnapshotCloneIndependence.java too... It depends on previous file removed. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972934#comment-14972934 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.2-IT #241 (See [https://builds.apache.org/job/HBase-1.2-IT/241/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others (stack: rev 4b1dd479cf01572c555f35f029f073e28ff8312a) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972875#comment-14972875 ] stack commented on HBASE-14678: --- I removed hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java from master, branch-1 and branch-1.2. It is failing. Here is example: https://builds.apache.org/job/HBase-1.3/jdk=latest1.7,label=Hadoop/302/consoleText Results : Flaked tests: org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence.testOnlineSnapshotMetadataChangesIndependent(org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence) Run 1: TestSnapshotCloneIndependence.testOnlineSnapshotMetadataChangesIndependent:152->runTestSnapshotMetadataChangesIndependent:357 null Run 2: TestSnapshotCloneIndependence.testOnlineSnapshotMetadataChangesIndependent:152->runTestSnapshotMetadataChangesIndependent:357 null Run 3: PASS Removed it till someone has chance to spend time on it. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972938#comment-14972938 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.3-IT #269 (See [https://builds.apache.org/job/HBase-1.3-IT/269/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 91bca7323a85bf0d2600d126fd6c2c2bb963776a) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972917#comment-14972917 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.2 #302 (See [https://builds.apache.org/job/HBase-1.2/302/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 4b1dd479cf01572c555f35f029f073e28ff8312a) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972929#comment-14972929 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.3 #303 (See [https://builds.apache.org/job/HBase-1.3/303/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 91bca7323a85bf0d2600d126fd6c2c2bb963776a) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972976#comment-14972976 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-TRUNK #6956 (See [https://builds.apache.org/job/HBase-TRUNK/6956/]) HBASE-14678 Experiment: Temporarily disable balancer and a few (stack: rev f8528f66ec253fbb3c08d2c3352dd73d9832a43b) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMobSnapshotCloneIndependence.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972326#comment-14972326 ] stack commented on HBASE-14678: --- Another Results : Failed tests: TestExportSnapshot.testExportRetry:325 expected:<0> but was:<1> Tests run: 2522, Failures: 1, Errors: 0, Skipped: 54 And this Results : Failed tests: TestMobExportSnapshot>TestExportSnapshot.testExportRetry:325 expected:<0> but was:<1> Tests run: 2522, Failures: 1, Errors: 0, Skipped: 54 > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[
https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972317#comment-14972317
]
stack commented on HBASE-14678:
---
More from trunk build
{code}
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient
Hanging test :
org.apache.hadoop.hbase.security.access.TestWithDisabledAuthorization
Hanging test : org.apache.hadoop.hbase.security.access.TestAccessController3
Hanging test :
org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions
Hanging test :
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas
Hanging test : org.apache.hadoop.hbase.security.access.TestTablePermissions
Hanging test : org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient
Printing Failing tests
Failing test : org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot
Failing test : org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Failing test : org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence
{code}
> Experiment: Temporarily disable balancer and a few others to see if root of
> crashed/timedout JVMs
> -
>
> Key: HBASE-14678
> URL: https://issues.apache.org/jira/browse/HBASE-14678
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Looking at recent builds of 1.2, I see a few of the runs finishing with kills
> and notice that a JVM exited without reporting back state. Running the
> hanging test finder, I can see at least that in one case that the balancer
> tests seem to be outstanding; looking in test output, seems to be still going
> on A few others are reported as hung but they look like they have just
> started running and are just killed by surefire.
> This issue is about trying to disable a few of the problematics like balancer
> tests to see if our overall stability improves. If so, I can concentrate on
> stabilizing these few tests. Else will just undo the experiment and put the
> tests back on line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972422#comment-14972422 ] Heng Chen commented on HBASE-14678: --- I see it too in HBASE-14674 https://builds.apache.org/job/PreCommit-HBASE-Build/16204//testReport/ > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971990#comment-14971990 ] stack commented on HBASE-14678: --- Here are other candidates to try turning off: kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/jdk=latest1.8,label=Hadoop/296/consoleText Fetching https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/jdk=latest1.8,label=Hadoop/296/consoleText Building remotely on H4 (Mapreduce zookeeper Hadoop Pig falcon Hdfs) in workspace /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop Printing hanging tests Hanging test : org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush Hanging test : org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint Hanging test : org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager Hanging test : org.apache.hadoop.hbase.regionserver.TestDefaultMemStore Hanging test : org.apache.hadoop.hbase.wal.TestWALSplit Hanging test : org.apache.hadoop.hbase.regionserver.TestRegionReplicas Printing Failing tests Failed here with the surefire crash Killed Killed Killed Killed (4 kills to match 4 hanging tests?). > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969746#comment-14969746 ] stack commented on HBASE-14678: --- What I removed: diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java diff --git a/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java b/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java diff --git a/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java b/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java On branch-1, added TestAssignmentManager to removal list. It is failing pretty reliably... just needs a fix but removing it for now. Leaving issue open because will put back these tests regardless. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[
https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970007#comment-14970007
]
Enis Soztutar commented on HBASE-14678:
---
The branch-1 build fails now.
{code}
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 42.187s
[INFO] Finished at: Thu Oct 22 15:13:21 PDT 2015
[INFO] Final Memory: 94M/589M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure: Compilation
failure:
[ERROR]
/Users/enis/projects/git-repos/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java:[55,56]
error: cannot find symbol
[ERROR] symbol: class FlushCache
[ERROR] location: class TimestampTestBase
[ERROR]
/Users/enis/projects/git-repos/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java:[177,10]
error: cannot find symbol
[ERROR] symbol: class FlushCache
[ERROR] location: class TimestampTestBase
[ERROR]
/Users/enis/projects/git-repos/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java:[391,10]
error: cannot find symbol
[ERROR] symbol: method flushcache()
[ERROR] location: variable loader of type Table
[ERROR]
/Users/enis/projects/git-repos/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java:[398,10]
error: cannot find symbol
[ERROR] symbol: method flushcache()
[ERROR] location: variable loader of type Table
[ERROR]
/Users/enis/projects/git-repos/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java:[102,46]
error: cannot find symbol
[ERROR] symbol: class FlushCache
[ERROR] location: class TestMultiVersions
[ERROR]
/Users/enis/projects/git-repos/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java:[110,57]
error: cannot find symbol
{code}
> Experiment: Temporarily disable balancer and a few others to see if root of
> crashed/timedout JVMs
> -
>
> Key: HBASE-14678
> URL: https://issues.apache.org/jira/browse/HBASE-14678
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>
> Looking at recent builds of 1.2, I see a few of the runs finishing with kills
> and notice that a JVM exited without reporting back state. Running the
> hanging test finder, I can see at least that in one case that the balancer
> tests seem to be outstanding; looking in test output, seems to be still going
> on A few others are reported as hung but they look like they have just
> started running and are just killed by surefire.
> This issue is about trying to disable a few of the problematics like balancer
> tests to see if our overall stability improves. If so, I can concentrate on
> stabilizing these few tests. Else will just undo the experiment and put the
> tests back on line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969991#comment-14969991 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.2-IT #234 (See [https://builds.apache.org/job/HBase-1.2-IT/234/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others (stack: rev 05e1e19565337d67fa76ea59ec09c26dcb580ec5) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970042#comment-14970042 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-TRUNK #6943 (See [https://builds.apache.org/job/HBase-TRUNK/6943/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others (stack: rev 93023f544b673ccc99fc0e327f2eca8964128097) * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969967#comment-14969967 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.3-IT #261 (See [https://builds.apache.org/job/HBase-1.3-IT/261/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 81d7d7ba7ec7936191639c681e2fbcc96c6e8baa) * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970230#comment-14970230 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.3-IT #262 (See [https://builds.apache.org/job/HBase-1.3-IT/262/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 3f7994b5ab66192f887f09b3edd296431126648d) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970084#comment-14970084 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.3 #292 (See [https://builds.apache.org/job/HBase-1.3/292/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 81d7d7ba7ec7936191639c681e2fbcc96c6e8baa) * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970257#comment-14970257 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.2-IT #235 (See [https://builds.apache.org/job/HBase-1.2-IT/235/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others (stack: rev b4fbace3ef7d746cb2d016f5de09b7591b2fe473) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970151#comment-14970151 ] stack commented on HBASE-14678: --- Over in https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/291/jdk=latest1.8,label=Hadoop/consoleText I see TestDistributedLogSplitting starting but not finishing and then report of crashed VM on end of test and it failing. The test logs 4MB of spew about not being able to go down. I removed it for now as likely cause of surefire crash committed to branch-1.2+ > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970155#comment-14970155 ] stack commented on HBASE-14678: --- Sorry about that [~enis] Overcommit on my part fixing. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970156#comment-14970156 ] stack commented on HBASE-14678: --- Fixed. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970221#comment-14970221 ] Enis Soztutar commented on HBASE-14678: --- No worries, just wanted to let you know. Thanks for working on the tests. > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970161#comment-14970161 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.2 #291 (See [https://builds.apache.org/job/HBase-1.2/291/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 05e1e19565337d67fa76ea59ec09c26dcb580ec5) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970342#comment-14970342 ] Hudson commented on HBASE-14678: FAILURE: Integrated in HBase-1.3 #293 (See [https://builds.apache.org/job/HBase-1.3/293/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev 3f7994b5ab66192f887f09b3edd296431126648d) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970360#comment-14970360 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-1.2 #292 (See [https://builds.apache.org/job/HBase-1.2/292/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others to (stack: rev b4fbace3ef7d746cb2d016f5de09b7591b2fe473) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs
[ https://issues.apache.org/jira/browse/HBASE-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970424#comment-14970424 ] Hudson commented on HBASE-14678: SUCCESS: Integrated in HBase-TRUNK #6945 (See [https://builds.apache.org/job/HBase-TRUNK/6945/]) HBASE-14678 Experiment: Temporarily disable balancer and a few others (stack: rev 129c48430e2102bb6f71b56047a8b15b31105fd2) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java > Experiment: Temporarily disable balancer and a few others to see if root of > crashed/timedout JVMs > - > > Key: HBASE-14678 > URL: https://issues.apache.org/jira/browse/HBASE-14678 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > > Looking at recent builds of 1.2, I see a few of the runs finishing with kills > and notice that a JVM exited without reporting back state. Running the > hanging test finder, I can see at least that in one case that the balancer > tests seem to be outstanding; looking in test output, seems to be still going > on A few others are reported as hung but they look like they have just > started running and are just killed by surefire. > This issue is about trying to disable a few of the problematics like balancer > tests to see if our overall stability improves. If so, I can concentrate on > stabilizing these few tests. Else will just undo the experiment and put the > tests back on line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
