[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314432#comment-16314432
]
Chia-Ping Tsai commented on HBASE-19714:
+1
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch,
> HBASE-19714.002.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-branch-2.patch
Patch for branch-2.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-branch-2.patch, HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Release Note:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignored since we have
already done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
On rolling upgrade: just do not do any replication peer modifications during
the rolling upgrading. There is no pb/layout changes on the peer/queue storage
on zk.
was:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
On rolling upgrade: just do not do any replication peer modifications during
the rolling upgrading. There is no pb/layout changes on the peer/queue storage
on zk.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-master-v1.patch
Retry.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314405#comment-16314405
]
Hadoop QA commented on HBASE-19397:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 33 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
36s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} The patch hbase-common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} hbase-client: The patch generated 0 new + 556
unchanged - 1 fixed = 556 total (was 557) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hbase-zookeeper passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hbase-replication: The patch generated 0 new + 22
unchanged - 36 fixed = 22 total (was 58) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hbase-procedure: The patch generated 0 new + 4
unchanged - 2 fixed = 4 total (was 6) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 6 new + 860
unchanged - 91 fixed = 866 total (was 951) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} hbase-mapreduce: The patch generated 0 new + 10
unchanged - 2 fixed = 10 total (was 12) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-shell passed checkstyle {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
3s{color} | {color:red} The patch generated 10 new + 65 unchanged - 1 fixed =
75 total (was 66) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
3s{color} | {color:red} The patch generated 4 new + 79 unchanged - 0 fixed = 83
total (was 79) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {c
[jira] [Commented] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314366#comment-16314366
]
Hudson commented on HBASE-19709:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4351 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4351/])
HBASE-19709 Ensure that we don't set a poolSize of 0 (elserj: rev
45e46bb24253ea84f68047676b419276a5994fb7)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-1
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add column to master web UI for displaying region states when listing regions of a table
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314364#comment-16314364 ] Hudson commented on HBASE-17436: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4351 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4351/]) HBASE-17436 Add column to master web UI for displaying region states (appy: rev 9cd0e1a490e118189289746686ec3aecef987b2a) * (edit) hbase-server/src/main/resources/hbase-webapps/master/table.jsp > Add column to master web UI for displaying region states when listing regions > of a table > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 3.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19717) IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-19717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314367#comment-16314367 ] Hudson commented on HBASE-19717: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4351 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4351/]) HBASE-19717 IntegrationTestDDLMasterFailover is using outdated value for (tedyu: rev 54928ce3ead61980a98cc64a323a661e82eb62f5) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestDDLMasterFailover.java > IntegrationTestDDLMasterFailover is using outdated values for > DataBlockEncoding > --- > > Key: HBASE-19717 > URL: https://issues.apache.org/jira/browse/HBASE-19717 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0-beta-1 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19717-branch-2.patch > > > We have removed PREFIX_TREE data block encoding, but > IntegrationTestDDLMasterFailover is still using it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314365#comment-16314365 ] Hudson commented on HBASE-19651: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4351 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4351/]) Revert "HBASE-19651 Remove LimitInputStream" (stack: rev e5f7030dedb12e257c1095326b7e3b5e49010c5f) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (add) hbase-common/src/main/appended-resources/META-INF/NOTICE * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java * (edit) NOTICE.txt Revert "Revert "HBASE-19651 Remove LimitInputStream"" (stack: rev 032701c96c64d1947393da2031f393fbeb8b7310) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (delete) hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java * (edit) NOTICE.txt * (delete) hbase-common/src/main/appended-resources/META-INF/NOTICE * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java Revert "Revert "Revert "HBASE-19651 Remove LimitInputStream""" (stack: rev 2bf2b5b2c111f89e8f452b1e9bbf9d2887043441) * (edit) NOTICE.txt * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java * (add) hbase-common/src/main/appended-resources/META-INF/NOTICE HBASE-19651 Remove LimitInputStream (stack: rev a30d9fe8d9a0bf0aacaf561ec494eaab6427074a) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (delete) hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java * (delete) hbase-common/src/main/appended-resources/META-INF/NOTICE * (edit) NOTICE.txt > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Release Note:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
On rolling upgrade: just do not do any replication peer modifications during
the rolling upgrading. There is no pb/layout changes on the peer/queue storage
on zk.
was:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-master-v1.patch
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19719) Fix checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-19719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19719: -- Resolution: Fixed Status: Resolved (was: Patch Available) Rebased and pushed to branch HBASE-19397. > Fix checkstyle issues > - > > Key: HBASE-19719 > URL: https://issues.apache.org/jira/browse/HBASE-19719 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: HBASE-19397 > > Attachments: HBASE-19719-HBASE-19397.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19719) Fix checkstyle issues
[
https://issues.apache.org/jira/browse/HBASE-19719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314328#comment-16314328
]
Hadoop QA commented on HBASE-19719:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HBASE-19397 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} HBASE-19397 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hbase-replication: The patch generated 0 new + 0
unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} hbase-server: The patch generated 0 new + 5
unchanged - 3 fixed = 5 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 23s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 53s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19719 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904888/HBASE-19719-HBASE-19397.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 08a50d324332 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| B
[jira] [Commented] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314281#comment-16314281 ] Appy commented on HBASE-19715: -- Time to ping recent authors of that file, [~chenheng]. Ping [~stack] for recommendations in case he has suggestion on how to dig deeper. In meantime, learning OQL to better analyze java heap dump. > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Attachment: screenshot-4.png > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314278#comment-16314278 ] Appy commented on HBASE-19715: -- It's just crazy how we have ~6 million Object[] instances. And lots of instances are of size 32 containing just AsyncReqestFutureImpl.class (yes, the Class object, see the common #1297 object id). It feels like it's some java internal array, but i am sure there's something wrong going by my observation of browsing around heap dump and seeing that repeat over and over. !screenshot-4.png|width=800px! > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Attachment: screenshot-3.png > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314241#comment-16314241 ] Appy commented on HBASE-19715: -- Trying to learn visualVM-foo. Seem to be creating too many Object array, and other arrays. Trying to find out the source of their creation. As soon as i enable profiling, i stop seeing crazy memory usage problem, so it's turning out a bit non-trivial to pinpoint the sources. !screenshot-3.png|width=500px! > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19717) IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-19717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-19717: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 2.0.0-beta-1) 2.0.0-beta-2 Status: Resolved (was: Patch Available) Thanks for the patch, Sergey. > IntegrationTestDDLMasterFailover is using outdated values for > DataBlockEncoding > --- > > Key: HBASE-19717 > URL: https://issues.apache.org/jira/browse/HBASE-19717 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0-beta-1 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19717-branch-2.patch > > > We have removed PREFIX_TREE data block encoding, but > IntegrationTestDDLMasterFailover is still using it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19717) IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding
[
https://issues.apache.org/jira/browse/HBASE-19717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314226#comment-16314226
]
Hadoop QA commented on HBASE-19717:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
16s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19717 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904882/HBASE-19717-branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux fa6371bc2663 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 443b27b707 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10909/testReport/ |
| modules | C: hbase-it U: hbase-it |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10909/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> IntegrationTestDDLMasterFailover is using outdated values for
> DataBlockEncoding
> ---
>
> Key: HBASE-19717
>
[jira] [Commented] (HBASE-19719) Fix checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-19719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314220#comment-16314220 ] Duo Zhang commented on HBASE-19719: --- The checkstyle issues for EventType and ExecutorType are not easy to fix, unless we destroy the old layout way for the enum constants... And also for HMaster and HBaseFsck, the issues are both method too long... > Fix checkstyle issues > - > > Key: HBASE-19719 > URL: https://issues.apache.org/jira/browse/HBASE-19719 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: HBASE-19397 > > Attachments: HBASE-19719-HBASE-19397.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19719) Fix checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-19719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19719: -- Attachment: HBASE-19719-HBASE-19397.patch > Fix checkstyle issues > - > > Key: HBASE-19719 > URL: https://issues.apache.org/jira/browse/HBASE-19719 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: HBASE-19397 > > Attachments: HBASE-19719-HBASE-19397.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19719) Fix checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-19719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19719: -- Status: Patch Available (was: Open) > Fix checkstyle issues > - > > Key: HBASE-19719 > URL: https://issues.apache.org/jira/browse/HBASE-19719 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: HBASE-19397 > > Attachments: HBASE-19719-HBASE-19397.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19719) Fix checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-19719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19719: -- Summary: Fix checkstyle issues (was: Fix checkstyle issues before merge) > Fix checkstyle issues > - > > Key: HBASE-19719 > URL: https://issues.apache.org/jira/browse/HBASE-19719 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: HBASE-19397 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314211#comment-16314211
]
Hadoop QA commented on HBASE-19714:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
27s{color} | {color:red} The patch generated 2 new + 718 unchanged - 11 fixed =
720 total (was 729) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 23 new + 1237 unchanged - 3 fixed
= 1260 total (was 1240) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
4s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19714 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904881/HBASE-19714.002.branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux 8ce68a11dd1a 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 443b27b707 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| rubocop | v0.52.1 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/10908/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| ruby-lint |
https://builds.apache.org/job/PreCommit-HBASE-Build/10908/artifact/patchprocess/diff-patch-ruby-lint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10908/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10908/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch,
> HBASE-19714.002.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314209#comment-16314209 ] BELUGA BEHR commented on HBASE-19651: - Thank you team! No worries about the credit. I just want HBase to be the best it can be :) I've talked to [~appy] offline and I'll start formatting the patches as requested. Sorry about that. Thank you for working around it. > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314203#comment-16314203
]
Duo Zhang commented on HBASE-19397:
---
Tried several times, TestHRegion can pass locally...
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19719) Fix checkstyle issues before merge
Duo Zhang created HBASE-19719: - Summary: Fix checkstyle issues before merge Key: HBASE-19719 URL: https://issues.apache.org/jira/browse/HBASE-19719 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang Assignee: Duo Zhang Fix For: HBASE-19397 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314202#comment-16314202 ] Hudson commented on HBASE-19483: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4350 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4350/]) HBASE-19483 Add proper privilege check for rsgroup commands (tedyu: rev fc7736eb00bb47d619a1345e32f0136751ea8c0b) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java * (edit) hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithLabels.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) src/main/asciidoc/_chapters/appendix_acl_matrix.adoc * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessChecker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/VisibilityTestUtil.java * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithVisibilityLabels.java * (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandlerWithLabels.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithVisibilityLabels.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java * (edit) src/main/asciidoc/_chapters/security.adoc HBASE-19483 Add proper privilege check for rsgroup commands - revert due (tedyu: rev 5ce93511290d2bcf69e41d7ca5cec7464076d81d) * (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandlerWithLabels.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithVisibilityLabels.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithVisibilityLabels.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * (edit) src/main/asciidoc/_chapters/appendix_acl_matrix.adoc * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc * (edit) hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithLabels.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/VisibilityTestUtil.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessChecker.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) src/main/asciidoc/_chapters/security.adoc > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: 19483.master.011.patch, 19483.v11.patch, > 19483.v11.patch, HBASE-19483.master.001.patch, HBASE-19483.master.002.patch, > HBASE-19483.master.003.patch, HBASE-19483.master.004.patch, > HBASE-19483.master.005.patch, HBASE-19483.master.006.patch, > HBASE-19483.master.007.patch, HBASE-19483.master.008.patch, > HBASE-19483.master.009.patch, HBASE-19483.master.010.patch, > HBASE-19483.master.011.patch, HBASE-19483.master.011.patch, > HBASE-19483.master.012.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgro
[jira] [Commented] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314196#comment-16314196 ] Appy commented on HBASE-19715: -- Ran some other tests to get a sense of memory usage in general. Here's data from two other tests: - TestAssignmentManager: Size: 321M; Used: 142M; Max: 4G - TestStochasticLoadBalancer2: Size: 520M; Used: 181M; Max: 4G - TestMultiRespectsLimits#testMultiLimits : Size: 3G; Used: 2G; Max: 4G *<-- clearly way off!* > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314194#comment-16314194
]
Duo Zhang commented on HBASE-19397:
---
Let me check the failed UT. And also open a sub task to address the checkstyle
issues.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19163) "Maximum lock count exceeded" from region server's batch processing
[
https://issues.apache.org/jira/browse/HBASE-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314193#comment-16314193
]
Hadoop QA commented on HBASE-19163:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_161 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-1 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_161 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
21s{color} | {color:red} hbase-server: The patch generated 3 new + 336
unchanged - 3 fixed = 339 total (was 339) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 0s{color} | {color:green} Patch does not cause any errors with Hadoop
2.4.1 2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m
27s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-19163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904865/HBASE-19163-branch-1-v001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck
[jira] [Created] (HBASE-19718) Remove PREFIX_TREE from compression.adoc
Ted Yu created HBASE-19718:
--
Summary: Remove PREFIX_TREE from compression.adoc
Key: HBASE-19718
URL: https://issues.apache.org/jira/browse/HBASE-19718
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Trivial
compression.adoc still refers to PREFIX_TREE though the encoding has been
removed:
{code}
-data_block_encodingEncoding algorithm (e.g. prefix compression) to
use for data blocks in the test column family, one
of [NONE, PREFIX, DIFF, FAST_DIFF, PREFIX_TREE].
{code}
ROW_INDEX_V1 should be put in its place.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19717) IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-19717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314184#comment-16314184 ] Ted Yu commented on HBASE-19717: +1 > IntegrationTestDDLMasterFailover is using outdated values for > DataBlockEncoding > --- > > Key: HBASE-19717 > URL: https://issues.apache.org/jira/browse/HBASE-19717 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0-beta-1 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19717-branch-2.patch > > > We have removed PREFIX_TREE data block encoding, but > IntegrationTestDDLMasterFailover is still using it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314176#comment-16314176 ] Appy commented on HBASE-19715: -- Profiling...2GB memory usage for single test function TestMultiRespectsLimits#testMultiLimits. Is that normal?? Need to figure out more. !screenshot-2.png!width=800px! > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314176#comment-16314176 ] Appy edited comment on HBASE-19715 at 1/6/18 12:09 AM: --- Profiling...2GB memory usage for single test function TestMultiRespectsLimits#testMultiLimits. Is that normal?? Need to figure out more. !screenshot-2.png|width=800px! was (Author: appy): Profiling...2GB memory usage for single test function TestMultiRespectsLimits#testMultiLimits. Is that normal?? Need to figure out more. !screenshot-2.png!width=800px! > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Attachment: screenshot-2.png > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png, > screenshot-2.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19717) IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-19717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-19717: Status: Patch Available (was: Open) Removed the outdated value for PREFIX_TREE and added ROW_INDEX_V1 (hope that's a valid for this test) > IntegrationTestDDLMasterFailover is using outdated values for > DataBlockEncoding > --- > > Key: HBASE-19717 > URL: https://issues.apache.org/jira/browse/HBASE-19717 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0-beta-1 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19717-branch-2.patch > > > We have removed PREFIX_TREE data block encoding, but > IntegrationTestDDLMasterFailover is still using it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19717) IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-19717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-19717: Attachment: HBASE-19717-branch-2.patch > IntegrationTestDDLMasterFailover is using outdated values for > DataBlockEncoding > --- > > Key: HBASE-19717 > URL: https://issues.apache.org/jira/browse/HBASE-19717 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0-beta-1 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19717-branch-2.patch > > > We have removed PREFIX_TREE data block encoding, but > IntegrationTestDDLMasterFailover is still using it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19717) IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding
Sergey Soldatov created HBASE-19717: --- Summary: IntegrationTestDDLMasterFailover is using outdated values for DataBlockEncoding Key: HBASE-19717 URL: https://issues.apache.org/jira/browse/HBASE-19717 Project: HBase Issue Type: Bug Components: integration tests Affects Versions: 2.0.0-beta-1 Reporter: Sergey Soldatov Assignee: Sergey Soldatov Fix For: 2.0.0-beta-1 We have removed PREFIX_TREE data block encoding, but IntegrationTestDDLMasterFailover is still using it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19714:
---
Attachment: HBASE-19714.002.branch-2.patch
.002 Fixes some of the rubylint stuff I can fix.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch,
> HBASE-19714.002.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314167#comment-16314167
]
Ted Yu commented on HBASE-19714:
+1
See if the rubocop warning can be fixed.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch,
> HBASE-19714.002.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314164#comment-16314164
]
Hadoop QA commented on HBASE-19714:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 9 new + 721 unchanged - 8 fixed =
730 total (was 729) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 23 new + 1237 unchanged - 3 fixed
= 1260 total (was 1240) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
15s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19714 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904874/HBASE-19714.001.branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux 99e39dfbec0e 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / ce80e8e38f |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| rubocop | v0.52.1 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/10907/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| ruby-lint |
https://builds.apache.org/job/PreCommit-HBASE-Build/10907/artifact/patchprocess/diff-patch-ruby-lint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10907/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10907/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> s
[jira] [Created] (HBASE-19716) Fix flaky test master.assignment.TestAssignmentManager
Umesh Agashe created HBASE-19716: Summary: Fix flaky test master.assignment.TestAssignmentManager Key: HBASE-19716 URL: https://issues.apache.org/jira/browse/HBASE-19716 Project: HBase Issue Type: Bug Components: amv2 Affects Versions: 2.0.0-beta-1 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-beta-2 Fix flaky test: master.assignment.TestAssignmentManager 89.7% (70 / 78). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19709:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the reviews, all.
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-1
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19709:
---
Fix Version/s: (was: 2.0.0-beta-2)
2.0.0-beta-1
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-1
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Description: !screenshot-1.png|width=800px! Attached logs for both cases, when it passes and fails. Link (temporary) to logs: passed: http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ failed: http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ Correlating across more runs, whenever the tests passes, it does so within 10-30sec of 3min deadline for medium tests. So i think we can make it pass by just increasing the timeout. But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in the log. Test code doesn't seem to be doing anything that intensive. Are we mismanaging the memory somewhere? was: !screenshot-1.png|width=800px! Attached logs for both cases, when it passes and fails. Link (temporary) to logs: passed: http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ failed: http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ Correlating across more runs, whenever the tests passes, it does so within 10-30 of 3min deadline for medium tests. So i think we can make it pass by just increasing the timeout. But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in the log. Test code doesn't seem to be doing anything that intensive. Are we mismanaging the memory somewhere? > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30sec of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Description: !screenshot-1.png|width=800px! Attached logs for both cases, when it passes and fails. Link (temporary) to logs: passed: http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ failed: http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ Correlating across more runs, whenever the tests passes, it does so within 10-30 of 3min deadline for medium tests. So i think we can make it pass by just increasing the timeout. But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in the log. Test code doesn't seem to be doing anything that intensive. Are we mismanaging the memory somewhere? was: !screenshot-1.png|width=800px! > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png > > > !screenshot-1.png|width=800px! > Attached logs for both cases, when it passes and fails. > Link (temporary) to logs: > passed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33449/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > failed: > http://104.198.223.121:8080/job/HBase-Flaky-Tests/33455/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestMultiRespectsLimits-output.txt/*view*/ > Correlating across more runs, whenever the tests passes, it does so within > 10-30 of 3min deadline for medium tests. > So i think we can make it pass by just increasing the timeout. > But I'm a bit skeptical after seeing all those long GC pauses (10sec +) in > the log. Test code doesn't seem to be doing anything that intensive. Are we > mismanaging the memory somewhere? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Attachment: failued.txt passed.txt > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: failued.txt, passed.txt, screenshot-1.png > > > !screenshot-1.png|width=800px! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Description: !screenshot-1.png|width=800px! was:!screenshot-1.png|width=800px! > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: screenshot-1.png > > > !screenshot-1.png|width=800px! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Description: !screenshot-1.png|width=800px! (was: !screenshot-1.png!) > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: screenshot-1.png > > > !screenshot-1.png|width=800px! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
Appy created HBASE-19715: Summary: Fix timing out test TestMultiRespectsLimits Key: HBASE-19715 URL: https://issues.apache.org/jira/browse/HBASE-19715 Project: HBase Issue Type: Bug Reporter: Appy Assignee: Appy -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Description: !screenshot-1.png! > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: screenshot-1.png > > > !screenshot-1.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19715) Fix timing out test TestMultiRespectsLimits
[ https://issues.apache.org/jira/browse/HBASE-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19715: - Attachment: screenshot-1.png > Fix timing out test TestMultiRespectsLimits > --- > > Key: HBASE-19715 > URL: https://issues.apache.org/jira/browse/HBASE-19715 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: screenshot-1.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314145#comment-16314145
]
Josh Elser commented on HBASE-19714:
:D
Busted by HBASE-19651. Resubmitted the same patch.
bq. Seems we should run the shell tests if any changes are added to client
module
Ya, that seems like it'd be a good idea.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19714:
---
Attachment: HBASE-19714.001.branch-2.patch
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314143#comment-16314143 ] Josh Elser commented on HBASE-19651: n/m, I see Stack just got it :) > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19714:
---
Attachment: (was: HBASE-19714.001.branch-2.patch)
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314134#comment-16314134
]
Chia-Ping Tsai edited comment on HBASE-19714 at 1/5/18 11:34 PM:
-
Thanks for the nice finding. Seems we should run the shell tests if any changes
are added to client module. +1 pending QA.
QA beats me by one second.:(
was (Author: chia7712):
Thanks for the nice finding. Seems we should run the shell tests if any changes
are added to client module. +1 padding QA
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314134#comment-16314134
]
Chia-Ping Tsai commented on HBASE-19714:
Thanks for the nice finding. Seems we should run the shell tests if any changes
are added to client module. +1 padding QA
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314130#comment-16314130 ] Josh Elser commented on HBASE-19651: Looks like some trivial merge conflict markers got committed to branch-2. Fixing. > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314125#comment-16314125
]
Hadoop QA commented on HBASE-19714:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 4m
28s{color} | {color:red} root in branch-2 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
15s{color} | {color:red} hbase-shell in branch-2 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
44s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
24s{color} | {color:red} The patch generated 9 new + 721 unchanged - 8 fixed =
730 total (was 729) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 23 new + 1237 unchanged - 3 fixed
= 1260 total (was 1240) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
7s{color} | {color:red} hbase-shell in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 7s{color}
| {color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 7m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19714 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904872/HBASE-19714.001.branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux ebffeb845694 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 88c7493948 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/artifact/patchprocess/branch-mvninstall-root.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/artifact/patchprocess/branch-javadoc-hbase-shell.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/artifact/patchprocess/patch-mvninstall-root.txt
|
| rubocop | v0.52.1 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| ruby-lint |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/artifact/patchprocess/diff-patch-ruby-lint.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/artifact/patchprocess/patch-javadoc-hbase-shell.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/artifact/patchprocess/patch-unit-hbase-shell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10906/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
>
[jira] [Commented] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314117#comment-16314117
]
stack commented on HBASE-19709:
---
+1 on push.
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-2
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314109#comment-16314109
]
Josh Elser commented on HBASE-19709:
{quote}
[WARNING] Tests run: 3311, Failures: 0, Errors: 0, Skipped: 33
{quote}
Seems like the runner died at the end? I think I'm OK to push this.
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-2
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19714:
---
Status: Patch Available (was: Open)
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19714:
---
Attachment: HBASE-19714.001.branch-2.patch
.001 Fixed some crappy unit tests and added some new tests which catch this
failure for the future.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19714.001.branch-2.patch
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19681) Online snapshot creation failing with missing store file
[
https://issues.apache.org/jira/browse/HBASE-19681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314095#comment-16314095
]
Anirban Roy commented on HBASE-19681:
-
Could it be related to
[lHBASE-16754|https://issues.apache.org/jira/browse/HBASE-16754] ? The stack
traces looks identical.
> Online snapshot creation failing with missing store file
>
>
> Key: HBASE-19681
> URL: https://issues.apache.org/jira/browse/HBASE-19681
> Project: HBase
> Issue Type: Bug
> Components: backup&restore, Performance, scaling, snapshots
>Affects Versions: 1.3.0
> Environment: Hadoop - 2.7.3
> HBase 1.3.0
> OS - GNU/Linux x86_64
> Cluster - Amazon Elastic Mapreduce
>Reporter: Anirban Roy
> Attachments: region-server-missing file-log.doc,
> region-server-snapshot-exception-log.doc
>
>
> We are facing problem creating online snapshot of our HBase table. The table
> contains 20TB data and receiving ~1 writes per second. The snapshot
> creating failing intermittently with error that some hfile missing, see the
> detailed output below. Once we locate the region server hosting the region
> and restart the region server, snapshot creation succeeds. It seems the
> missing hfile removed due to minor compaction, but region server still holds
> the pointer to the file.
> [hadoop@ip-10-0-12-164 ~]$ hbase shell
> HBase Shell; enter 'help' for list of supported commands.
> Type "exit" to leave the HBase Shell
> Version 1.3.0, rUnknown, Fri Feb 17 18:15:07 UTC 2017
>
> hbase(main):001:0> snapshot ‘x_table’, ‘x_snapshot’
>
> ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
> ss=x_snapshot table=x_table type=FLUSH } had an error. Procedure x_snapshot
> { waiting=[] done=[ip-10-0-9-31.ec2.internal,16020,1508372578254,
> ip-10-0-0-32.ec2.internal,16020,1508372591059,
> ip-10-0-14-221.ec2.internal,16020,1508372580873,
> ip-10-0-15-185.ec2.internal,16020,1508372588507,
> ip-10-0-9-43.ec2.internal,16020,1508372569107,
> ip-10-0-10-62.ec2.internal,16020,1512885921693,
> ip-10-0-8-216.ec2.internal,16020,1508372584133,
> ip-10-0-1-207.ec2.internal,16020,1508372580144,
> ip-10-0-0-173.ec2.internal,16020,1508372584969,
> ip-10-0-4-79.ec2.internal,16020,1508372587161,
> ip-10-0-3-165.ec2.internal,16020,1508372593566,
> ip-10-0-14-137.ec2.internal,16020,1508372583225,
> ip-10-0-6-33.ec2.internal,16020,1508372581587,
> ip-10-0-15-199.ec2.internal,16020,1508372587478,
> ip-10-0-5-253.ec2.internal,16020,1508372581243,
> ip-10-0-1-99.ec2.internal,16020,1508372609684] }
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:354)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1058)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:61089)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by:
> org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
> ip-10-0-3-13.ec2.internal,16020,1508372563772:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
> java.io.FileNotFoundException: File does not exist:
> hdfs://ip-10-0-12-164.ec2.internal:8020/user/hbase/data/default/x_table/ecbb3aeaf7c5b1f65742deab5812362c/d/f76d8827c29244b99bf9344982956523
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:315)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:344)
> ... 6 more
> Caused by:
> org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
> java.io.FileNotFoundException: File does not exist:
> hdfs://ip-10-0-12-164.ec2.internal:8020/user/hbase/data/default/x_table/ecbb3aeaf7c5b1f65742deab5812362c/d/f76d8827c29244b99bf9344982956523
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:347)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:140)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:160)
>
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314070#comment-16314070 ] stack commented on HBASE-19651: --- I see what Ted is on about now. Let me fix (revert, fix, re-push). > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add column to master web UI for displaying region states when listing regions of a table
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314043#comment-16314043 ] Appy commented on HBASE-17436: -- Ahh, -p0 takes care of not striping the first part. Good to know. Thanks Drob! > Add column to master web UI for displaying region states when listing regions > of a table > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 3.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314042#comment-16314042 ] stack commented on HBASE-19651: --- [~belugabehr] In case you are looking for more info on formatting patches, http://hbase.apache.org/book.html#submitting.patches.create > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17436) Add column to master web UI for displaying region states when listing regions of a table
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17436: - Fix Version/s: (was: 2.0.0-beta-2) 3.0.0 > Add column to master web UI for displaying region states when listing regions > of a table > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 3.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add column to master web UI for displaying region states when listing regions of a table
[
https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314040#comment-16314040
]
Mike Drob commented on HBASE-17436:
---
bq. After sometime, got it working by adding "a/" and "b/" to file paths in the
diff. Is there some other way too? Always fun to know what we don't already.
{{curl patch.txt | patch -p0}} should do the trick.
> Add column to master web UI for displaying region states when listing regions
> of a table
>
>
> Key: HBASE-17436
> URL: https://issues.apache.org/jira/browse/HBASE-17436
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
> Labels: ui, usability
> Fix For: 3.0.0
>
> Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt,
> 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch,
> HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch,
> HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch,
> HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot
> 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png,
> Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35
> PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png
>
>
> [~rpednekar] and I were looking at a case where the count displayed under
> Other Regions was high (~1200).
> Since the table page just maintains a count instead of List of region names,
> it is very difficult for user to determine which regions belong to this
> category.
> We should provide facility to provide more details for this category (LOG,
> JMX, etc).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17436) Add column to master web UI for displaying region states when listing regions of a table
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17436: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed the changes in table.jsp as discussed. Thanks for the patch [~gubjanos]. > Add column to master web UI for displaying region states when listing regions > of a table > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0-beta-2 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17436) Add column to master web UI for displaying region states when listing regions of a table
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17436: - Summary: Add column to master web UI for displaying region states when listing regions of a table (was: Add facility to provide more information for Other Regions seen on Master UI) > Add column to master web UI for displaying region states when listing regions > of a table > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0-beta-2 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17436) Add column to master web UI for displaying region states when listing regions of a table
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17436: - Fix Version/s: (was: 2.0.0) 2.0.0-beta-2 > Add column to master web UI for displaying region states when listing regions > of a table > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0-beta-2 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[
https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314024#comment-16314024
]
Appy edited comment on HBASE-17436 at 1/5/18 10:29 PM:
---
Wasn't able to apply the patch for sometime (to master/to some commit around
20th dec when you posted it) and it kept showing me {{error:
src/main/resources/hbase-webapps/master/table.jsp: No such file or directory}}.
Got too worked up in just trying to applying itsmile. Please follow
[this|http://hbase.apache.org/book.html#submitting.patches.create] for future
patches, will make life easier :).
After sometime, got it working by adding "a/" and "b/" to file paths in the
diff. Is there some other way too? Always fun to know what we don't already.
was (Author: appy):
Wasn't able to apply the patch for sometime (to master/to some commit around
20th dec when you posted it) and it kept showing me {{error:
src/main/resources/hbase-webapps/master/table.jsp: No such file or directory}}.
Got too worked up in just trying to applying itsmile. Please follow
[this|http://hbase.apache.org/book.html#submitting.patches.create] for future
patches, will make life easier :).
After sometime, got it working by adding "a/" and "b/" to file paths in the
diff. Is there some other way too? Would love to know what i don't already.
> Add facility to provide more information for Other Regions seen on Master UI
>
>
> Key: HBASE-17436
> URL: https://issues.apache.org/jira/browse/HBASE-17436
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
> Labels: ui, usability
> Fix For: 2.0.0
>
> Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt,
> 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch,
> HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch,
> HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch,
> HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot
> 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png,
> Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35
> PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png
>
>
> [~rpednekar] and I were looking at a case where the count displayed under
> Other Regions was high (~1200).
> Since the table page just maintains a count instead of List of region names,
> it is very difficult for user to determine which regions belong to this
> category.
> We should provide facility to provide more details for this category (LOG,
> JMX, etc).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[
https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314024#comment-16314024
]
Appy commented on HBASE-17436:
--
Wasn't able to apply the patch for sometime (to master/to some commit around
20th dec when you posted it) and it kept showing me {{error:
src/main/resources/hbase-webapps/master/table.jsp: No such file or directory}}.
Got too worked up in just trying to applying itsmile. Please follow
[this|http://hbase.apache.org/book.html#submitting.patches.create] for future
patches, will make life easier :).
After sometime, got it working by adding "a/" and "b/" to file paths in the
diff. Is there some other way too? Would love to know what i don't already.
> Add facility to provide more information for Other Regions seen on Master UI
>
>
> Key: HBASE-17436
> URL: https://issues.apache.org/jira/browse/HBASE-17436
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
> Labels: ui, usability
> Fix For: 2.0.0
>
> Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt,
> 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch,
> HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch,
> HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch,
> HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot
> 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png,
> Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35
> PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png
>
>
> [~rpednekar] and I were looking at a case where the count displayed under
> Other Regions was high (~1200).
> Since the table page just maintains a count instead of List of region names,
> it is very difficult for user to determine which regions belong to this
> category.
> We should provide facility to provide more details for this category (LOG,
> JMX, etc).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314010#comment-16314010 ] Appy commented on HBASE-19483: -- no worries then :) > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: 19483.master.011.patch, 19483.v11.patch, > 19483.v11.patch, HBASE-19483.master.001.patch, HBASE-19483.master.002.patch, > HBASE-19483.master.003.patch, HBASE-19483.master.004.patch, > HBASE-19483.master.005.patch, HBASE-19483.master.006.patch, > HBASE-19483.master.007.patch, HBASE-19483.master.008.patch, > HBASE-19483.master.009.patch, HBASE-19483.master.010.patch, > HBASE-19483.master.011.patch, HBASE-19483.master.011.patch, > HBASE-19483.master.012.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgroup / > get_rsgroup commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
[
https://issues.apache.org/jira/browse/HBASE-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313991#comment-16313991
]
Josh Elser commented on HBASE-19714:
Was not a compatibility issue -- {{getRegionsInTransition}} was marked as
Private.
Let me try to add a quick shell test and throw up a patch.
> `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on
> ClusterStatus
>
>
> Key: HBASE-19714
> URL: https://issues.apache.org/jira/browse/HBASE-19714
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
>
> {noformat}
> hbase(main):003:0> status 'detailed'
> version 2.0.0-beta-1
> ERROR: undefined method `getRegionsInTransition' for
> #
> Did you mean? get_region_states_in_transition
>getRegionStatesInTransition
> Show cluster status. Can be 'summary', 'simple', 'detailed', or
> 'replication'. The
> default is 'summary'. Examples:
> hbase> status
> hbase> status 'simple'
> hbase> status 'summary'
> hbase> status 'detailed'
> hbase> status 'replication'
> hbase> status 'replication', 'source'
> hbase> status 'replication', 'sink'
> Took 0.1814 seconds
> {noformat}
> Looks like the method is now {{getRegionStatesInTransition}} instead of
> {{getRegionsInTransition}}.
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19163) "Maximum lock count exceeded" from region server's batch processing
[
https://issues.apache.org/jira/browse/HBASE-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun updated HBASE-19163:
-
Attachment: HBASE-19163-branch-1-v001.patch
> "Maximum lock count exceeded" from region server's batch processing
> ---
>
> Key: HBASE-19163
> URL: https://issues.apache.org/jira/browse/HBASE-19163
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 1.2.7, 2.0.0-alpha-3
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-19163-branch-1-v001.patch,
> HBASE-19163-branch-1-v001.patch, HBASE-19163-master-v001.patch,
> HBASE-19163.master.001.patch, HBASE-19163.master.002.patch,
> HBASE-19163.master.004.patch, HBASE-19163.master.005.patch,
> HBASE-19163.master.006.patch, HBASE-19163.master.007.patch,
> HBASE-19163.master.008.patch, HBASE-19163.master.009.patch,
> HBASE-19163.master.009.patch, HBASE-19163.master.010.patch, unittest-case.diff
>
>
> In one of use cases, we found the following exception and replication is
> stuck.
> {code}
> 2017-10-25 19:41:17,199 WARN [hconnection-0x28db294f-shared--pool4-t936]
> client.AsyncProcess: #3, table=foo, attempt=5/5 failed=262836ops, last
> exception: java.io.IOException: java.io.IOException: Maximum lock count
> exceeded
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
> Caused by: java.lang.Error: Maximum lock count exceeded
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1327)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:5163)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3018)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2877)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:715)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2148)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> ... 3 more
> {code}
> While we are still examining the data pattern, it is sure that there are too
> many mutations in the batch against the same row, this exceeds the maximum
> 64k shared lock count and it throws an error and failed the whole batch.
> There are two approaches to solve this issue.
> 1). Let's say there are mutations against the same row in the batch, we just
> need to acquire the lock once for the same row vs to acquire the lock for
> each mutation.
> 2). We catch the error and start to process whatever it gets and loop back.
> With HBASE-17924, approach 1 seems easy to implement now.
> Create the jira and will post update/patch when investigation moving forward.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19163) "Maximum lock count exceeded" from region server's batch processing
[
https://issues.apache.org/jira/browse/HBASE-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313992#comment-16313992
]
huaxiang sun commented on HBASE-19163:
--
Run locally that the failed unittest passed. reupload the same patch.
> "Maximum lock count exceeded" from region server's batch processing
> ---
>
> Key: HBASE-19163
> URL: https://issues.apache.org/jira/browse/HBASE-19163
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 1.2.7, 2.0.0-alpha-3
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-19163-branch-1-v001.patch,
> HBASE-19163-branch-1-v001.patch, HBASE-19163-master-v001.patch,
> HBASE-19163.master.001.patch, HBASE-19163.master.002.patch,
> HBASE-19163.master.004.patch, HBASE-19163.master.005.patch,
> HBASE-19163.master.006.patch, HBASE-19163.master.007.patch,
> HBASE-19163.master.008.patch, HBASE-19163.master.009.patch,
> HBASE-19163.master.009.patch, HBASE-19163.master.010.patch, unittest-case.diff
>
>
> In one of use cases, we found the following exception and replication is
> stuck.
> {code}
> 2017-10-25 19:41:17,199 WARN [hconnection-0x28db294f-shared--pool4-t936]
> client.AsyncProcess: #3, table=foo, attempt=5/5 failed=262836ops, last
> exception: java.io.IOException: java.io.IOException: Maximum lock count
> exceeded
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
> Caused by: java.lang.Error: Maximum lock count exceeded
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1327)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:5163)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3018)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2877)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:715)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2148)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> ... 3 more
> {code}
> While we are still examining the data pattern, it is sure that there are too
> many mutations in the batch against the same row, this exceeds the maximum
> 64k shared lock count and it throws an error and failed the whole batch.
> There are two approaches to solve this issue.
> 1). Let's say there are mutations against the same row in the batch, we just
> need to acquire the lock once for the same row vs to acquire the lock for
> each mutation.
> 2). We catch the error and start to process whatever it gets and loop back.
> With HBASE-17924, approach 1 seems easy to implement now.
> Create the jira and will post update/patch when investigation moving forward.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19681) Online snapshot creation failing with missing store file
[
https://issues.apache.org/jira/browse/HBASE-19681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313972#comment-16313972
]
Anirban Roy commented on HBASE-19681:
-
Also see the following exception in region server during compaction -
2018-01-05 13:31:55,910 ERROR
[regionserver/ip-10-0-1-237.ec2.internal/10.0.1.237:16020-longCompactions-1508372592608]
regionserver.CompactSplitThread: Compaction selection failed Store = d, pri = 5
java.io.FileNotFoundException: File does not exist:
hdfs://ip-10-0-12-164.ec2.internal:8020/user/hbase/data/default/x_table/396a31774fbb8b8ed1020850e6035973/d/4a46f33587ae43d2986cbf0e45379c83
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:431)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
at
org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
at
org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
at
org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:64)
at
org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
at
org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1661)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:369)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:494)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:564)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
> Online snapshot creation failing with missing store file
>
>
> Key: HBASE-19681
> URL: https://issues.apache.org/jira/browse/HBASE-19681
> Project: HBase
> Issue Type: Bug
> Components: backup&restore, Performance, scaling, snapshots
>Affects Versions: 1.3.0
> Environment: Hadoop - 2.7.3
> HBase 1.3.0
> OS - GNU/Linux x86_64
> Cluster - Amazon Elastic Mapreduce
>Reporter: Anirban Roy
> Attachments: region-server-missing file-log.doc,
> region-server-snapshot-exception-log.doc
>
>
> We are facing problem creating online snapshot of our HBase table. The table
> contains 20TB data and receiving ~1 writes per second. The snapshot
> creating failing intermittently with error that some hfile missing, see the
> detailed output below. Once we locate the region server hosting the region
> and restart the region server, snapshot creation succeeds. It seems the
> missing hfile removed due to minor compaction, but region server still holds
> the pointer to the file.
> [hadoop@ip-10-0-12-164 ~]$ hbase shell
> HBase Shell; enter 'help' for list of supported commands.
> Type "exit" to leave the HBase Shell
> Version 1.3.0, rUnknown, Fri Feb 17 18:15:07 UTC 2017
>
> hbase(main):001:0> snapshot ‘x_table’, ‘x_snapshot’
>
> ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
> ss=x_snapshot table=x_table type=FLUSH } had an error. Procedure x_snapshot
> { waiting=[] done=[ip-10-0-9-31.ec2.internal,16020,1508372578254,
> ip-10-0-0-32.ec2.internal,16020,1508372591059,
> ip-10-0-14-221.ec2.internal,16020,1508372580873,
> ip-10-0-15-185.ec2.internal,16020,1508372588507,
> ip-10-0-9-43.ec2.internal,16020,1508372569107,
> ip-10-0-10-62.ec2.internal,16020,1512885921693,
> ip-10-0-8-216.ec2.internal,16020,1508372584133,
> ip-10-0-1-207.ec2.internal,16020,1508372580144,
> ip-10-0-0-173.ec2.internal,16020,1508372584969,
> ip-10-0-4-79.ec2.internal,16020,
[jira] [Created] (HBASE-19714) `status 'detailed'` invokes nonexistent "getRegionsInTransition" method on ClusterStatus
Josh Elser created HBASE-19714:
--
Summary: `status 'detailed'` invokes nonexistent
"getRegionsInTransition" method on ClusterStatus
Key: HBASE-19714
URL: https://issues.apache.org/jira/browse/HBASE-19714
Project: HBase
Issue Type: Bug
Components: shell
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Critical
Fix For: 2.0.0-beta-1
{noformat}
hbase(main):003:0> status 'detailed'
version 2.0.0-beta-1
ERROR: undefined method `getRegionsInTransition' for
#
Did you mean? get_region_states_in_transition
getRegionStatesInTransition
Show cluster status. Can be 'summary', 'simple', 'detailed', or 'replication'.
The
default is 'summary'. Examples:
hbase> status
hbase> status 'simple'
hbase> status 'summary'
hbase> status 'detailed'
hbase> status 'replication'
hbase> status 'replication', 'source'
hbase> status 'replication', 'sink'
Took 0.1814 seconds
{noformat}
Looks like the method is now {{getRegionStatesInTransition}} instead of
{{getRegionsInTransition}}.
FYI [~stack].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313909#comment-16313909
]
Hadoop QA commented on HBASE-19709:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
56s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 34s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 52s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19709 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904840/HBASE-19709.002.branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux fc13bdc02814 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 4c19e1f4e3 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/10904/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10904/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10904/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Guard against a ThreadPool size of 0 in CleanerChore
> -
[jira] [Commented] (HBASE-19651) Remove LimitInputStream
[ https://issues.apache.org/jira/browse/HBASE-19651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313824#comment-16313824 ] Hudson commented on HBASE-19651: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4349 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4349/]) HBASE-19651 Remove LimitInputStream (stack: rev 032fdc53de7bc1c59b2a44dda65d9fdf38eb9e3c) * (delete) hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java * (edit) NOTICE.txt * (delete) hbase-common/src/main/appended-resources/META-INF/NOTICE * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java > Remove LimitInputStream > --- > > Key: HBASE-19651 > URL: https://issues.apache.org/jira/browse/HBASE-19651 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19651.1.patch, HBASE-19651.2.patch, > HBASE-19651.3.patch, HBASE-19651.4.patch, HBASE-19651.5.patch, > HBASE-19651.6.patch > > > Let us "drink our own champagne" and use the existing Apache Commons > BoundedInputStream instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19696) Filter returning INCLUDE_AND_NEXT_COL doesn't skip remaining versions when scan has explicit columns
[ https://issues.apache.org/jira/browse/HBASE-19696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313812#comment-16313812 ] Ted Yu commented on HBASE-19696: [~anoopsamjohn]: Is the patch good by you ? > Filter returning INCLUDE_AND_NEXT_COL doesn't skip remaining versions when > scan has explicit columns > > > Key: HBASE-19696 > URL: https://issues.apache.org/jira/browse/HBASE-19696 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-1 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19696.patch, HBASE-19696_v1.patch, > HBASE-19696_v2.patch > > > INCLUDE_AND_NEXT_COL from filter doesn't skip remaining versions of the cell > if the scan has explicit columns. > This is because we use a column hint from a column tracker to prepare a cell > for seeking to next column but we are not updating column tracker with next > column when filter returns INCLUDE_AND_NEXT_COL. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313796#comment-16313796 ] Ted Yu commented on HBASE-19483: bq. lgtm. last few comments, can go in then I just saw your last comment on review board - before which I thought Guangxu has addressed your comments. Reverted. > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: 19483.master.011.patch, 19483.v11.patch, > 19483.v11.patch, HBASE-19483.master.001.patch, HBASE-19483.master.002.patch, > HBASE-19483.master.003.patch, HBASE-19483.master.004.patch, > HBASE-19483.master.005.patch, HBASE-19483.master.006.patch, > HBASE-19483.master.007.patch, HBASE-19483.master.008.patch, > HBASE-19483.master.009.patch, HBASE-19483.master.010.patch, > HBASE-19483.master.011.patch, HBASE-19483.master.011.patch, > HBASE-19483.master.012.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgroup / > get_rsgroup commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313764#comment-16313764 ] huaxiang sun commented on HBASE-18693: -- Thanks [~jingcheng.du] for the review. > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-18693.master.001.patch, > HBASE-18693.master.002.patch, HBASE-18693.master.003.patch > > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313750#comment-16313750 ] Appy commented on HBASE-19483: -- [~tedyu] Reason for the hurry please. Review is ongoing on RB. > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: 19483.master.011.patch, 19483.v11.patch, > 19483.v11.patch, HBASE-19483.master.001.patch, HBASE-19483.master.002.patch, > HBASE-19483.master.003.patch, HBASE-19483.master.004.patch, > HBASE-19483.master.005.patch, HBASE-19483.master.006.patch, > HBASE-19483.master.007.patch, HBASE-19483.master.008.patch, > HBASE-19483.master.009.patch, HBASE-19483.master.010.patch, > HBASE-19483.master.011.patch, HBASE-19483.master.011.patch, > HBASE-19483.master.012.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgroup / > get_rsgroup commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19708) Avoid NPE when the RPC listener's accept channel is closed
[
https://issues.apache.org/jira/browse/HBASE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313747#comment-16313747
]
Hudson commented on HBASE-19708:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #330 (See
[https://builds.apache.org/job/HBase-1.3-IT/330/])
HBASE-19708 Avoid NPE when the RPC listener's accept channel is closed
(apurtell: rev 09b32ca380af8fd63f665b01270a1227978b0c2f)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> Avoid NPE when the RPC listener's accept channel is closed
> --
>
> Key: HBASE-19708
> URL: https://issues.apache.org/jira/browse/HBASE-19708
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.24
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7
>
> Attachments: HBASE-19708-branch-1.patch, HBASE-19708-branch-1.patch
>
>
> Rare NPE when the listener's accept channel is closed. We serialize access to
> related state to avoid a previously fixed related NPE and need to do the same
> for {{acceptChannel}}. Seen in a 0.98 deploy but I think applicable to later
> code lines. Let me check.
> Exception in thread "MetadataRpcServer.handler=191,queue=0,port=60020"
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.getAddress(RpcServer.java:858)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.getListenerAddress(RpcServer.java:2338)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:140)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19681) Online snapshot creation failing with missing store file
[
https://issues.apache.org/jira/browse/HBASE-19681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anirban Roy updated HBASE-19681:
Attachment: region-server-snapshot-exception-log.doc
Attached the log snippet of region server where the exception is thrown upon
creating online snapshot. Can someone explain what is going on here?
> Online snapshot creation failing with missing store file
>
>
> Key: HBASE-19681
> URL: https://issues.apache.org/jira/browse/HBASE-19681
> Project: HBase
> Issue Type: Bug
> Components: backup&restore, Performance, scaling, snapshots
>Affects Versions: 1.3.0
> Environment: Hadoop - 2.7.3
> HBase 1.3.0
> OS - GNU/Linux x86_64
> Cluster - Amazon Elastic Mapreduce
>Reporter: Anirban Roy
> Attachments: region-server-missing file-log.doc,
> region-server-snapshot-exception-log.doc
>
>
> We are facing problem creating online snapshot of our HBase table. The table
> contains 20TB data and receiving ~1 writes per second. The snapshot
> creating failing intermittently with error that some hfile missing, see the
> detailed output below. Once we locate the region server hosting the region
> and restart the region server, snapshot creation succeeds. It seems the
> missing hfile removed due to minor compaction, but region server still holds
> the pointer to the file.
> [hadoop@ip-10-0-12-164 ~]$ hbase shell
> HBase Shell; enter 'help' for list of supported commands.
> Type "exit" to leave the HBase Shell
> Version 1.3.0, rUnknown, Fri Feb 17 18:15:07 UTC 2017
>
> hbase(main):001:0> snapshot ‘x_table’, ‘x_snapshot’
>
> ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
> ss=x_snapshot table=x_table type=FLUSH } had an error. Procedure x_snapshot
> { waiting=[] done=[ip-10-0-9-31.ec2.internal,16020,1508372578254,
> ip-10-0-0-32.ec2.internal,16020,1508372591059,
> ip-10-0-14-221.ec2.internal,16020,1508372580873,
> ip-10-0-15-185.ec2.internal,16020,1508372588507,
> ip-10-0-9-43.ec2.internal,16020,1508372569107,
> ip-10-0-10-62.ec2.internal,16020,1512885921693,
> ip-10-0-8-216.ec2.internal,16020,1508372584133,
> ip-10-0-1-207.ec2.internal,16020,1508372580144,
> ip-10-0-0-173.ec2.internal,16020,1508372584969,
> ip-10-0-4-79.ec2.internal,16020,1508372587161,
> ip-10-0-3-165.ec2.internal,16020,1508372593566,
> ip-10-0-14-137.ec2.internal,16020,1508372583225,
> ip-10-0-6-33.ec2.internal,16020,1508372581587,
> ip-10-0-15-199.ec2.internal,16020,1508372587478,
> ip-10-0-5-253.ec2.internal,16020,1508372581243,
> ip-10-0-1-99.ec2.internal,16020,1508372609684] }
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:354)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1058)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:61089)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by:
> org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
> ip-10-0-3-13.ec2.internal,16020,1508372563772:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
> java.io.FileNotFoundException: File does not exist:
> hdfs://ip-10-0-12-164.ec2.internal:8020/user/hbase/data/default/x_table/ecbb3aeaf7c5b1f65742deab5812362c/d/f76d8827c29244b99bf9344982956523
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:315)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:344)
> ... 6 more
> Caused by:
> org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
> java.io.FileNotFoundException: File does not exist:
> hdfs://ip-10-0-12-164.ec2.internal:8020/user/hbase/data/default/x_table/ecbb3aeaf7c5b1f65742deab5812362c/d/f76d8827c29244b99bf9344982956523
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:347)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:140)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubp
[jira] [Commented] (HBASE-19708) Avoid NPE when the RPC listener's accept channel is closed
[
https://issues.apache.org/jira/browse/HBASE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313732#comment-16313732
]
Hudson commented on HBASE-19708:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1054 (See
[https://builds.apache.org/job/HBase-1.2-IT/1054/])
HBASE-19708 Avoid NPE when the RPC listener's accept channel is closed
(apurtell: rev d48f9e4d40aacb2fb624c8cfd8ecbc83be5074fd)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> Avoid NPE when the RPC listener's accept channel is closed
> --
>
> Key: HBASE-19708
> URL: https://issues.apache.org/jira/browse/HBASE-19708
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.24
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7
>
> Attachments: HBASE-19708-branch-1.patch, HBASE-19708-branch-1.patch
>
>
> Rare NPE when the listener's accept channel is closed. We serialize access to
> related state to avoid a previously fixed related NPE and need to do the same
> for {{acceptChannel}}. Seen in a 0.98 deploy but I think applicable to later
> code lines. Let me check.
> Exception in thread "MetadataRpcServer.handler=191,queue=0,port=60020"
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.getAddress(RpcServer.java:858)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.getListenerAddress(RpcServer.java:2338)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:140)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313720#comment-16313720
]
Josh Elser commented on HBASE-19709:
Thanks boss. Will commit on QA.
FYI [~stack]
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-2
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19708) Avoid NPE when the RPC listener's accept channel is closed
[
https://issues.apache.org/jira/browse/HBASE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-19708:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.2.7
Status: Resolved (was: Patch Available)
> Avoid NPE when the RPC listener's accept channel is closed
> --
>
> Key: HBASE-19708
> URL: https://issues.apache.org/jira/browse/HBASE-19708
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.24
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 1.3.2, 1.4.1, 1.5.0, 1.2.7
>
> Attachments: HBASE-19708-branch-1.patch, HBASE-19708-branch-1.patch
>
>
> Rare NPE when the listener's accept channel is closed. We serialize access to
> related state to avoid a previously fixed related NPE and need to do the same
> for {{acceptChannel}}. Seen in a 0.98 deploy but I think applicable to later
> code lines. Let me check.
> Exception in thread "MetadataRpcServer.handler=191,queue=0,port=60020"
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.getAddress(RpcServer.java:858)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.getListenerAddress(RpcServer.java:2338)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:140)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[ https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313705#comment-16313705 ] Ted Yu commented on HBASE-19483: Integrated to master branch. There are a few rejected hunks in RSGroupAdminEndpoint.java in branch-2 Please prepare patch for branch-2. > Add proper privilege check for rsgroup commands > --- > > Key: HBASE-19483 > URL: https://issues.apache.org/jira/browse/HBASE-19483 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Guangxu Cheng > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-2 > > Attachments: 19483.master.011.patch, 19483.v11.patch, > 19483.v11.patch, HBASE-19483.master.001.patch, HBASE-19483.master.002.patch, > HBASE-19483.master.003.patch, HBASE-19483.master.004.patch, > HBASE-19483.master.005.patch, HBASE-19483.master.006.patch, > HBASE-19483.master.007.patch, HBASE-19483.master.008.patch, > HBASE-19483.master.009.patch, HBASE-19483.master.010.patch, > HBASE-19483.master.011.patch, HBASE-19483.master.011.patch, > HBASE-19483.master.012.patch > > > Currently list_rsgroups command can be executed by any user. > This is inconsistent with other list commands such as list_peers and > list_peer_configs. > We should add proper privilege check for list_rsgroups command. > privilege check should be added for get_table_rsgroup / get_server_rsgroup / > get_rsgroup commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19483) Add proper privilege check for rsgroup commands
[
https://issues.apache.org/jira/browse/HBASE-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313693#comment-16313693
]
Hadoop QA commented on HBASE-19483:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 9m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} hbase-server: The patch generated 0 new + 81
unchanged - 11 fixed = 81 total (was 92) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch hbase-mapreduce passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} hbase-thrift: The patch generated 0 new + 0
unchanged - 283 fixed = 0 total (was 283) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-rsgroup passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch hbase-it passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hbase-rest passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} root: The patch generated 0 new + 137 unchanged -
294 fixed = 137 total (was 431) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
38s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 10s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
27s{color} | {color:red} hbase-server generated 1 new + 2 unchanged - 0 fixed =
3 total (was 2) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
43s{color} | {color:red} root generated 1 new + 30 unchanged - 0 fixed = 31
total (was 30) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}157m
20s{color} | {color:green} root in the patch passed. {color} |
| {color:gre
[jira] [Commented] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313692#comment-16313692
]
Chia-Ping Tsai commented on HBASE-19709:
+1 on .002
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-2
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19709) Guard against a ThreadPool size of 0 in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19709:
---
Attachment: HBASE-19709.002.branch-2.patch
.002 Adds a debug message when we compute a number of threads that is less than
1. Slightly different implementation but the same action is taken as the first
patch.
> Guard against a ThreadPool size of 0 in CleanerChore
>
>
> Key: HBASE-19709
> URL: https://issues.apache.org/jira/browse/HBASE-19709
> Project: HBase
> Issue Type: Bug
>Reporter: Siddharth Wagle
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-beta-2
>
> Attachments: HBASE-19709.001.branch-2.patch,
> HBASE-19709.002.branch-2.patch
>
>
> Post HBASE-18309, we choose the number of threads by the following logic:
> {code}
> + /**
> + * If it is an integer and >= 1, it would be the size;
> + * if 0.0 < size <= 1.0, size would be available processors * size.
> + * Pay attention that 1.0 is different from 1, former indicates it will
> use 100% of cores,
> + * while latter will use only 1 thread for chore to scan dir.
> + */
> {code}
> [~swagle] has found on his VM that despite having two virtual processors,
> {{Runtime.getRuntime().availableProcessors()}} returns 0, which results in 0
> threads for the pool which throws an exception.
> {noformat}
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ForkJoinPool.checkParallelism(ForkJoinPool.java:2546)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2536)
> at java.util.concurrent.ForkJoinPool.(ForkJoinPool.java:2505)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:112)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.(CleanerChore.java:83)
> at
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.(LogCleaner.java:65)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1130)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:813)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:223)
> at org.apache.hadoop.hbase.master.HMaster$4.run(HMaster.java:2016)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should make sure that we take the max of {{1}} and the computed number of
> threads.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313598#comment-16313598 ] Appy edited comment on HBASE-17436 at 1/5/18 6:24 PM: -- Sg. Let me commit the ui part. I myself not familiar with rest api codebase, but maybe looking at /table/regions endpoint might help? was (Author: appy): Sg. I myself not familiar with rest api codebase, but maybe looking at /table/regions endpoint might help? > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313598#comment-16313598 ] Appy commented on HBASE-17436: -- Sg. I myself not familiar with rest api codebase, but maybe looking at /table/regions endpoint might help? > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Janos Gub > Labels: ui, usability > Fix For: 2.0.0 > > Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt, > 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch, > HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch, > HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch, > HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot > 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png, > Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35 > PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png > > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
