[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637828#comment-16637828
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #7 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/7//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/7//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/7//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21267) Spark in java examples throws 'A master URL must be set in your configuration'
[
https://issues.apache.org/jira/browse/HBASE-21267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637813#comment-16637813
]
ramkrishna.s.vasudevan commented on HBASE-21267:
[~malaskat] - You have any suggestions here.
> Spark in java examples throws 'A master URL must be set in your configuration'
> --
>
> Key: HBASE-21267
> URL: https://issues.apache.org/jira/browse/HBASE-21267
> Project: HBase
> Issue Type: Test
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Trying out some spark on java examples I found this
> 'A master URL must be set in your configuration'.
> It comes from here
> bq.SparkConf sparkConf = new
> SparkConf().setAppName("JavaHBaseBulkPutExample " + tableName);
> Adding the below line solved the problem,
> bq. SparkConf sparkConf = new SparkConf().setAppName("JavaHBaseBulkPutExample
> " + tableName).setMaster("local[2]")
> But am not sure if there is any better way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21267) Spark in java examples throws 'A master URL must be set in your configuration'
ramkrishna.s.vasudevan created HBASE-21267:
--
Summary: Spark in java examples throws 'A master URL must be set
in your configuration'
Key: HBASE-21267
URL: https://issues.apache.org/jira/browse/HBASE-21267
Project: HBase
Issue Type: Test
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Trying out some spark on java examples I found this
'A master URL must be set in your configuration'.
It comes from here
bq.SparkConf sparkConf = new
SparkConf().setAppName("JavaHBaseBulkPutExample " + tableName);
Adding the below line solved the problem,
bq. SparkConf sparkConf = new SparkConf().setAppName("JavaHBaseBulkPutExample "
+ tableName).setMaster("local[2]")
But am not sure if there is any better way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637768#comment-16637768 ] Andrew Purtell commented on HBASE-21265: Thanks for the reviews. I see the checkstyle report and will fix them upon commit. > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9 > > Attachments: HBASE-21265-branch-1.patch > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637746#comment-16637746 ] Vikas Vishwakarma commented on HBASE-21265: --- looks good +1 few minor checkstyle and whitespace warnings in the generated test report [~apurtell] > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9 > > Attachments: HBASE-21265-branch-1.patch > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637685#comment-16637685 ] Ted Yu commented on HBASE-21265: +1 > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9 > > Attachments: HBASE-21265-branch-1.patch > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21257) misspelled words.[occured -> occurred]
[
https://issues.apache.org/jira/browse/HBASE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637675#comment-16637675
]
Hadoop QA commented on HBASE-21257:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 22s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}125m
44s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
7s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21257 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941821/0001-misspelled-words.-occured-occurred.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b4e551947796 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision |
[jira] [Commented] (HBASE-21265) Split up TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637649#comment-16637649
]
Hadoop QA commented on HBASE-21265:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-rsgroup: The patch generated 10 new + 10
unchanged - 6 fixed = 20 total (was 16) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 36s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
47s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21265 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942335/HBASE-21265-branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux
[jira] [Updated] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21265: --- Fix Version/s: 1.4.9 > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9 > > Attachments: HBASE-21265-branch-1.patch > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21265) Split up TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637615#comment-16637615
]
Andrew Purtell edited comment on HBASE-21265 at 10/3/18 11:41 PM:
--
Here's a patch for branch-1. It's a simple refactor. I just moved test units
around. Patch for master and later branches would follow the same approach.
When run in suite, results look approximately like this now:
{noformat}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.715
s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin1
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.974
s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin1
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin2
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.73 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin2
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsOfflineMode
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.154 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsOfflineMode
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsBasics
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.288 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsBasics
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestEnableRSGroups
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.844 s
- in org.apache.hadoop.hbase.rsgroup.TestEnableRSGroups
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsBalance
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.899 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsBalance
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.817 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS
[INFO] Running
org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.622 s
- in
org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal
[INFO] Running
org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancer
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.874 s
- in org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancer
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 43, Failures: 0, Errors: 0, Skipped: 0
{noformat}
was (Author: apurtell):
Here's a patch for branch-1. Patch for master and later branches would follow
the same approach. When run in suite, results look approximately like this now:
{noformat}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.715
s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin1
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.974
s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin1
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin2
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.73 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin2
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsOfflineMode
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.154 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsOfflineMode
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsBasics
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.288 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsBasics
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestEnableRSGroups
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.844 s
- in org.apache.hadoop.hbase.rsgroup.TestEnableRSGroups
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsBalance
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.899 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsBalance
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.817 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS
[INFO] Running
[jira] [Commented] (HBASE-21265) Split up TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637615#comment-16637615
]
Andrew Purtell commented on HBASE-21265:
Here's a patch for branch-1. Patch for master and later branches would follow
the same approach. When run in suite, results look approximately like this now:
{noformat}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.715
s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin1
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.974
s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin1
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin2
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.73 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsAdmin2
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsOfflineMode
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.154 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsOfflineMode
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsBasics
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.288 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsBasics
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestEnableRSGroups
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.844 s
- in org.apache.hadoop.hbase.rsgroup.TestEnableRSGroups
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsBalance
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.899 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsBalance
[INFO] Running org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.817 s
- in org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS
[INFO] Running
org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.622 s
- in
org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal
[INFO] Running
org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancer
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.874 s
- in org.apache.hadoop.hbase.master.balancer.TestRSGroupBasedLoadBalancer
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 43, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Split up TestRSGroups
> -
>
> Key: HBASE-21265
> URL: https://issues.apache.org/jira/browse/HBASE-21265
> Project: HBase
> Issue Type: Task
> Components: rsgroup, test
>Affects Versions: 1.4.8
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Labels: beginner, beginners
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21265-branch-1.patch
>
>
> TestRSGroups is flaky. It is stable when run in isolation but when run as
> part of the suite with concurrent executors it can fail. The current running
> time of this unit on my dev box is ~240 seconds (4 minutes), which is far too
> much time. This unit should be broken up 5 to 8 ways, grouped by
> functionality under test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21265: --- Status: Patch Available (was: Open) > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21265-branch-1.patch > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21265: --- Attachment: HBASE-21265-branch-1.patch > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21265-branch-1.patch > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637545#comment-16637545
]
Josh Elser commented on HBASE-20952:
[~stack], spent the majority of the last three days with Ted working on the doc
[https://docs.google.com/document/d/1o552MkKq9wF3BXY2nVcsCXBAImUH6r132Cxv9WHL3D8/edit#.]
By no means does it go into exact method signatures for every single case, but
I hope the detail we've gone into is sufficient to paint a picture as to what
needs to change. Of course, please feel free to ask for more
info/details/clarity. Thanks in advance if you find the time to give it another
review.
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21257) misspelled words.[occured -> occurred]
[ https://issues.apache.org/jira/browse/HBASE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-21257: -- Status: Patch Available (was: Reopened) > misspelled words.[occured -> occurred] > -- > > Key: HBASE-21257 > URL: https://issues.apache.org/jira/browse/HBASE-21257 > Project: HBase > Issue Type: Bug >Reporter: zhanggangxue >Priority: Trivial > Labels: occured, occurred, typo > Attachments: 0001-misspelled-words.-occured-occurred.patch > > > I found some spelling errors on the master Branch.Found in [misspelled > words|https://github.com/apache/hbase/pull/91] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Summary: Not running balancer because processing dead regionservers, but empty dead rs list (was: Not running balancer because processing dead regionservers, but empty rs list, and state does not recover) > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21266) Not running balancer because processing dead regionservers, but empty rs list, and state does not recover
Andrew Purtell created HBASE-21266: -- Summary: Not running balancer because processing dead regionservers, but empty rs list, and state does not recover Key: HBASE-21266 URL: https://issues.apache.org/jira/browse/HBASE-21266 Project: HBase Issue Type: Bug Affects Versions: 1.4.8 Reporter: Andrew Purtell Fix For: 1.5.0, 1.4.9 Found during ITBLL testing. AM in master gets into a state where manual attempts from the shell to run the balancer always return false and this is printed in the master log: 2018-10-03 19:17:14,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: Not running balancer because processing dead regionserver(s): Note the empty list. This errant state did not recover without intervention by way of master restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21200) Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-21200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637289#comment-16637289 ] Toshihiro Suzuki commented on HBASE-21200: -- It looks like the QA was successful. After applying the optimization in the patch, the test in the patch finishes much faster. Could you please review the patch? [~yuzhih...@gmail.com] [~ramkrishna.s.vasude...@gmail.com] [~anoop.hbase] > Memstore flush doesn't finish because of seekToPreviousRow() in memstore > scanner. > - > > Key: HBASE-21200 > URL: https://issues.apache.org/jira/browse/HBASE-21200 > Project: HBase > Issue Type: Bug > Components: Scanners >Reporter: dongjin2193.jeon >Assignee: Toshihiro Suzuki >Priority: Major > Attachments: HBASE-21200-UT.patch, HBASE-21200.master.001.patch, > RegionServerJstack.log > > > The issue of delaying memstore flush still occurs after backport hbase-15871. > Reverse scan takes a long time to seek previous row in the memstore full of > deleted cells. > > jstack : > "MemStoreFlusher.0" #114 prio=5 os_prio=0 tid=0x7fa3d0729000 nid=0x486a > waiting on condition [0x7fa3b9b6b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xa465fe60> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > at > org.apache.hadoop.hbase.regionserver.*StoreScanner.updateReaders(StoreScanner.java:695)* > at > org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1127) > at > org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1106) > at > org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:130) > at > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2455) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2519) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2256) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2218) > at > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2110) > at > org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2036) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:501) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) > at java.lang.Thread.run(Thread.java:748) > > "RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=16020" #65 daemon prio=5 > os_prio=0 tid=0x7fa3e628 nid=0x4801 runnable [0x7fa3bd29a000] > java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.getNext(DefaultMemStore.java:780) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekInSubLists(DefaultMemStore.java:826) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seek(DefaultMemStore.java:818) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekToPreviousRow(DefaultMemStore.java:1000) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:136) > at > org.apache.hadoop.hbase.regionserver.*StoreScanner.next(StoreScanner.java:629)* > at >
[jira] [Assigned] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21265: -- Assignee: Andrew Purtell > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0 > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21265) Split up TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21265: --- Labels: beginner beginners (was: ) > Split up TestRSGroups > - > > Key: HBASE-21265 > URL: https://issues.apache.org/jira/browse/HBASE-21265 > Project: HBase > Issue Type: Task > Components: rsgroup, test >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 2.2.0 > > > TestRSGroups is flaky. It is stable when run in isolation but when run as > part of the suite with concurrent executors it can fail. The current running > time of this unit on my dev box is ~240 seconds (4 minutes), which is far too > much time. This unit should be broken up 5 to 8 ways, grouped by > functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19444) RSGroups test units cannot be concurrently executed
[ https://issues.apache.org/jira/browse/HBASE-19444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637154#comment-16637154 ] Andrew Purtell commented on HBASE-19444: See HBASE-21265 > RSGroups test units cannot be concurrently executed > --- > > Key: HBASE-19444 > URL: https://issues.apache.org/jira/browse/HBASE-19444 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Priority: Minor > > TestRSGroups and friends cannot be concurrently executed or they are very > likely to flake, failing with constraint exceptions. If executed serially all > units pass. Fix for concurrent execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21265) Split up TestRSGroups
Andrew Purtell created HBASE-21265: -- Summary: Split up TestRSGroups Key: HBASE-21265 URL: https://issues.apache.org/jira/browse/HBASE-21265 Project: HBase Issue Type: Task Components: rsgroup, test Affects Versions: 1.4.8 Reporter: Andrew Purtell Fix For: 3.0.0, 1.5.0, 2.2.0 TestRSGroups is flaky. It is stable when run in isolation but when run as part of the suite with concurrent executors it can fail. The current running time of this unit on my dev box is ~240 seconds (4 minutes), which is far too much time. This unit should be broken up 5 to 8 ways, grouped by functionality under test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-19444) RSGroups test units cannot be concurrently executed
[ https://issues.apache.org/jira/browse/HBASE-19444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-19444. Resolution: Duplicate Fix Version/s: (was: 1.4.9) (was: 2.2.0) (was: 1.5.0) (was: 3.0.0) Duping this out. Replacing with a task issue to break up TestRSGroups into smaller units. Current run time is ~240 seconds / 4 minutes and the test is only stable when run by itself. > RSGroups test units cannot be concurrently executed > --- > > Key: HBASE-19444 > URL: https://issues.apache.org/jira/browse/HBASE-19444 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Priority: Minor > > TestRSGroups and friends cannot be concurrently executed or they are very > likely to flake, failing with constraint exceptions. If executed serially all > units pass. Fix for concurrent execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.addendum.txt, 21221.v10.txt, 21221.v11.txt,
> 21221.v12.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636829#comment-16636829 ] Anoop Sam John commented on HBASE-20716: Ya one class for Unsafe. That will be the best!. Both Utils can make use of that. We have the Comparor and Converter now. The Unsafe or PureJava version does not need to keep any static singleton ref. The Comparator/Convertor bridge as such can have a ref to this interface type which refers to the singleton best instance. That loading of Unsafe version class also should be via FQCN.Or else we have the UnsafeAvailChecker now.. Can just make use of that (static methods on that) to know whether Unsafe is available and can be used and based on that can instantiate the Convertor/Comparator class. > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: Sahil Aggarwal >Priority: Critical > Labels: beginner > Attachments: HBASE-20716.master.001.patch, > HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, > HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png > > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21264) Namespace Quota : Split after exceeding maxregions- regions went to other regions state and never come back online
[
https://issues.apache.org/jira/browse/HBASE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandra Sekhar updated HBASE-21264:
---
Fix Version/s: 2.1.1
> Namespace Quota : Split after exceeding maxregions- regions went to other
> regions state and never come back online
> --
>
> Key: HBASE-21264
> URL: https://issues.apache.org/jira/browse/HBASE-21264
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Chandra Sekhar
>Priority: Critical
> Fix For: 2.1.1
>
>
> {{As my namespace quota (max regions = 5) exceeds, when i perform split
> region more than 4 times, regions went to Other Regions state and never come
> back online. It was showing 0 Online Regions}}
> {{Scenario:}}
> {noformat}
> 1. Create namespace with quota - 5 max regions
> 2. Create table
> 3. Split the table more than 5 split keys
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21264) Namespace Quota : Split after exceeding maxregions- regions went to other regions state and never come back online
Chandra Sekhar created HBASE-21264:
--
Summary: Namespace Quota : Split after exceeding maxregions-
regions went to other regions state and never come back online
Key: HBASE-21264
URL: https://issues.apache.org/jira/browse/HBASE-21264
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: Chandra Sekhar
{{As my namespace quota (max regions = 5) exceeds, when i perform split region
more than 4 times, regions went to Other Regions state and never come back
online. It was showing 0 Online Regions}}
{{Scenario:}}
{noformat}
1. Create namespace with quota - 5 max regions
2. Create table
3. Split the table more than 5 split keys
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-21222) [amv2] Closing region on a non-existent server creates STUCK regions
[
https://issues.apache.org/jira/browse/HBASE-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-21222.
---
Resolution: Duplicate
Resolving as duplicate of HBASE-21259. It goes into more depth on why we
suspend/get STUCK.
> [amv2] Closing region on a non-existent server creates STUCK regions
>
>
> Key: HBASE-21222
> URL: https://issues.apache.org/jira/browse/HBASE-21222
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> Ran into this one where a Region had been on a server but after a bunch of
> crashing and meddling in Master Proc WALs, any attempt at unassign has the
> procedure fail (see below) and then report the region as STUCK.
> I broke the lock w/ new hbck2 tooling and then tried to offline again but
> same thing happened. Bug. Fix.
> {code}
> 2018-09-22 18:36:41,900 INFO
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch
> pid=138650, ppid=121871, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true; UnassignProcedure
> table=IntegrationTestBigLinkedList_20180614072614,
> region=51cdade76ca7217ec191f39e5f56c61c,
> server=vd0637.halxg.cloudera.com,22101,1537397969558; rit=CLOSING,
> location=vd0637.halxg.cloudera.com,22101,1537397969558
> 2018-09-22 18:36:41,899 INFO
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler:
> pid=138646, ppid=121871, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
> UnassignProcedure table=IntegrationTestBigLinkedList_20180614072614,
> region=0780467efe4c5901887fb12bfa406fa7,
> server=vc1228.halxg.cloudera.com,22101,1537578279837 checking lock on
> 0780467efe4c5901887fb12bfa406fa7
> 2018-09-22 18:36:41,900 WARN
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Remote
> call failed vd0637.halxg.cloudera.com,22101,1537397969558; pid=138650,
> ppid=121871, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true;
> UnassignProcedure table=IntegrationTestBigLinkedList_20180614072614,
> region=51cdade76ca7217ec191f39e5f56c61c,
> server=vd0637.halxg.cloudera.com,22101,1537397969558; rit=CLOSING,
> location=vd0637.halxg.cloudera.com,22101,1537397969558;
> exception=NoServerDispatchException
> org.apache.hadoop.hbase.procedure2.NoServerDispatchException:
> vd0637.halxg.cloudera.com,22101,1537397969558; pid=138650, ppid=121871,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; UnassignProcedure
> table=IntegrationTestBigLinkedList_20180614072614,
> region=51cdade76ca7217ec191f39e5f56c61c,
> server=vd0637.halxg.cloudera.com,22101,1537397969558
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:177)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:277)
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:202)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:370)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:924)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1684)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1471)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:77)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1983)
> 2018-09-22 18:36:41,903 WARN
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure: Expiring
> vd0637.halxg.cloudera.com,22101,1537397969558, pid=138650, ppid=121871,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; UnassignProcedure
> table=IntegrationTestBigLinkedList_20180614072614,
> region=51cdade76ca7217ec191f39e5f56c61c,
> server=vd0637.halxg.cloudera.com,22101,1537397969558 rit=CLOSING,
> location=vd0637.halxg.cloudera.com,22101,1537397969558;
> exception=NoServerDispatchException
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636688#comment-16636688 ] stack commented on HBASE-20716: --- [~anoop.hbase] See above boss. I took a look at patch: Thats good fixes in FuzzyRowFilter Could fix this: return UnsafeComparer.INSTSANCE; ... static final UnsafeComparer INSTSANCE = new UnsafeComparer(); ... bq. Since in BBUtils we don't use Unsafe directly, are you suggesting removing any direct ref in Bytes class too? What are you thinking? One class used by Bytes and BBUtil that does all unsafe? Or something else? It would be sweet if just one conduit to unsafe. Thanks. > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: Sahil Aggarwal >Priority: Critical > Labels: beginner > Attachments: HBASE-20716.master.001.patch, > HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, > HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png > > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636611#comment-16636611
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #6 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/6/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/6//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/6//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/6//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/6//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-20472) InfoServer doesnot honour any acl set by the admin
[
https://issues.apache.org/jira/browse/HBASE-20472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain resolved HBASE-20472.
Resolution: Duplicate
> InfoServer doesnot honour any acl set by the admin
> --
>
> Key: HBASE-20472
> URL: https://issues.apache.org/jira/browse/HBASE-20472
> Project: HBase
> Issue Type: Bug
> Components: security, UI
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-20472.master.001.patch
>
>
> The adminsAcl property can be used to restrict access to certain sections of
> the web UI only to a particular set of users/groups. But in hbase, adminAcl
> variable for InfoServer is always null, rendering it to not honour any acl
> set by the admin. In fact I could not find any property in hbase to specify
> acl list for web server.
> *Analysis*:
> * *InfoSever* object forgets(?) to set any *adminAcl* in the builder object
> for http server.
> {code:java}
> public InfoServer(String name, String bindAddress, int port, boolean findPort,
> final Configuration c) {
> .
> .
>
> HttpServer.Builder builder =
> new org.apache.hadoop.hbase.http.HttpServer.Builder();
> .
> .
> this.httpServer = builder.build();
> }{code}
> [See InfoServer
> constructor|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/InfoServer.java#L55]
> * http server retreives a null value and sets it as adminsAcl, which is
> passed to *createWebAppContext*() method
> {code:java}
> private HttpServer(final Builder b) throws IOException {
> .
> .
> .
> this.adminsAcl = b.adminsAcl;
> this.webAppContext = createWebAppContext(b.name, b.conf, adminsAcl,
> appDir);
>
> .
> .
> }{code}
> [See L527
> HttpServer.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L527]
> * This method next sets *ADMIN_ACL* attribute for the servlet context to
> *null*
> {code:java}
> private static WebAppContext createWebAppContext(String name,
> Configuration conf, AccessControlList adminsAcl, final String appDir) {
> WebAppContext ctx = new WebAppContext();
> .
> .
> ctx.getServletContext().setAttribute(ADMINS_ACL, adminsAcl);
> .
> .
> }
> {code}
> * Now any page having *HttpServer.hasAdministratorAccess*() will allow
> access to everyone, making this check useless.
> {code:java}
> @Override
> public void doGet(HttpServletRequest request, HttpServletResponse response
> ) throws ServletException, IOException {
> // Do the authorization
> if (!HttpServer.hasAdministratorAccess(getServletContext(), request,
> response)) {
> return;
> }
> .
> .
> }{code}
> [For example See L104
> LogLevel.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/log/LogLevel.java#L104]
> * *hasAdministratorAccess()* checks for the following and returns true, in
> any case as *ADMIN_ACL* is always *null*
> {code:java}
> public static boolean hasAdministratorAccess(
> ServletContext servletContext, HttpServletRequest request,
> HttpServletResponse response) throws IOException {
> .
> .
> if (servletContext.getAttribute(ADMINS_ACL) != null &&
> !userHasAdministratorAccess(servletContext, remoteUser)) {
> response.sendError(HttpServletResponse.SC_UNAUTHORIZED, "User "
> + remoteUser + " is unauthorized to access this page.");
>return false;
> }
> return true;
> }{code}
> [See line 1196 in
> HttpServer|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L1196]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-20472) InfoServer doesnot honour any acl set by the admin
[
https://issues.apache.org/jira/browse/HBASE-20472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain reopened HBASE-20472:
> InfoServer doesnot honour any acl set by the admin
> --
>
> Key: HBASE-20472
> URL: https://issues.apache.org/jira/browse/HBASE-20472
> Project: HBase
> Issue Type: Bug
> Components: security, UI
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-20472.master.001.patch
>
>
> The adminsAcl property can be used to restrict access to certain sections of
> the web UI only to a particular set of users/groups. But in hbase, adminAcl
> variable for InfoServer is always null, rendering it to not honour any acl
> set by the admin. In fact I could not find any property in hbase to specify
> acl list for web server.
> *Analysis*:
> * *InfoSever* object forgets(?) to set any *adminAcl* in the builder object
> for http server.
> {code:java}
> public InfoServer(String name, String bindAddress, int port, boolean findPort,
> final Configuration c) {
> .
> .
>
> HttpServer.Builder builder =
> new org.apache.hadoop.hbase.http.HttpServer.Builder();
> .
> .
> this.httpServer = builder.build();
> }{code}
> [See InfoServer
> constructor|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/InfoServer.java#L55]
> * http server retreives a null value and sets it as adminsAcl, which is
> passed to *createWebAppContext*() method
> {code:java}
> private HttpServer(final Builder b) throws IOException {
> .
> .
> .
> this.adminsAcl = b.adminsAcl;
> this.webAppContext = createWebAppContext(b.name, b.conf, adminsAcl,
> appDir);
>
> .
> .
> }{code}
> [See L527
> HttpServer.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L527]
> * This method next sets *ADMIN_ACL* attribute for the servlet context to
> *null*
> {code:java}
> private static WebAppContext createWebAppContext(String name,
> Configuration conf, AccessControlList adminsAcl, final String appDir) {
> WebAppContext ctx = new WebAppContext();
> .
> .
> ctx.getServletContext().setAttribute(ADMINS_ACL, adminsAcl);
> .
> .
> }
> {code}
> * Now any page having *HttpServer.hasAdministratorAccess*() will allow
> access to everyone, making this check useless.
> {code:java}
> @Override
> public void doGet(HttpServletRequest request, HttpServletResponse response
> ) throws ServletException, IOException {
> // Do the authorization
> if (!HttpServer.hasAdministratorAccess(getServletContext(), request,
> response)) {
> return;
> }
> .
> .
> }{code}
> [For example See L104
> LogLevel.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/log/LogLevel.java#L104]
> * *hasAdministratorAccess()* checks for the following and returns true, in
> any case as *ADMIN_ACL* is always *null*
> {code:java}
> public static boolean hasAdministratorAccess(
> ServletContext servletContext, HttpServletRequest request,
> HttpServletResponse response) throws IOException {
> .
> .
> if (servletContext.getAttribute(ADMINS_ACL) != null &&
> !userHasAdministratorAccess(servletContext, remoteUser)) {
> response.sendError(HttpServletResponse.SC_UNAUTHORIZED, "User "
> + remoteUser + " is unauthorized to access this page.");
>return false;
> }
> return true;
> }{code}
> [See line 1196 in
> HttpServer|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L1196]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.
[ https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636513#comment-16636513 ] Nihal Jain commented on HBASE-18822: Ping, all watchers/reviewers! Can we have a discussion regarding the above comments.? > Create table for peer cluster automatically when creating table in source > cluster of using namespace replication. > - > > Key: HBASE-18822 > URL: https://issues.apache.org/jira/browse/HBASE-18822 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0-alpha-2 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-18822.v1.patch, HBASE-18822.v1.patch > > > In our cluster of using namespace replication, we always forget to create > table in peer cluster, which lead to replication get stuck. > We have implemented the feature in our cluster: create table for peer > cluster automatically when creating table in source cluster of using > namespace replication. > > I'm not sure if someone else needs this feature, so create an issue here for > discussing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20472) InfoServer doesnot honour any acl set by the admin
[
https://issues.apache.org/jira/browse/HBASE-20472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20472:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> InfoServer doesnot honour any acl set by the admin
> --
>
> Key: HBASE-20472
> URL: https://issues.apache.org/jira/browse/HBASE-20472
> Project: HBase
> Issue Type: Bug
> Components: security, UI
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-20472.master.001.patch
>
>
> The adminsAcl property can be used to restrict access to certain sections of
> the web UI only to a particular set of users/groups. But in hbase, adminAcl
> variable for InfoServer is always null, rendering it to not honour any acl
> set by the admin. In fact I could not find any property in hbase to specify
> acl list for web server.
> *Analysis*:
> * *InfoSever* object forgets(?) to set any *adminAcl* in the builder object
> for http server.
> {code:java}
> public InfoServer(String name, String bindAddress, int port, boolean findPort,
> final Configuration c) {
> .
> .
>
> HttpServer.Builder builder =
> new org.apache.hadoop.hbase.http.HttpServer.Builder();
> .
> .
> this.httpServer = builder.build();
> }{code}
> [See InfoServer
> constructor|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/InfoServer.java#L55]
> * http server retreives a null value and sets it as adminsAcl, which is
> passed to *createWebAppContext*() method
> {code:java}
> private HttpServer(final Builder b) throws IOException {
> .
> .
> .
> this.adminsAcl = b.adminsAcl;
> this.webAppContext = createWebAppContext(b.name, b.conf, adminsAcl,
> appDir);
>
> .
> .
> }{code}
> [See L527
> HttpServer.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L527]
> * This method next sets *ADMIN_ACL* attribute for the servlet context to
> *null*
> {code:java}
> private static WebAppContext createWebAppContext(String name,
> Configuration conf, AccessControlList adminsAcl, final String appDir) {
> WebAppContext ctx = new WebAppContext();
> .
> .
> ctx.getServletContext().setAttribute(ADMINS_ACL, adminsAcl);
> .
> .
> }
> {code}
> * Now any page having *HttpServer.hasAdministratorAccess*() will allow
> access to everyone, making this check useless.
> {code:java}
> @Override
> public void doGet(HttpServletRequest request, HttpServletResponse response
> ) throws ServletException, IOException {
> // Do the authorization
> if (!HttpServer.hasAdministratorAccess(getServletContext(), request,
> response)) {
> return;
> }
> .
> .
> }{code}
> [For example See L104
> LogLevel.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/log/LogLevel.java#L104]
> * *hasAdministratorAccess()* checks for the following and returns true, in
> any case as *ADMIN_ACL* is always *null*
> {code:java}
> public static boolean hasAdministratorAccess(
> ServletContext servletContext, HttpServletRequest request,
> HttpServletResponse response) throws IOException {
> .
> .
> if (servletContext.getAttribute(ADMINS_ACL) != null &&
> !userHasAdministratorAccess(servletContext, remoteUser)) {
> response.sendError(HttpServletResponse.SC_UNAUTHORIZED, "User "
> + remoteUser + " is unauthorized to access this page.");
>return false;
> }
> return true;
> }{code}
> [See line 1196 in
> HttpServer|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L1196]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
