[GitHub] [hbase] Apache9 opened a new pull request #1022: HBASE-23680 RegionProcedureStore missing cleaning of hfile archive

[GitBox]

Apache9 opened a new pull request #1022: HBASE-23680 RegionProcedureStore 
missing cleaning of hfile archive
URL: https://github.com/apache/hbase/pull/1022
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013394#comment-17013394
 ] 

Hudson commented on HBASE-22514:


Results for branch HBASE-22514
[build #238 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/238/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/238//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/238//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/238//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
>  Issue Type: Umbrella
>  Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch, 
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public 
> we need to use java api  RSGroupAdminClient  to manager RSG 
> so  RSGroupAdminClient should be public
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (HBASE-23045) currentPath may be stitched in a loop in replication source code.

[Viraj Jasani (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013387#comment-17013387
 ] 

Viraj Jasani edited comment on HBASE-23045 at 1/11/20 5:41 AM:
---

[~gk_coder] merged branch-1.2 PR. The patch is not directly applied to 
branch-1, it requires very minor change. Although we can make the change and 
merge, if you could confirm the fix with your manual tests, that would be great 
since we don't have unit test for this change.

Also, please note that branch-1.2 is EOL'ed long back.

branch-1 patch:
{code:java}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index 5acb70922f..a26bbd00cd 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
@@ -938,8 +938,9 @@ public class ReplicationSource extends Thread implements 
ReplicationSourceInterf
         Path p = rs.getPath();
         FileStatus[] logs = fs.listStatus(p);
         for (FileStatus log : logs) {
-          p = new Path(p, log.getPath().getName());
-          if (p.getName().equals(path.getName())) {
+          String logName = log.getPath().getName();
+          if (logName.equals(path.getName())) {
+            p = new Path(p, log.getPath().getName());
             LOG.info("Log " + p.getName() + " found at " + p);
             return p;
           }{code}
cc: [~stack]


was (Author: vjasani):
[~gk_coder] merged branch-1.2 PR. The patch is not directly applied to 
branch-1, it requires very minor change. Although we can make the change and 
merge, if you could confirm the fix with your manual tests, that would be great 
since we don't have unit test for this change.

Also, please note that branch-1.2 is EOL'ed long back.

branch-1 patch:
{code:java}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index 5acb70922f..a26bbd00cd 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
@@ -938,8 +938,9 @@ public class ReplicationSource extends Thread implements 
ReplicationSourceInterf
         Path p = rs.getPath();
         FileStatus[] logs = fs.listStatus(p);
         for (FileStatus log : logs) {
-          p = new Path(p, log.getPath().getName());
-          if (p.getName().equals(path.getName())) {
+          String logName = log.getPath().getName();
+          if (logName.equals(path.getName())) {
+            p = new Path(p, log.getPath().getName());
             LOG.info("Log " + p.getName() + " found at " + p);
             return p;
           }{code}

>  currentPath may be stitched in a loop in replication source code.
> --
>
> Key: HBASE-23045
> URL: https://issues.apache.org/jira/browse/HBASE-23045
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.2.6.1
>Reporter: kangkang.guo
>Assignee: kangkang.guo
>Priority: Critical
> Fix For: 1.2.6.1, 1.6.0, 1.3.7, 1.4.13
>
> Attachments: HBASE-23045.branch-1.2.0001.patch
>
>
> When the openReader encounters a FileNotFoundException, we may go to all 
> possible directories to find the current hlog. When found, the path may be 
> wrong, and it is looped together.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23045) currentPath may be stitched in a loop in replication source code.

[Viraj Jasani (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013387#comment-17013387
 ] 

Viraj Jasani commented on HBASE-23045:
--

[~gk_coder] merged branch-1.2 PR. The patch is not directly applied to 
branch-1, it requires very minor change. Although we can make the change and 
merge, if you could confirm the fix with your manual tests, that would be great 
since we don't have unit test for this change.

Also, please note that branch-1.2 is EOL'ed long back.

branch-1 patch:
{code:java}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index 5acb70922f..a26bbd00cd 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
@@ -938,8 +938,9 @@ public class ReplicationSource extends Thread implements 
ReplicationSourceInterf
         Path p = rs.getPath();
         FileStatus[] logs = fs.listStatus(p);
         for (FileStatus log : logs) {
-          p = new Path(p, log.getPath().getName());
-          if (p.getName().equals(path.getName())) {
+          String logName = log.getPath().getName();
+          if (logName.equals(path.getName())) {
+            p = new Path(p, log.getPath().getName());
             LOG.info("Log " + p.getName() + " found at " + p);
             return p;
           }{code}

>  currentPath may be stitched in a loop in replication source code.
> --
>
> Key: HBASE-23045
> URL: https://issues.apache.org/jira/browse/HBASE-23045
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.2.6.1
>Reporter: kangkang.guo
>Assignee: kangkang.guo
>Priority: Critical
> Fix For: 1.2.6.1, 1.6.0, 1.3.7, 1.4.13
>
> Attachments: HBASE-23045.branch-1.2.0001.patch
>
>
> When the openReader encounters a FileNotFoundException, we may go to all 
> possible directories to find the current hlog. When found, the path may be 
> wrong, and it is looped together.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on issue #1018: HBASE-23652 Move the unsupported procedure type check before migratin…

[GitBox]

Apache-HBase commented on issue #1018: HBASE-23652 Move the unsupported 
procedure type check before migratin…
URL: https://github.com/apache/hbase/pull/1018#issuecomment-573283398
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 24s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   6m  5s |  master passed  |
   | +1 :green_heart: |  compile  |   1m  1s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 33s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m  5s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  master passed  |
   | +0 :ok: |  spotbugs  |   4m 51s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   4m 49s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 33s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 58s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 58s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m 30s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  shadedjars  |   5m  3s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  17m 52s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 or 3.1.2.  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |   5m  9s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 162m 59s |  hbase-server in the patch failed.  |
   | +1 :green_heart: |  asflicense  |   0m 31s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 228m  7s |   |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | 
hadoop.hbase.master.assignment.TestRegionMoveAndAbandon |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1018 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile |
   | uname | Linux 8d9ed280d2c1 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 
10:36:11 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1018/out/precommit/personality/provided.sh
 |
   | git revision | master / 4ed466966c |
   | Default Java | 1.8.0_181 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/2/artifact/out/patch-unit-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/2/testReport/
 |
   | Max. process+thread count | 4823 (vs. ulimit of 1) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/2/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] virajjasani merged pull request #1006: HBASE-23045.currentPath may be stitched in a loop in replication source code.

[GitBox]

virajjasani merged pull request #1006: HBASE-23045.currentPath may be stitched 
in a loop in replication source code.
URL: https://github.com/apache/hbase/pull/1006
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (HBASE-23045) currentPath may be stitched in a loop in replication source code.

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani updated HBASE-23045:
-
Fix Version/s: 1.4.13
   1.3.7
   1.6.0

>  currentPath may be stitched in a loop in replication source code.
> --
>
> Key: HBASE-23045
> URL: https://issues.apache.org/jira/browse/HBASE-23045
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.2.6.1
>Reporter: kangkang.guo
>Assignee: kangkang.guo
>Priority: Critical
> Fix For: 1.2.6.1, 1.6.0, 1.3.7, 1.4.13
>
> Attachments: HBASE-23045.branch-1.2.0001.patch
>
>
> When the openReader encounters a FileNotFoundException, we may go to all 
> possible directories to find the current hlog. When found, the path may be 
> wrong, and it is looped together.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-23045) currentPath may be stitched in a loop in replication source code.

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani reopened HBASE-23045:
--

Not yet committed to any branch. Reopening.

>  currentPath may be stitched in a loop in replication source code.
> --
>
> Key: HBASE-23045
> URL: https://issues.apache.org/jira/browse/HBASE-23045
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.2.6.1
>Reporter: kangkang.guo
>Assignee: kangkang.guo
>Priority: Critical
> Fix For: 1.2.6.1
>
> Attachments: HBASE-23045.branch-1.2.0001.patch
>
>
> When the openReader encounters a FileNotFoundException, we may go to all 
> possible directories to find the current hlog. When found, the path may be 
> wrong, and it is looped together.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] virajjasani commented on issue #1006: HBASE-23045.currentPath may be stitched in a loop in replication source code.

[GitBox]

virajjasani commented on issue #1006: HBASE-23045.currentPath may be stitched 
in a loop in replication source code.
URL: https://github.com/apache/hbase/pull/1006#issuecomment-573282022
 
 
   Can you please also confirm @G302 ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] virajjasani commented on issue #1006: HBASE-23045.currentPath may be stitched in a loop in replication source code.

[GitBox]

virajjasani commented on issue #1006: HBASE-23045.currentPath may be stitched 
in a loop in replication source code.
URL: https://github.com/apache/hbase/pull/1006#issuecomment-573281891
 
 
   This seems applicable to all active branch-1s only and not to branch-2+
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (HBASE-23662) Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani updated HBASE-23662:
-
Hadoop Flags: Reviewed
  Resolution: Fixed
  Status: Resolved  (was: Patch Available)

Pushed to master. Backports not required since the usage removal is applicable 
to release 3.0.0.

> Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
> 
>
> Key: HBASE-23662
> URL: https://issues.apache.org/jira/browse/HBASE-23662
> Project: HBase
>  Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0
>
>
> We have deprecated HColumnDescriptor and HTableDescriptor some time back in 
> 2.0.0. Although there are many use-cases still using them, we can start 
> replacing them with builder pattern: ColumnFamilyDescriptorBuilder and 
> TableDescriptorBuilder.
> In an attempt to do so, starting with replacing specific constructor of 
> HColumnDescriptor:
> {code:java}
> public HTableDescriptor(final TableName name) {
>   this(new ModifyableTableDescriptor(name));
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] virajjasani closed pull request #1003: HBASE-23662 : Replace HColumnDescriptor(String cf) with ColumnFamilyD…

[GitBox]

virajjasani closed pull request #1003: HBASE-23662 : Replace 
HColumnDescriptor(String cf) with ColumnFamilyD…
URL: https://github.com/apache/hbase/pull/1003
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] virajjasani commented on issue #1003: HBASE-23662 : Replace HColumnDescriptor(String cf) with ColumnFamilyD…

[GitBox]

virajjasani commented on issue #1003: HBASE-23662 : Replace 
HColumnDescriptor(String cf) with ColumnFamilyD…
URL: https://github.com/apache/hbase/pull/1003#issuecomment-573280746
 
 
   Thanks for the review @petersomogyi @HorizonNet 
   Will merge this in a while


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] virajjasani commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters…

[GitBox]

virajjasani commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on 
meta table (fix bloomfilters…
URL: https://github.com/apache/hbase/pull/1012#issuecomment-573280665
 
 
   @saintstack TestProcedurePriority failure doesn't seem relevant here


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (HBASE-23349) Low refCount preventing archival of compacted away files

[Viraj Jasani (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013344#comment-17013344
 ] 

Viraj Jasani edited comment on HBASE-23349 at 1/11/20 4:30 AM:
---

I agree on considering removal of refCount system based on your suggestions 
[~apurtell] [~larsh]

However, I am just trying to give one chance to consider both points:
 # Perf improvement as part of HBASE-13082
 # Scanner reset during compaction if required(config based)

Tried to use volatile enum(NONE, FLUSH, COMPACTION) instead of 2 volatile 
booleans for Scanner.next(), seek() calls to not let perf degrade for normal 
scans. Hence, if archival is not happening correctly, we can notify open 
scanners and reset KV Heap in the next(), seek() runs. However, whether next() 
has to reset KVHeap is something that can be determined based on volatile enum 
value which would be set while notifying scanners.

[https://github.com/apache/hbase/pull/939] with some tests for Scanner reset 
during compaction and successful archival thereafter. You might prefer this 
over direct removal of refCount system.

Considering refCount presence in HBase for some time, someone might have 
started building some system(alert, recovery etc) based on refCount usecase. In 
fact, we have also done auto region reopen etc but other users might have built 
some other usecases too.


was (Author: vjasani):
I agree on considering removal of refCount system based on your suggestions 
[~apurtell] [~larsh]

However, I am just trying to give one chance to consider both points:
 # Perf improvement as part of HBASE-13082
 # Scanner reset during compaction if required(config based)

Tried to use volatile enum(NONE, FLUSH, COMPACTION) instead of 2 volatile 
booleans for Scanner.next(), seek() calls to not let perf degrade for normal 
scans. Hence, if archival is not happening correctly, we can notify open 
scanners and reset KV Heap in the next(), seek() runs. However, whether next() 
has to reset KVHeap is something that can be determined based on volatile enum 
value which would be set while notifying scanners.

[https://github.com/apache/hbase/pull/939] with some tests for Scanner reset 
during compaction and successful archival thereafter.

Considering refCount presence in HBase for some time, someone might have 
started building some system(alert, recovery etc) based on refCount usecase. In 
fact, we have also done auto region reopen etc but other users might have built 
some other usecases too.

> Low refCount preventing archival of compacted away files
> 
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> We have observed that refCount on compacted away store files as low as 1 is 
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file 
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
>  because of either isCompactedAway=true or file has reference, 
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread) 
> gracefully resolve reader lock issue by resetting ongoing scanners to start 
> pointing to new store files instead of compacted away store files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (HBASE-23279) Switch default block encoding to ROW_INDEX_V1

[Viraj Jasani (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013351#comment-17013351
 ] 

Viraj Jasani edited comment on HBASE-23279 at 1/11/20 4:25 AM:
---

[~stack] Size difference has been one obstacle behind completion of this task.

Since GetClosest is deprecated some time back, I didn’t spend much time and 
commented the test in the patch. Reg other test failures, I tried considering 
other encodings to see diffs and almost they were failing for few others. 
 However, I don’t recall reason behind TestProcedurePriority failure, jenkins 
link is also expired. Let me spend some time on this next week may be?


was (Author: vjasani):
[~stack] Size difference has been one obstacle behind completion of this task.

Since GetClosest is deprecated some time back, I didn’t spend much time and 
commented the test in the patch. Reg other test failures, I tried considering 
other compressions to see diffs and almost they were failing for few others. 
However, I don’t recall reason behind TestProcedurePriority failure, jenkins 
link is also expired. Let me spend some time on this next week may be?

> Switch default block encoding to ROW_INDEX_V1
> -
>
> Key: HBASE-23279
> URL: https://issues.apache.org/jira/browse/HBASE-23279
> Project: HBase
>  Issue Type: Wish
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Lars Hofhansl
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-23279.master.000.patch, 
> HBASE-23279.master.001.patch, HBASE-23279.master.002.patch, 
> HBASE-23279.master.003.patch
>
>
> Currently we set both block encoding and compression to NONE.
> ROW_INDEX_V1 has many advantages and (almost) no disadvantages (the hfiles 
> are slightly larger about 3% or so). I think that would a better default than 
> NONE.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters…

[GitBox]

Apache-HBase commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on 
meta table (fix bloomfilters…
URL: https://github.com/apache/hbase/pull/1012#issuecomment-573278378
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 44s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   7m 12s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 15s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 43s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m 58s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 42s |  master passed  |
   | +0 :ok: |  spotbugs  |   5m 19s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   5m 18s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   6m 26s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 13s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 13s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m 55s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  shadedjars  |   5m 59s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  21m  1s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 or 3.1.2.  |
   | +1 :green_heart: |  javadoc  |   0m 43s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |   5m 27s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 159m 42s |  hbase-server in the patch failed.  |
   | +1 :green_heart: |  asflicense  |   0m 37s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 234m 23s |   |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hbase.master.procedure.TestProcedurePriority |
   |   | hadoop.hbase.regionserver.TestGetClosestAtOrBefore |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/3/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1012 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile |
   | uname | Linux 13315c15d56d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1012/out/precommit/personality/provided.sh
 |
   | git revision | master / 4ed466966c |
   | Default Java | 1.8.0_181 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/3/artifact/out/patch-unit-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/3/testReport/
 |
   | Max. process+thread count | 5183 (vs. ulimit of 1) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/3/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-23279) Switch default block encoding to ROW_INDEX_V1

[Viraj Jasani (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013351#comment-17013351
 ] 

Viraj Jasani commented on HBASE-23279:
--

[~stack] Size difference has been one obstacle behind completion of this task.

Since GetClosest is deprecated some time back, I didn’t spend much time and 
commented the test in the patch. Reg other test failures, I tried considering 
other compressions to see diffs and almost they were failing for few others. 
However, I don’t recall reason behind TestProcedurePriority failure, jenkins 
link is also expired. Let me spend some time on this next week may be?

> Switch default block encoding to ROW_INDEX_V1
> -
>
> Key: HBASE-23279
> URL: https://issues.apache.org/jira/browse/HBASE-23279
> Project: HBase
>  Issue Type: Wish
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Lars Hofhansl
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-23279.master.000.patch, 
> HBASE-23279.master.001.patch, HBASE-23279.master.002.patch, 
> HBASE-23279.master.003.patch
>
>
> Currently we set both block encoding and compression to NONE.
> ROW_INDEX_V1 has many advantages and (almost) no disadvantages (the hfiles 
> are slightly larger about 3% or so). I think that would a better default than 
> NONE.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23680) RegionProcedureStore missing cleaning of hfile archive

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013349#comment-17013349
 ] 

Duo Zhang commented on HBASE-23680:
---

OK, things here are a bit different. All the files are stored on the WAL 
filesystem for procedures, for speed, and usually it will not cost too much as 
the procedures will be deleted soon. So this introduce a problem, that, the 
HFile archive directory is on the HFile filesystem, by design, it could be 
different from the WAL fileysystem.

So for now, I prefer we just delete the HFile archives directly, as we can 
rebuild the procedure data once we have enough WAL files. Introducing a 
separated cleaner chore looks overkill, and introduces more configs, which is 
not convenience to users.

Shout if you have other good choices.

Thanks.

> RegionProcedureStore missing cleaning of hfile archive
> --
>
> Key: HBASE-23680
> URL: https://issues.apache.org/jira/browse/HBASE-23680
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.3.0
>
>
> See tail of parent issue. The new RegionProcedureStore accumulates deleted 
> hfiles in its local archive dir. Needs a cleaner like the one that watches 
> over /hbase/archive.
> Is there a problem clearning the new $masterproc$ files from the oldWALs too? 
> These seem to stick around also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-7811) Cleanup code to use HRegionFileSystem, remove getDir and fs operations

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-7811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013345#comment-17013345
 ] 

Duo Zhang commented on HBASE-7811:
--

OK, so this is the issue where we want to remove the HStore.getStoreHomedir. 
Makes sense. So let me keep the Deprecated annotations.

If anyone has time, it will be good to continue these issues, I think this is 
the correct direction.

> Cleanup code to use HRegionFileSystem, remove getDir and fs operations
> --
>
> Key: HBASE-7811
> URL: https://issues.apache.org/jira/browse/HBASE-7811
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Matteo Bertozzi
>Priority: Minor
>
> At this point all the server code should rely on MasterFileSystem and 
> HRegionFileSystem, remove the getDir methods and get rid of all the new 
> Path() and fs.listStatus() still in use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23349) Low refCount preventing archival of compacted away files

[Viraj Jasani (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013344#comment-17013344
 ] 

Viraj Jasani commented on HBASE-23349:
--

I agree on considering removal of refCount system based on your suggestions 
[~apurtell] [~larsh]

However, I am just trying to give one chance to consider both points:
 # Perf improvement as part of HBASE-13082
 # Scanner reset during compaction if required(config based)

Tried to use volatile enum(NONE, FLUSH, COMPACTION) instead of 2 volatile 
booleans for Scanner.next(), seek() calls to not let perf degrade for normal 
scans. Hence, if archival is not happening correctly, we can notify open 
scanners and reset KV Heap in the next(), seek() runs. However, whether next() 
has to reset KVHeap is something that can be determined based on volatile enum 
value which would be set while notifying scanners.

[https://github.com/apache/hbase/pull/939] with some tests for Scanner reset 
during compaction and successful archival thereafter.

Considering refCount presence in HBase for some time, someone might have 
started building some system(alert, recovery etc) based on refCount usecase. In 
fact, we have also done auto region reopen etc but other users might have built 
some other usecases too.

> Low refCount preventing archival of compacted away files
> 
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> We have observed that refCount on compacted away store files as low as 1 is 
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file 
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
>  because of either isCompactedAway=true or file has reference, 
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread) 
> gracefully resolve reader lock issue by resetting ongoing scanners to start 
> pointing to new store files instead of compacted away store files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on issue #1011: HBASE-23055 Alter hbase:meta

[GitBox]

Apache-HBase commented on issue #1011: HBASE-23055 Alter hbase:meta
URL: https://github.com/apache/hbase/pull/1011#issuecomment-573270424
 
 
   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 32s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 36s |  master passed  |
   | +1 :green_heart: |  compile  |   0m 59s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 38s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m  7s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  master passed  |
   | +0 :ok: |  spotbugs  |   4m 35s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   4m 34s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 20s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 57s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 57s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m 19s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  shadedjars  |   4m 40s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  15m 52s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 or 3.1.2.  |
   | +1 :green_heart: |  javadoc  |   0m 36s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |   4m 31s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 153m 32s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  asflicense  |   0m 37s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 212m 59s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1011/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1011 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile |
   | uname | Linux 22d8c4b69881 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 
16:55:30 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1011/out/precommit/personality/provided.sh
 |
   | git revision | master / 8ca614857d |
   | Default Java | 1.8.0_181 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1011/1/testReport/
 |
   | Max. process+thread count | 5355 (vs. ulimit of 1) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1011/1/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] joshelser commented on issue #1019: HBASE-23679 Use new FileSystem objects during bulk loads

[GitBox]

joshelser commented on issue #1019: HBASE-23679 Use new FileSystem objects 
during bulk loads
URL: https://github.com/apache/hbase/pull/1019#issuecomment-573270351
 
 
   Thinking about this before I leave it for the night -- my initial thought 
was that as long as, in SecureBulkLoadManager, we managed the lifecycle of our 
FileSystem correctly, we'd be fine. But, the reality is that other parts of 
HBase are (inadvertently) relying on a FileSystem being cached.
   
   The smaller fix may be to re-add the caching, keep the reference counting, 
but correct the clean-up logic.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] Apache-HBase commented on issue #1021: Backport HBASE-23653 Expose content of meta table in web ui

[GitBox]

Apache-HBase commented on issue #1021: Backport HBASE-23653 Expose content of 
meta table in web ui
URL: https://github.com/apache/hbase/pull/1021#issuecomment-573270181
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 59s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 
8 new or modified test files.  |
   ||| _ branch-2 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 24s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   5m 14s |  branch-2 passed  |
   | +1 :green_heart: |  compile  |   3m  0s |  branch-2 passed  |
   | +1 :green_heart: |  checkstyle  |   4m 27s |  branch-2 passed  |
   | +1 :green_heart: |  shadedjars  |   4m 13s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 43s |  branch-2 passed  |
   | +0 :ok: |  spotbugs  |   3m  3s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  16m 52s |  branch-2 passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 22s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 52s |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 57s |  the patch passed  |
   | +1 :green_heart: |  javac  |   2m 57s |  the patch passed  |
   | -1 :x: |  checkstyle  |   1m 24s |  hbase-server: The patch generated 10 
new + 217 unchanged - 1 fixed = 227 total (was 218)  |
   | -1 :x: |  checkstyle  |   2m 23s |  root: The patch generated 10 new + 255 
unchanged - 1 fixed = 265 total (was 256)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  xml  |   0m  3s |  The patch has no ill-formed XML 
file.  |
   | +1 :green_heart: |  shadedjars  |   4m 13s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  15m 17s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 or 3.1.2.  |
   | +1 :green_heart: |  javadoc  |   3m 46s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |  17m 40s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  |  48m 24s |  root in the patch failed.  |
   | +1 :green_heart: |  asflicense  |   0m 56s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 149m 42s |   |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hbase.master.webapp.TestMetaBrowserNoCluster |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1021/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1021 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile xml |
   | uname | Linux e6e95e65ca0b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1021/out/precommit/personality/provided.sh
 |
   | git revision | branch-2 / 76a6fe6a8b |
   | Default Java | 1.8.0_181 |
   | checkstyle | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1021/1/artifact/out/diff-checkstyle-hbase-server.txt
 |
   | checkstyle | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1021/1/artifact/out/diff-checkstyle-root.txt
 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1021/1/artifact/out/patch-unit-root.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1021/1/testReport/
 |
   | Max. process+thread count | 706 (vs. ulimit of 1) |
   | modules | C: hbase-client hbase-server . U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1021/1/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git 

[jira] [Updated] (HBASE-23650) lack of HMaster parameters in the SnapshotLogCleaner leads to NLP

[sangjiaqi (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sangjiaqi updated HBASE-23650:
--
Attachment: HBASE-23650.patch

> lack of HMaster parameters in the SnapshotLogCleaner leads to NLP
> -
>
> Key: HBASE-23650
> URL: https://issues.apache.org/jira/browse/HBASE-23650
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.11
>Reporter: sangjiaqi
>Priority: Major
> Fix For: 1.2.11
>
> Attachments: HBASE-23650.patch
>
>
> NullPointException occurs when the SnapshotCleaner calls the init function to 
> initialize the HMaster parameter. Similar to the construction method of 
> HFileCleaner, because the construction method of LogCleaner did not have 
> HMaster parameter, leading to errors. Therefore, the construction method of 
> LogCleaner solves the problem by adding HMaster parameters.
> The Log:
> {code:java}
> java.lang.NullPointerExceptionjava.lang.NullPointerException at 
> org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner.init(SnapshotLogCleaner.java:74)
>  at 
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotAndLogCleaner.testSnapshotLogCleaning(TestSnapshotAndLogCleaner.java:68){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-23650) lack of HMaster parameters in the SnapshotLogCleaner leads to NLP

[sangjiaqi (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sangjiaqi updated HBASE-23650:
--
Attachment: (was: HBASE-23650.patch)

> lack of HMaster parameters in the SnapshotLogCleaner leads to NLP
> -
>
> Key: HBASE-23650
> URL: https://issues.apache.org/jira/browse/HBASE-23650
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.11
>Reporter: sangjiaqi
>Priority: Major
> Fix For: 1.2.11
>
> Attachments: HBASE-23650.patch
>
>
> NullPointException occurs when the SnapshotCleaner calls the init function to 
> initialize the HMaster parameter. Similar to the construction method of 
> HFileCleaner, because the construction method of LogCleaner did not have 
> HMaster parameter, leading to errors. Therefore, the construction method of 
> LogCleaner solves the problem by adding HMaster parameters.
> The Log:
> {code:java}
> java.lang.NullPointerExceptionjava.lang.NullPointerException at 
> org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner.init(SnapshotLogCleaner.java:74)
>  at 
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotAndLogCleaner.testSnapshotLogCleaning(TestSnapshotAndLogCleaner.java:68){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23680) RegionProcedureStore missing cleaning of hfile archive

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013328#comment-17013328
 ] 

Duo Zhang commented on HBASE-23680:
---

You have the archive directory set up sir? [~stack]

It is a bit strange. After checking the code, it is done by a chore called 
CompactedHFilesDischarger and we do not schedule this chore here. So you 
restarted your HMaster several times? OK this is possible, when opening and 
closing regions we will archived the compacted files.

Since there is no reader for the RegionProcedureStore after restart, I think we 
can just call HStore.closeAndArchiveCompactedFiles after compaction to archive 
the store files, to the archive directory, and then move them all to the global 
archive directory.

> RegionProcedureStore missing cleaning of hfile archive
> --
>
> Key: HBASE-23680
> URL: https://issues.apache.org/jira/browse/HBASE-23680
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.3.0
>
>
> See tail of parent issue. The new RegionProcedureStore accumulates deleted 
> hfiles in its local archive dir. Needs a cleaner like the one that watches 
> over /hbase/archive.
> Is there a problem clearning the new $masterproc$ files from the oldWALs too? 
> These seem to stick around also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] joshelser edited a comment on issue #1019: HBASE-23679 Use new FileSystem objects during bulk loads

[GitBox]

joshelser edited a comment on issue #1019: HBASE-23679 Use new FileSystem 
objects during bulk loads
URL: https://github.com/apache/hbase/pull/1019#issuecomment-573258755
 
 
   ```
   2020-01-11 00:15:00,797 WARN  
[RpcServer.default.FPBQ.Fifo.handler=99,queue=9,port=16020] fs.FileSystem: 
Caching new filesystem: -279427062
   java.lang.Exception
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3365)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at 
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:761)
at 
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5958)
at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:264)
at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:233)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:233)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2338)
at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42004)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
   ```
   
   Looks like this isn't quite sufficient. Another leak (albeit, much slower) 
coming here. Need to do more to push down that DFS instance we made and use 
that until we move the files into the final location.
   
   Added some debug to FileSystem.java to see the above. Testing is just done 
via IntegrationTestBulkLoad with high number of loops but small chain length.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] joshelser edited a comment on issue #1019: HBASE-23679 Use new FileSystem objects during bulk loads

[GitBox]

joshelser edited a comment on issue #1019: HBASE-23679 Use new FileSystem 
objects during bulk loads
URL: https://github.com/apache/hbase/pull/1019#issuecomment-573269130
 
 
   Ugh, there's a few of these where, down in HStore, HRegion, and even the WAL 
code (ugh), which is all invoked via bulk load where we do a 
`FileSystem.get(conf)` or `path.getFileSystem(conf)`. All of them will leak a 
FileSystem instance with the SBLM changes in 2.x.
   
   ```
   2020-01-11 01:42:31,080 WARN  
[RpcServer.default.FPBQ.Fifo.handler=97,queue=7,port=16020] fs.FileSystem: 
Caching new filesystem: -1042984133
   java.lang.Exception
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3365)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at 
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getServerNameFromWALDirectoryName(AbstractFSWALProvider.java:330)
at 
org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks.reorderBlocks(HFileSystem.java:426)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:378)
at com.sun.proxy.$Proxy20.getBlockLocations(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:862)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:851)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:840)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1004)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:322)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:334)
at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:164)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
at 
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:125)
at 
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:102)
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:254)
at 
org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:367)
at 
org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:475)
at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:690)
at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:683)
at 
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:854)
at 
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6057)
at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:264)
at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:233)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:233)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2338)
at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42004)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
   ```
   
   The fix is to just push down the FileSystem or use one that is already 
created, but this gets tricky in some places. Will need to step back from this 
all and see if there's a better way to do this.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] joshelser commented on issue #1019: HBASE-23679 Use new FileSystem objects during bulk loads

[GitBox]

joshelser commented on issue #1019: HBASE-23679 Use new FileSystem objects 
during bulk loads
URL: https://github.com/apache/hbase/pull/1019#issuecomment-573269130
 
 
   Ugh, there's a few of these where, down in HStore, HRegion, and even the WAL 
code (ugh), which is all invoked via bulk load where we do a 
`FileSystem.get(conf)` or `path.getFileSystem(conf)`. All of them will leak a 
FileSystem instance with the SBLM changes in 2.x.
   
   ```
   ctr-e141-1563959304486-133915-01-08: 2020-01-11 01:42:31,080 WARN  
[RpcServer.default.FPBQ.Fifo.handler=97,queue=7,port=16020] fs.FileSystem: 
Caching new filesystem: -1042984133
   ctr-e141-1563959304486-133915-01-08: java.lang.Exception
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3365)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getServerNameFromWALDirectoryName(AbstractFSWALProvider.java:330)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks.reorderBlocks(HFileSystem.java:426)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:378)
   ctr-e141-1563959304486-133915-01-08: at 
com.sun.proxy.$Proxy20.getBlockLocations(Unknown Source)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:862)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:851)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:840)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1004)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:322)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:334)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:164)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:125)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:102)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:254)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:367)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:475)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:690)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:683)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:854)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6057)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:264)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:233)
   ctr-e141-1563959304486-133915-01-08: at 
java.security.AccessController.doPrivileged(Native Method)
   ctr-e141-1563959304486-133915-01-08: at 
javax.security.auth.Subject.doAs(Subject.java:360)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
   ctr-e141-1563959304486-133915-01-08: at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:233)
   

[GitHub] [hbase] Apache-HBase commented on issue #1019: HBASE-23679 Use new FileSystem objects during bulk loads

[GitBox]

Apache-HBase commented on issue #1019: HBASE-23679 Use new FileSystem objects 
during bulk loads
URL: https://github.com/apache/hbase/pull/1019#issuecomment-573268948
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 32s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 33s |  master passed  |
   | +1 :green_heart: |  compile  |   0m 56s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 24s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   4m 39s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  master passed  |
   | +0 :ok: |  spotbugs  |   4m 25s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   4m 22s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 56s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 59s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 59s |  the patch passed  |
   | -1 :x: |  checkstyle  |   1m 24s |  hbase-server: The patch generated 2 
new + 206 unchanged - 0 fixed = 208 total (was 206)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  shadedjars  |   4m 39s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  15m 35s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 or 3.1.2.  |
   | +1 :green_heart: |  javadoc  |   0m 36s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |   4m 29s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 155m 44s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  asflicense  |   0m 37s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 213m 29s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1019/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1019 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile |
   | uname | Linux 4121411bd04c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1019/out/precommit/personality/provided.sh
 |
   | git revision | master / 8ca614857d |
   | Default Java | 1.8.0_181 |
   | checkstyle | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1019/1/artifact/out/diff-checkstyle-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1019/1/testReport/
 |
   | Max. process+thread count | 5823 (vs. ulimit of 1) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1019/1/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta

[GitBox]

Apache9 commented on a change in pull request #1011: HBASE-23055 Alter 
hbase:meta
URL: https://github.com/apache/hbase/pull/1011#discussion_r365489207
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
 ##
 @@ -54,8 +53,20 @@
 // TODO: Make this a guava Service
 @InterfaceAudience.Private
 public class TableStateManager {
-
   private static final Logger LOG = 
LoggerFactory.getLogger(TableStateManager.class);
+
+  /**
+   * All table state is kept in hbase:meta except that of hbase:meta itself.
 
 Review comment:
   I know splittable meta is not implemented by this one but I think we should 
prepare for it? Not adding new difficulty, by introducing more 'there is only 
one meta region' assumptions...
   
   And I do not think we need to disable a table before altering it. The issue 
here is named 'Altering meta', not 'disabling meta', so let's try remove the 
disable/enable stuff and just implement the alter?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta

[GitBox]

Apache9 commented on a change in pull request #1011: HBASE-23055 Alter 
hbase:meta
URL: https://github.com/apache/hbase/pull/1011#discussion_r365488937
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RejectReplicationRequestStateChecker.java
 ##
 @@ -1,4 +1,6 @@
-/**
+/*
+ * Copyright The Apache Software Foundation
 
 Review comment:
   I mean the 'Copyright The Apache Software Foundation'.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-23637) Generate CHANGES.md and RELEASENOTES.md for 2.2.3

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013317#comment-17013317
 ] 

Hudson commented on HBASE-23637:


Results for branch branch-2.2
[build #750 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Generate CHANGES.md and RELEASENOTES.md for 2.2.3
> -
>
> Key: HBASE-23637
> URL: https://issues.apache.org/jira/browse/HBASE-23637
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Priority: Major
> Fix For: 2.2.3
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013316#comment-17013316
 ] 

Hudson commented on HBASE-23659:


Results for branch branch-2.2
[build #750 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> BaseLoadBalancer#wouldLowerAvailability should consider region replicas
> ---
>
> Key: HBASE-23659
> URL: https://issues.apache.org/jira/browse/HBASE-23659
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Found this issue when try to fix the flaky unit test  TestRegionReplicaSplit. 
> It may fail as 
> java.lang.AssertionError: Splitted regions should not be assigned to same 
> region server.
> See 
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/].
>  
> Now  wouldLowerAvailability method only consider the primary regions. The 
> replica region can't assign to same server with primary region. But can be 
> assigned to same server with other replica regions. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013315#comment-17013315
 ] 

Hudson commented on HBASE-23601:


Results for branch branch-2.2
[build #750 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/750//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> OutputSink.WriterThread exception gets stuck and repeated indefinietly
> --
>
> Key: HBASE-23601
> URL: https://issues.apache.org/jira/browse/HBASE-23601
> Project: HBase
>  Issue Type: Bug
>  Components: read replicas
>Affects Versions: 2.2.2
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.9, 2.2.4
>
>
> When a WriterThread runs into an exception (ie: NotServingRegionException), 
> the exception is stored in the controller. It is never removed and can not be 
> overwritten either.
>  
> {code:java}
> public void run()  {
>   try {
> doRun();
>   } catch (Throwable t) {
> LOG.error("Exiting thread", t);
> controller.writerThreadError(t);
>   }
> }{code}
> Thanks to this every time PipelineController.checkForErrors() is called the 
> same old exception is rethrown.
>  
> For example in RegionReplicaReplicationEndpoint.replicate there is a while 
> loop that does the actual replicating. Every time it loops, it calls 
> checkForErrors(), catches the rethrown exception, logs it but does nothing 
> about it. This results in ~2GB log files in ~5min in my experience.
>  
> My proposal would be to clean up the stored exception when it reaches 
> RegionReplicaReplicationEndpoint.replicate and make sure we restart the 
> WriterThread that died throwing it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on issue #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

Apache-HBase commented on issue #1020: HBASE-23653 Expose content of meta table 
in web ui
URL: https://github.com/apache/hbase/pull/1020#issuecomment-573265347
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 58s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  The patch appears to include 
8 new or modified test files.  |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 46s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   5m 14s |  master passed  |
   | +1 :green_heart: |  compile  |   2m 58s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   2m 36s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   4m 37s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   3m 48s |  master passed  |
   | +0 :ok: |  spotbugs  |   3m 49s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |  19m  2s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 22s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 57s |  the patch passed  |
   | +1 :green_heart: |  compile  |   3m  0s |  the patch passed  |
   | +1 :green_heart: |  javac  |   3m  0s |  the patch passed  |
   | -1 :x: |  checkstyle  |   2m 33s |  root: The patch generated 10 new + 242 
unchanged - 1 fixed = 252 total (was 243)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  xml  |   0m  3s |  The patch has no ill-formed XML 
file.  |
   | +1 :green_heart: |  shadedjars  |   4m 35s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  15m 42s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 or 3.1.2.  |
   | +1 :green_heart: |  javadoc  |   3m 50s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |  19m 42s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  |  47m 56s |  root in the patch failed.  |
   | +1 :green_heart: |  asflicense  |   0m 57s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 151m 46s |   |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hbase.master.webapp.TestMetaBrowserNoCluster |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1020/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1020 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile xml |
   | uname | Linux e612dcd65782 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 
16:55:30 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1020/out/precommit/personality/provided.sh
 |
   | git revision | master / 8ca614857d |
   | Default Java | 1.8.0_181 |
   | checkstyle | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1020/1/artifact/out/diff-checkstyle-root.txt
 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1020/1/artifact/out/patch-unit-root.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1020/1/testReport/
 |
   | Max. process+thread count | 706 (vs. ulimit of 1) |
   | modules | C: hbase-client hbase-server . U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1020/1/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (HBASE-23680) RegionProcedureStore missing cleaning of hfile archive

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang reassigned HBASE-23680:
-

Assignee: Duo Zhang

> RegionProcedureStore missing cleaning of hfile archive
> --
>
> Key: HBASE-23680
> URL: https://issues.apache.org/jira/browse/HBASE-23680
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.3.0
>
>
> See tail of parent issue. The new RegionProcedureStore accumulates deleted 
> hfiles in its local archive dir. Needs a cleaner like the one that watches 
> over /hbase/archive.
> Is there a problem clearning the new $masterproc$ files from the oldWALs too? 
> These seem to stick around also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HBASE-23680) RegionProcedureStore missing cleaning of hfile archive

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-23680 started by Duo Zhang.
-
> RegionProcedureStore missing cleaning of hfile archive
> --
>
> Key: HBASE-23680
> URL: https://issues.apache.org/jira/browse/HBASE-23680
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.3.0
>
>
> See tail of parent issue. The new RegionProcedureStore accumulates deleted 
> hfiles in its local archive dir. Needs a cleaner like the one that watches 
> over /hbase/archive.
> Is there a problem clearning the new $masterproc$ files from the oldWALs too? 
> These seem to stick around also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013312#comment-17013312
 ] 

Duo Zhang commented on HBASE-23326:
---

I think you'd better change it to a small value first to see if it works? BTW, 
there is a UT to confirm this...

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23055) Alter hbase:meta

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013311#comment-17013311
 ] 

Duo Zhang commented on HBASE-23055:
---

Let me explain my point. In general, the procedure store change is emergency, 
and 'altering meta, splittable meta' are important but not emergency.

I've seen people sending emails to the mailing list about broken RIT procedures 
hang the whole system and can not recover. Usually the solution is to remove 
all the content under the MasterProcWALs and try restart again, and then 
manually fix everything, this is almost a mission-impossible for junior users. 
And proc-v2 is one of the most important feature for 2.x, so we have to try our 
best to stablize it on branch-2.

And for meta table, yes, we all know that the single meta region is a bottle 
neck on large cluster but at least, it does not introduce damage to all the 2.x 
users right? And it is not likely that we could implement splittable meta 
quickly on branch-2. So I define this type of things "important but not 
emergency", That means, we have plenty of time to polish our design, think of 
the future, set the road map and milestones, and finally have a (almost) 
perfect solution for splittable meta.

Back to the patch here, yes, I think it is a bit like the RegionProcedureStore. 
In fact, if it is not emergency, I would like to refactoring the HRegion class 
and make it decouple with HRegionServer, and then the code for 
RegionProcedureStore will be much cleaner. But it is not likely that we could 
finish this in the near soon so I've done some dirty hacks to make it work. But 
as I explained above, I do not think 'altering meta' needs the dirty hacks such 
as transient meta state, modifying meta from client directly. And who has the 
permission to modify meta table? And we also removed the checks of system table 
so users can also modify system tables now. Have we considered this critically? 
How about ACLs? In general, It is not a big deal to not get this in 2.3.0 or 
even 2.4.0. We should have a clear road map on how these feature are 
implemented and released.

Thanks.

> Alter hbase:meta
> 
>
> Key: HBASE-23055
> URL: https://issues.apache.org/jira/browse/HBASE-23055
> Project: HBase
>  Issue Type: Task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> hbase:meta is currently hardcoded. Its schema cannot be change.
> This issue is about allowing edits to hbase:meta schema. It will allow our 
> being able to set encodings such as the block-with-indexes which will help 
> quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first 
> step on road to being able to split meta.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Michael Stack (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013303#comment-17013303
 ] 

Michael Stack commented on HBASE-23326:
---

Ok. 7 days default. Will keep an eye on it.

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013300#comment-17013300
 ] 

Duo Zhang commented on HBASE-23326:
---

See TimeToLiveProcedureWALCleaner.

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365474032
 
 

 ##
 File path: hbase-server/src/main/resources/hbase-webapps/master/table.jsp
 ##
 @@ -129,6 +151,7 @@
   pageContext.setAttribute("pageTitle", pageTitle);
   AsyncConnection connection = 
ConnectionFactory.createAsyncConnection(master.getConfiguration()).get();
 
 Review comment:
   Not your change, but creating a new connection every time this page loads 
seems a little heavy? Why not reuse the async connection created at the master 
startup? (master.getAsyncConnection()).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365475813
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/webapp/MetaBrowser.java
 ##
 @@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.webapp;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+import javax.servlet.http.HttpServletRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.apache.hadoop.hbase.client.AsyncTable;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.ScanResultConsumer;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.PrefixFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import 
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import 
org.apache.hbase.thirdparty.io.netty.handler.codec.http.QueryStringEncoder;
+
+/**
+ * A support class for the "Meta Entries" section in
+ * {@code resources/hbase-webapps/master/table.jsp}.
+ */
+@InterfaceAudience.Private
+public class MetaBrowser implements Iterable {
+  private static final Logger logger = 
LoggerFactory.getLogger(MetaBrowser.class);
+
+  public static final String NAME_PARAM = "name";
+  public static final String SCAN_LIMIT_PARAM = "scan_limit";
+  public static final String SCAN_REGION_STATE_PARAM = "scan_region_state";
+  public static final String SCAN_START_PARAM = "scan_start";
+  public static final String SCAN_TABLE_PARAM = "scan_table";
+
+  public static final int SCAN_LIMIT_DEFAULT = 10;
+  public static final int SCAN_LIMIT_MAX = 10_000;
+
+  private final AsyncConnection connection;
+  private final HttpServletRequest request;
+  private final ExecutorService pool;
+  private final List errorMessages;
+  private final String name;
+  private final Integer scanLimit;
+  private final RegionState.State scanRegionState;
+  private final byte[] scanStart;
+  private final TableName scanTable;
+
+  public MetaBrowser(final AsyncConnection connection, final 
HttpServletRequest request) {
+this.connection = connection;
+this.request = request;
+this.pool = buildThreadPool();
+this.errorMessages = new LinkedList<>();
+this.name = resolveName(request);
+this.scanLimit = resolveScanLimit(request);
+this.scanRegionState = resolveScanRegionState(request);
+this.scanStart = resolveScanStart(request);
+this.scanTable = resolveScanTable(request);
+  }
+
+  public List getErrorMessages() {
+return errorMessages;
+  }
+
+  public String getName() {
+return name;
+  }
+
+  public Integer getScanLimit() {
+return scanLimit;
+  }
+
+  public byte[] getScanStart() {
+return scanStart;
+  }
+
+  public RegionState.State getScanRegionState() {
+return scanRegionState;
+  }
+
+  public TableName getScanTable() {
+return scanTable;
+  }
+
+  @Override
+  public Iterator iterator() {
+return limitIterator();
+  }
+
+  public LimitIterator limitIterator() {
+logger.debug("initiating meta scan, {}", this);
+
+final AsyncTable asyncTable =
+  

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365476199
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/webapp/MetaBrowser.java
 ##
 @@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.webapp;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+import javax.servlet.http.HttpServletRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.apache.hadoop.hbase.client.AsyncTable;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.ScanResultConsumer;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.PrefixFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import 
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import 
org.apache.hbase.thirdparty.io.netty.handler.codec.http.QueryStringEncoder;
+
+/**
+ * A support class for the "Meta Entries" section in
+ * {@code resources/hbase-webapps/master/table.jsp}.
+ */
+@InterfaceAudience.Private
+public class MetaBrowser implements Iterable {
+  private static final Logger logger = 
LoggerFactory.getLogger(MetaBrowser.class);
+
+  public static final String NAME_PARAM = "name";
+  public static final String SCAN_LIMIT_PARAM = "scan_limit";
+  public static final String SCAN_REGION_STATE_PARAM = "scan_region_state";
+  public static final String SCAN_START_PARAM = "scan_start";
+  public static final String SCAN_TABLE_PARAM = "scan_table";
+
+  public static final int SCAN_LIMIT_DEFAULT = 10;
+  public static final int SCAN_LIMIT_MAX = 10_000;
+
+  private final AsyncConnection connection;
+  private final HttpServletRequest request;
+  private final ExecutorService pool;
+  private final List errorMessages;
+  private final String name;
+  private final Integer scanLimit;
+  private final RegionState.State scanRegionState;
+  private final byte[] scanStart;
+  private final TableName scanTable;
+
+  public MetaBrowser(final AsyncConnection connection, final 
HttpServletRequest request) {
+this.connection = connection;
+this.request = request;
+this.pool = buildThreadPool();
+this.errorMessages = new LinkedList<>();
+this.name = resolveName(request);
+this.scanLimit = resolveScanLimit(request);
+this.scanRegionState = resolveScanRegionState(request);
+this.scanStart = resolveScanStart(request);
+this.scanTable = resolveScanTable(request);
+  }
+
+  public List getErrorMessages() {
+return errorMessages;
+  }
+
+  public String getName() {
+return name;
+  }
+
+  public Integer getScanLimit() {
+return scanLimit;
+  }
+
+  public byte[] getScanStart() {
+return scanStart;
+  }
+
+  public RegionState.State getScanRegionState() {
+return scanRegionState;
+  }
+
+  public TableName getScanTable() {
+return scanTable;
+  }
+
+  @Override
+  public Iterator iterator() {
+return limitIterator();
+  }
+
+  public LimitIterator limitIterator() {
+logger.debug("initiating meta scan, {}", this);
+
+final AsyncTable asyncTable =
+  

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365477051
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/webapp/MetaBrowser.java
 ##
 @@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.webapp;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+import javax.servlet.http.HttpServletRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.apache.hadoop.hbase.client.AsyncTable;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.ScanResultConsumer;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.PrefixFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import 
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import 
org.apache.hbase.thirdparty.io.netty.handler.codec.http.QueryStringEncoder;
+
+/**
+ * A support class for the "Meta Entries" section in
+ * {@code resources/hbase-webapps/master/table.jsp}.
+ */
+@InterfaceAudience.Private
+public class MetaBrowser implements Iterable {
+  private static final Logger logger = 
LoggerFactory.getLogger(MetaBrowser.class);
+
+  public static final String NAME_PARAM = "name";
+  public static final String SCAN_LIMIT_PARAM = "scan_limit";
+  public static final String SCAN_REGION_STATE_PARAM = "scan_region_state";
+  public static final String SCAN_START_PARAM = "scan_start";
+  public static final String SCAN_TABLE_PARAM = "scan_table";
+
+  public static final int SCAN_LIMIT_DEFAULT = 10;
+  public static final int SCAN_LIMIT_MAX = 10_000;
+
+  private final AsyncConnection connection;
+  private final HttpServletRequest request;
+  private final ExecutorService pool;
+  private final List errorMessages;
+  private final String name;
+  private final Integer scanLimit;
+  private final RegionState.State scanRegionState;
+  private final byte[] scanStart;
+  private final TableName scanTable;
+
+  public MetaBrowser(final AsyncConnection connection, final 
HttpServletRequest request) {
+this.connection = connection;
+this.request = request;
+this.pool = buildThreadPool();
+this.errorMessages = new LinkedList<>();
+this.name = resolveName(request);
+this.scanLimit = resolveScanLimit(request);
+this.scanRegionState = resolveScanRegionState(request);
+this.scanStart = resolveScanStart(request);
+this.scanTable = resolveScanTable(request);
+  }
+
+  public List getErrorMessages() {
+return errorMessages;
+  }
+
+  public String getName() {
+return name;
+  }
+
+  public Integer getScanLimit() {
+return scanLimit;
+  }
+
+  public byte[] getScanStart() {
+return scanStart;
+  }
+
+  public RegionState.State getScanRegionState() {
+return scanRegionState;
+  }
+
+  public TableName getScanTable() {
+return scanTable;
+  }
+
+  @Override
+  public Iterator iterator() {
+return limitIterator();
+  }
+
+  public LimitIterator limitIterator() {
+logger.debug("initiating meta scan, {}", this);
+
+final AsyncTable asyncTable =
+  

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365481059
 
 

 ##
 File path: 
hbase-server/src/test/java/org/apache/hadoop/hbase/TestClusterRule.java
 ##
 @@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import java.io.IOException;
+import java.util.concurrent.CompletableFuture;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.junit.Rule;
+import org.junit.rules.ExternalResource;
+
+/**
+ * A {@link Rule} that manages an instance of the {@link MiniHBaseCluster}.
+ */
+public class TestClusterRule extends ExternalResource {
+  private final HBaseTestingUtility testingUtility;
+  private final StartMiniClusterOption miniClusterOptions;
+
+  private MiniHBaseCluster miniCluster;
+
+  public TestClusterRule() {
+this(StartMiniClusterOption.builder().build());
+  }
+
+  public TestClusterRule(final StartMiniClusterOption miniClusterOptions) {
+this.testingUtility = new HBaseTestingUtility();
+this.miniClusterOptions = miniClusterOptions;
+  }
+
+  public CompletableFuture createConnection() {
+if (miniCluster == null) { throw new IllegalStateException("test cluster 
not initialized"); }
 
 Review comment:
   nit: You'll probably run into checkstyle issues with inline if blocks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365480952
 
 

 ##
 File path: 
hbase-server/src/test/java/org/apache/hadoop/hbase/ClearUserNamespacesAndTablesRule.java
 ##
 @@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.StringJoiner;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hbase.client.AsyncAdmin;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.junit.Rule;
+import org.junit.rules.ExternalResource;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link Rule} that clears all user tables and namespaces before the test 
executes.
+ */
+public class ClearUserNamespacesAndTablesRule extends ExternalResource {
+  private static final Logger logger =
+LoggerFactory.getLogger(ClearUserNamespacesAndTablesRule.class);
+
+  private final Supplier connectionSupplier;
 
 Review comment:
   May be add a comment that this class doesn't close the connection by the end 
of it and needs to chained with TestConnectionRule which does it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365468451
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/webapp/LimitIterator.java
 ##
 @@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.webapp;
+
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.collect.Iterators;
+
+/**
+ * An {@link Iterator} over {@code delegate} that limits results to the first 
{@code limit}
 
 Review comment:
   Any reason not to use Iterators.limit() from guava?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365480306
 
 

 ##
 File path: 
hbase-server/src/test/java/org/apache/hadoop/hbase/ClearUserNamespacesAndTablesRule.java
 ##
 @@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.StringJoiner;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hbase.client.AsyncAdmin;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.junit.Rule;
+import org.junit.rules.ExternalResource;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link Rule} that clears all user tables and namespaces before the test 
executes.
+ */
+public class ClearUserNamespacesAndTablesRule extends ExternalResource {
 
 Review comment:
   nice.. for some reason, I thought we already had this somewhere, looked 
around but didn't find one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365478702
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/webapp/RegionReplicaInfo.java
 ##
 @@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.webapp;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStateStore;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * A POJO that consolidates the information about a single region replica 
that's stored in meta.
+ */
+@InterfaceAudience.Private
+public final class RegionReplicaInfo {
+  private final byte[] row;
+  private final RegionInfo regionInfo;
+  private final byte[] regionName;
+  private final byte[] startKey;
+  private final byte[] endKey;
+  private final Integer replicaId;
+  private final RegionState.State regionState;
+  private final ServerName serverName;
+
+  private RegionReplicaInfo(final Result result, final HRegionLocation 
location) {
+final Optional maybeResult = Optional.ofNullable(result);
+final Optional maybeLocation = 
Optional.ofNullable(location);
+final Optional maybeRegionInfo = 
maybeLocation.map(HRegionLocation::getRegion);
+
+this.row = maybeResult.map(Result::getRow).orElse(null);
 
 Review comment:
   nit: the accessors methods could work directly on regioninfo object ? (less 
code).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] bharathv commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

bharathv commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365477170
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/webapp/MetaBrowser.java
 ##
 @@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.webapp;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+import javax.servlet.http.HttpServletRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.apache.hadoop.hbase.client.AsyncTable;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.ScanResultConsumer;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.PrefixFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import 
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import 
org.apache.hbase.thirdparty.io.netty.handler.codec.http.QueryStringEncoder;
+
+/**
+ * A support class for the "Meta Entries" section in
+ * {@code resources/hbase-webapps/master/table.jsp}.
+ */
+@InterfaceAudience.Private
+public class MetaBrowser implements Iterable {
+  private static final Logger logger = 
LoggerFactory.getLogger(MetaBrowser.class);
+
+  public static final String NAME_PARAM = "name";
+  public static final String SCAN_LIMIT_PARAM = "scan_limit";
+  public static final String SCAN_REGION_STATE_PARAM = "scan_region_state";
+  public static final String SCAN_START_PARAM = "scan_start";
+  public static final String SCAN_TABLE_PARAM = "scan_table";
+
+  public static final int SCAN_LIMIT_DEFAULT = 10;
+  public static final int SCAN_LIMIT_MAX = 10_000;
+
+  private final AsyncConnection connection;
+  private final HttpServletRequest request;
+  private final ExecutorService pool;
+  private final List errorMessages;
+  private final String name;
+  private final Integer scanLimit;
+  private final RegionState.State scanRegionState;
+  private final byte[] scanStart;
+  private final TableName scanTable;
+
+  public MetaBrowser(final AsyncConnection connection, final 
HttpServletRequest request) {
+this.connection = connection;
+this.request = request;
+this.pool = buildThreadPool();
+this.errorMessages = new LinkedList<>();
+this.name = resolveName(request);
+this.scanLimit = resolveScanLimit(request);
+this.scanRegionState = resolveScanRegionState(request);
+this.scanStart = resolveScanStart(request);
+this.scanTable = resolveScanTable(request);
+  }
+
+  public List getErrorMessages() {
+return errorMessages;
+  }
+
+  public String getName() {
+return name;
+  }
+
+  public Integer getScanLimit() {
+return scanLimit;
+  }
+
+  public byte[] getScanStart() {
+return scanStart;
+  }
+
+  public RegionState.State getScanRegionState() {
+return scanRegionState;
+  }
+
+  public TableName getScanTable() {
+return scanTable;
+  }
+
+  @Override
+  public Iterator iterator() {
+return limitIterator();
+  }
+
+  public LimitIterator limitIterator() {
+logger.debug("initiating meta scan, {}", this);
+
+final AsyncTable asyncTable =
+  

[GitHub] [hbase] joshelser commented on issue #1019: HBASE-23679 Use new FileSystem objects during bulk loads

[GitBox]

joshelser commented on issue #1019: HBASE-23679 Use new FileSystem objects 
during bulk loads
URL: https://github.com/apache/hbase/pull/1019#issuecomment-573258755
 
 
   ```
   ctr-e141-1563959304486-133915-01-04: 2020-01-11 00:15:00,797 WARN  
[RpcServer.default.FPBQ.Fifo.handler=99,queue=9,port=16020] fs.FileSystem: 
Caching new filesystem: -279427062
   ctr-e141-1563959304486-133915-01-04: java.lang.Exception
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3365)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:761)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5958)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:264)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:233)
   ctr-e141-1563959304486-133915-01-04: at 
java.security.AccessController.doPrivileged(Native Method)
   ctr-e141-1563959304486-133915-01-04: at 
javax.security.auth.Subject.doAs(Subject.java:360)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:233)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2338)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42004)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
   ctr-e141-1563959304486-133915-01-04: at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
   ```
   
   Looks like this isn't quite sufficient. Another leak (albeit, much slower) 
coming here. Need to do more to push down that DFS instance we made and use 
that until we move the files into the final location.
   
   Added some debug to FileSystem.java to see the above. Testing is just done 
via IntegrationTestBulkLoad with high number of loops but small chain length.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013288#comment-17013288
 ] 

Hudson commented on HBASE-23601:


Results for branch branch-2
[build #2412 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> OutputSink.WriterThread exception gets stuck and repeated indefinietly
> --
>
> Key: HBASE-23601
> URL: https://issues.apache.org/jira/browse/HBASE-23601
> Project: HBase
>  Issue Type: Bug
>  Components: read replicas
>Affects Versions: 2.2.2
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.9, 2.2.4
>
>
> When a WriterThread runs into an exception (ie: NotServingRegionException), 
> the exception is stored in the controller. It is never removed and can not be 
> overwritten either.
>  
> {code:java}
> public void run()  {
>   try {
> doRun();
>   } catch (Throwable t) {
> LOG.error("Exiting thread", t);
> controller.writerThreadError(t);
>   }
> }{code}
> Thanks to this every time PipelineController.checkForErrors() is called the 
> same old exception is rethrown.
>  
> For example in RegionReplicaReplicationEndpoint.replicate there is a while 
> loop that does the actual replicating. Every time it loops, it calls 
> checkForErrors(), catches the rethrown exception, logs it but does nothing 
> about it. This results in ~2GB log files in ~5min in my experience.
>  
> My proposal would be to clean up the stored exception when it reaches 
> RegionReplicaReplicationEndpoint.replicate and make sure we restart the 
> WriterThread that died throwing it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23668) Master log start filling with "Flush journal status" messages

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013289#comment-17013289
 ] 

Hudson commented on HBASE-23668:


Results for branch branch-2
[build #2412 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Master log start filling with "Flush journal status" messages
> -
>
> Key: HBASE-23668
> URL: https://issues.apache.org/jira/browse/HBASE-23668
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2, RegionProcedureStore
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Takes a while to get into this condition. Not each to tell how because all 
> logs have rolled off and I only have logs filled w/ below:
> {code}
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226854 at 1578553261723
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226855 at 1578553261723
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226856 at 1578553261723
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226857 at 1578553261723
> {code}
> ... I added the printing of flushresult... i.e. cannot flush because store is 
> empty.
> Digging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23664) Upgrade JUnit to 4.13

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013291#comment-17013291
 ] 

Hudson commented on HBASE-23664:


Results for branch branch-2
[build #2412 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Upgrade JUnit to 4.13
> -
>
> Key: HBASE-23664
> URL: https://issues.apache.org/jira/browse/HBASE-23664
> Project: HBase
>  Issue Type: Task
>  Components: integration tests, test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> New JUnit released a week ago. Let's give it a spin.
> https://github.com/junit-team/junit4/blob/master/doc/ReleaseNotes4.13.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013290#comment-17013290
 ] 

Hudson commented on HBASE-23659:


Results for branch branch-2
[build #2412 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2412//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> BaseLoadBalancer#wouldLowerAvailability should consider region replicas
> ---
>
> Key: HBASE-23659
> URL: https://issues.apache.org/jira/browse/HBASE-23659
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> Found this issue when try to fix the flaky unit test  TestRegionReplicaSplit. 
> It may fail as 
> java.lang.AssertionError: Splitted regions should not be assigned to same 
> region server.
> See 
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/].
>  
> Now  wouldLowerAvailability method only consider the primary regions. The 
> replica region can't assign to same server with primary region. But can be 
> assigned to same server with other replica regions. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] ndimiduk commented on a change in pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

ndimiduk commented on a change in pull request #1020: HBASE-23653 Expose 
content of meta table in web ui
URL: https://github.com/apache/hbase/pull/1020#discussion_r365479666
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/webapp/MetaBrowser.java
 ##
 @@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.webapp;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+import javax.servlet.http.HttpServletRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncConnection;
+import org.apache.hadoop.hbase.client.AsyncTable;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.ScanResultConsumer;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.PrefixFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import 
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import 
org.apache.hbase.thirdparty.io.netty.handler.codec.http.QueryStringEncoder;
+
+/**
+ * A support class for the "Meta Entries" section in
+ * {@code resources/hbase-webapps/master/table.jsp}.
+ */
+@InterfaceAudience.Private
+public class MetaBrowser implements Iterable {
+  private static final Logger logger = 
LoggerFactory.getLogger(MetaBrowser.class);
+
+  public static final String NAME_PARAM = "name";
+  public static final String SCAN_LIMIT_PARAM = "scan_limit";
+  public static final String SCAN_REGION_STATE_PARAM = "scan_region_state";
+  public static final String SCAN_START_PARAM = "scan_start";
+  public static final String SCAN_TABLE_PARAM = "scan_table";
+
+  public static final int SCAN_LIMIT_DEFAULT = 10;
+  public static final int SCAN_LIMIT_MAX = 10_000;
+
+  private final AsyncConnection connection;
+  private final HttpServletRequest request;
+  private final ExecutorService pool;
+  private final List errorMessages;
+  private final String name;
+  private final Integer scanLimit;
+  private final RegionState.State scanRegionState;
+  private final byte[] scanStart;
+  private final TableName scanTable;
+
+  public MetaBrowser(final AsyncConnection connection, final 
HttpServletRequest request) {
+this.connection = connection;
+this.request = request;
+this.pool = buildThreadPool();
+this.errorMessages = new LinkedList<>();
+this.name = resolveName(request);
+this.scanLimit = resolveScanLimit(request);
+this.scanRegionState = resolveScanRegionState(request);
+this.scanStart = resolveScanStart(request);
+this.scanTable = resolveScanTable(request);
+  }
+
+  public List getErrorMessages() {
+return errorMessages;
+  }
+
+  public String getName() {
+return name;
+  }
+
+  public Integer getScanLimit() {
+return scanLimit;
+  }
+
+  public byte[] getScanStart() {
+return scanStart;
+  }
+
+  public RegionState.State getScanRegionState() {
+return scanRegionState;
+  }
+
+  public TableName getScanTable() {
+return scanTable;
+  }
+
+  @Override
+  public Iterator iterator() {
+return limitIterator();
+  }
+
+  public LimitIterator limitIterator() {
+logger.debug("initiating meta scan, {}", this);
+
+final AsyncTable asyncTable =
+  

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013284#comment-17013284
 ] 

Duo Zhang commented on HBASE-23326:
---

Proc wal has its own TTL setting, which is much longer than normal WAL files.

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23652) Move the unsupported procedure type check before migrating to RegionProcedureStore

[Michael Stack (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013282#comment-17013282
 ] 

Michael Stack commented on HBASE-23652:
---

Bit more info... I see these logs... when the oldWALs cleaner runs:

2020-01-10 00:00:45,232 DEBUG 
org.apache.hadoop.hbase.master.cleaner.LogCleaner: Attempting to delete old WAL 
file: 
HdfsLocatedFileStatus{path=hdfs://nameservice1/hbase/genie/oldWALs/h5%2C16020%2C1578604828426.h5%2C16020%2C1578604828426.regiongroup-0.1578613886971;
 isDirectory=false; length=203930698; replication=3; blocksize=268435456; 
modification_time=1578614072377; access_time=1578613886978; owner=hbase; 
group=supergroup; permission=rw-r--r--; isSymlink=false; hasAcl=false; 
isEncrypted=false; isErasureCoded=false}

 but they never mention 'masterproc' files.

> Move the unsupported procedure type check before migrating to 
> RegionProcedureStore
> --
>
> Key: HBASE-23652
> URL: https://issues.apache.org/jira/browse/HBASE-23652
> Project: HBase
>  Issue Type: Task
>  Components: master
>Affects Versions: 2.2.2
>Reporter: Nick Dimiduk
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 3.0.0, 2.3.0
>
>
> Per 
> [discussion|https://lists.apache.org/thread.html/c5f960e6cf3c89ffbf8f8302afb121462a08fd9a89d384c6795a7b81%40%3Cdev.hbase.apache.org%3E]
>  on the list, let's figure out how to make the upgrade from a v2.1 procedure 
> store less error-prone. Could be a simple as documenting runbook steps to 
> execute during the rolling upgrade, but it would be nice if the software 
> could roll over the data versions gracefully.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23680) RegionProcedureStore missing cleaning of hfile archive

[Michael Stack (Jira)]

Michael Stack created HBASE-23680:
-

 Summary: RegionProcedureStore missing cleaning of hfile archive
 Key: HBASE-23680
 URL: https://issues.apache.org/jira/browse/HBASE-23680
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Michael Stack
 Fix For: 2.3.0


See tail of parent issue. The new RegionProcedureStore accumulates deleted 
hfiles in its local archive dir. Needs a cleaner like the one that watches over 
/hbase/archive.

Is there a problem clearning the new $masterproc$ files from the oldWALs too? 
These seem to stick around also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Michael Stack (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013280#comment-17013280
 ] 

Michael Stack commented on HBASE-23326:
---

[~zhangduo] Made a sub-issue of this. Thanks.

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23644) Thrift2 DemoClient.py

[Madhukumar (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013277#comment-17013277
 ] 

Madhukumar commented on HBASE-23644:


See [https://github.com/apache/hbase/pull/988].What needs to be done to get the 
file DemoClient.py committed in github? It is updated to work with python 3.X. 

> Thrift2 DemoClient.py
> -
>
> Key: HBASE-23644
> URL: https://issues.apache.org/jira/browse/HBASE-23644
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.2.2
>Reporter: Madhukumar
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: DemoClient.py
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> *hbase-examples/py*
> I tried to connect to hbase thrift2 server with python with given demo file 
> DemoClient.py.  Issue is the DemoClient.py is not updated for python 3.X.  
> Python 2.X support has ended. Even the github vesrion is not updated, happy 
> to update it. 
> I'm using python 3.8. {color:#ff}_Attached file for DemoClient.py for 
> python 3.X._ I'm new to Jira and hbase. {color} __ 
>  
>  hbase.thrift file that was used to generate gen-py in 2.2.22 release 
> examples does seem like this file 
> hbase-src-2.2.2/hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift.
>  _I'm looking for hbase.thrift that was used to get gen-py in py examples. If 
> you can point me to that, that will be great._
> {color:#172b4d}If you generate gen-py using thrift compiler 0.13, this is 
> error that I'm getting when running DemoClient.py, {color}
> {color:#172b4d}Traceback (most recent call last):
>  File "DemoClient.py", line 68, in 
>  client.put(table, put)
>  File 
> "/Users/madhuseshadri/Projects/hbase-pyclient-thrift2/gen-py/hbase/THBaseService.py",
>  line 813, in put
>  self.send_put(table, tput)
>  File 
> "/Users/madhuseshadri/Projects/hbase-pyclient-thrift2/gen-py/hbase/THBaseService.py",
>  line 821, in send_put
>  args.write(self._oprot)
>  File 
> "/Users/madhuseshadri/Projects/hbase-pyclient-thrift2/gen-py/hbase/THBaseService.py",
>  line 4461, in write
>  oprot.writeBinary(self.table)
>  File 
> "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/thrift-0.13.0-py3.8-macosx-10.9-x86_64.egg/thrift/protocol/TBinaryProtocol.py",
>  line 131, in writeBinary
>  self.trans.write(str)
>  File 
> "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/thrift-0.13.0-py3.8-macosx-10.9-x86_64.egg/thrift/transport/TTransport.py",
>  line 174, in write
>  raise e
>  File 
> "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/thrift-0.13.0-py3.8-macosx-10.9-x86_64.egg/thrift/transport/TTransport.py",
>  line 170, in write
>  self.__wbuf.write(buf)
>  TypeError: a bytes-like object is required, not 'str'{color}
> {color:#172b4d}Please note that I'm using 0.13 py client package for thrift. 
> It does not work with thrift==0.9.3 as well.{color}
> Out of the box gen-py that came with 2.2.2 source examples has the following 
> issues working with thrift 0.13 py packages and it is also not code generated 
> for python 3.0 but 2.0.
>  * ttypes.py > self.__dict.__iteritems() does not work in python 3.0, so it 
> need to be items 
>  * xrange does not work in python 3.0 so all that need to be range 
>  
> thrift compiler 0.13 was available at this point from apache/thrift. It 
> generates the files for python 2.0 it seems because of above two issues 
> because hbase.thrift that is src (link given above), not sure what can be 
> done about this. 
> So, I have 0.13 python thrift package which works with python 3.0. 
> DemoClient.py updated to 3.0 but issue is gen-py generation. I don't want to 
> edit these generated py files, so if right hbase.thrift file that works with 
> thrift 0.13.0 compiler, that will be great. Thanks.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Duo Zhang (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013276#comment-17013276
 ] 

Duo Zhang commented on HBASE-23326:
---

This is a missing part. Let’s open a follow on issue. I think we can just 
follow what we have done for archiving WAL, that move the archived hfiles to 
the global archive directory after compaction.

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] pytalkweb commented on issue #988: Made DemoClient python 3.0 compatible ..

[GitBox]

pytalkweb commented on issue #988: Made DemoClient python 3.0 compatible ..
URL: https://github.com/apache/hbase/pull/988#issuecomment-573251595
 
 
   The file DemoClient.py that I made a new pull request in
   https://github.com/apache/hbase works with python 3.X. I'm not sure what
   the issue with github is, it has broken heart with -1 on top with some
   error from pylint. I'm not sure what I need to do to get it fixed. If you
   can let me know, I can get it committed.
   
   On Mon, Jan 6, 2020 at 9:34 AM Sean Busbey  wrote:
   
   > Does this still work with python 2? Or is it python 3 only?
   >
   > I'd be fine with the master branch being python 3 only if we can make
   > branch-2 python 2 and python 3. This could be done with mutliple examples
   > if needed.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] ndimiduk opened a new pull request #1021: Backport HBASE-23653 Expose content of meta table in web ui

[GitBox]

ndimiduk opened a new pull request #1021: Backport HBASE-23653 Expose content 
of meta table in web ui
URL: https://github.com/apache/hbase/pull/1021
 
 
   Adds a display of the content of 'hbase:meta' to the Master's
   table.jsp, when that table is selected. Supports basic pagination,
   filtering, 
   
   Backport of #1020


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Michael Stack (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013270#comment-17013270
 ] 

Michael Stack commented on HBASE-23326:
---

And this seem to be hanging around... 
%2C16000%2C1578524235076.1578524242389$masterproc$ though straight WALs are 
cleared.

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing

[GitBox]

ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] 
`fixHoles` should queue assignment procedures for any regions its fixing
URL: https://github.com/apache/hbase/pull/917#discussion_r365467011
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
 ##
 @@ -86,74 +84,133 @@ void fix() throws IOException {
* If hole, it papers it over by adding a region in the filesystem and to 
hbase:meta.
* Does not assign.
*/
-  void fixHoles(CatalogJanitor.Report report) throws IOException {
-List> holes = report.getHoles();
+  void fixHoles(CatalogJanitor.Report report) {
+final List> holes = report.getHoles();
 if (holes.isEmpty()) {
-  LOG.debug("No holes.");
+  LOG.info("CatalogJanitor Report contains no holes to fix. Skipping.");
   return;
 }
-for (Pair p: holes) {
-  RegionInfo ri = getHoleCover(p);
-  if (ri == null) {
-continue;
-  }
-  Configuration configuration = this.masterServices.getConfiguration();
-  HRegion.createRegionDir(configuration, ri, 
FSUtils.getRootDir(configuration));
-  // If an error here, then we'll have a region in the filesystem but not
-  // in hbase:meta (if the below fails). Should be able to rerun the fix.
-  // Add to hbase:meta and then update in-memory state so it knows of new
-  // Region; addRegionToMeta adds region and adds a state column set to 
CLOSED.
-  MetaTableAccessor.addRegionToMeta(this.masterServices.getConnection(), 
ri);
-  this.masterServices.getAssignmentManager().getRegionStates().
-  updateRegionState(ri, RegionState.State.CLOSED);
-  LOG.info("Fixed hole by adding {} in CLOSED state; region NOT assigned 
(assign to ONLINE).",
-  ri);
-}
+
+LOG.info("Identified {} region holes to fix. Detailed fixup progress 
logged at DEBUG.",
+  holes.size());
+
+final List newRegionInfos = createRegionInfosForHoles(holes);
+final List newMetaEntries = createMetaEntries(masterServices, 
newRegionInfos);
+final TransitRegionStateProcedure[] assignProcedures = masterServices
+  .getAssignmentManager()
+  .createRoundRobinAssignProcedures(newMetaEntries);
+
+
masterServices.getMasterProcedureExecutor().submitProcedures(assignProcedures);
+LOG.info(
 
 Review comment:
   The procedure engine logs a ton, but how will the operator know which 
procIds are the the ones that've been requested by this fixup? I'd think it's 
good to keep this message, it's an effective summary of the work done.
   
   Make it DEBUG though?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing

[GitBox]

ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] 
`fixHoles` should queue assignment procedures for any regions its fixing
URL: https://github.com/apache/hbase/pull/917#discussion_r365466674
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
 ##
 @@ -86,74 +84,133 @@ void fix() throws IOException {
* If hole, it papers it over by adding a region in the filesystem and to 
hbase:meta.
* Does not assign.
*/
-  void fixHoles(CatalogJanitor.Report report) throws IOException {
-List> holes = report.getHoles();
+  void fixHoles(CatalogJanitor.Report report) {
+final List> holes = report.getHoles();
 if (holes.isEmpty()) {
-  LOG.debug("No holes.");
+  LOG.info("CatalogJanitor Report contains no holes to fix. Skipping.");
 
 Review comment:
   Good sit, this log line is emitted by `MetaFixer`, speaking of its input. 
It'll be run just when an operator has invoked hbck2.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing

[GitBox]

ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] 
`fixHoles` should queue assignment procedures for any regions its fixing
URL: https://github.com/apache/hbase/pull/917#discussion_r365466933
 
 

 ##
 File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
 ##
 @@ -86,74 +84,133 @@ void fix() throws IOException {
* If hole, it papers it over by adding a region in the filesystem and to 
hbase:meta.
* Does not assign.
*/
-  void fixHoles(CatalogJanitor.Report report) throws IOException {
-List> holes = report.getHoles();
+  void fixHoles(CatalogJanitor.Report report) {
+final List> holes = report.getHoles();
 if (holes.isEmpty()) {
-  LOG.debug("No holes.");
+  LOG.info("CatalogJanitor Report contains no holes to fix. Skipping.");
   return;
 }
-for (Pair p: holes) {
-  RegionInfo ri = getHoleCover(p);
-  if (ri == null) {
-continue;
-  }
-  Configuration configuration = this.masterServices.getConfiguration();
-  HRegion.createRegionDir(configuration, ri, 
FSUtils.getRootDir(configuration));
-  // If an error here, then we'll have a region in the filesystem but not
-  // in hbase:meta (if the below fails). Should be able to rerun the fix.
-  // Add to hbase:meta and then update in-memory state so it knows of new
-  // Region; addRegionToMeta adds region and adds a state column set to 
CLOSED.
-  MetaTableAccessor.addRegionToMeta(this.masterServices.getConnection(), 
ri);
-  this.masterServices.getAssignmentManager().getRegionStates().
-  updateRegionState(ri, RegionState.State.CLOSED);
-  LOG.info("Fixed hole by adding {} in CLOSED state; region NOT assigned 
(assign to ONLINE).",
-  ri);
-}
+
+LOG.info("Identified {} region holes to fix. Detailed fixup progress 
logged at DEBUG.",
 
 Review comment:
   I want the system to log at least acknowledgement that the operator 
requested a run. That's the above log line -- an empty CatalogJanitor Report 
means no work to do. This line is setting the stage for how much work it thinks 
needs done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-23653) Expose content of meta table in web ui

[Nick Dimiduk (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013262#comment-17013262
 ] 

Nick Dimiduk commented on HBASE-23653:
--

Here's a couple screen shots of the new meta browser.

> Expose content of meta table in web ui
> --
>
> Key: HBASE-23653
> URL: https://issues.apache.org/jira/browse/HBASE-23653
> Project: HBase
>  Issue Type: New Feature
>  Components: master, Operability, UI
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: 23653.0.png, 23653.1.png
>
>
> While testing improvements to hbck2, breaking and fixing a cluster's region 
> assignments, we've found that quite a few questions can only be answered by 
> dropping into a shell and scanning meta. This is in contrast to the nice web 
> view we have on procedures and their locks. It would be nice to have a 
> similar view into the content of meta. I've put together and extension to the 
> existing {{table.jsp}} that does this. It's had a couple rounds of operator 
> feedback. Will post up some screenshots along with the PR.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23679) FileSystem instance leaks due to bulk loads with Kerberos enabled

[Josh Elser (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013260#comment-17013260
 ] 

Josh Elser commented on HBASE-23679:


Dropping some more details in here, should anyone else ever find themselves 
back here.
 * Funny that this user's RegionServers never (rarely?) actually OOME'ed. They 
were left in this crap state because the JVM just kept hanging on. GC verbose 
logging showed us that heap was just sitting completely full. We just sat with 
30-60s GC pauses, getting next to nothing done.
 * A heapdump showed (at worst) 60k+ instances of DistributedFileSystem sitting 
on the heap, nearly all of them referenced by {{FileSystem$CACHE}}. It takes 
some digging to realize this.
 * Digging into each DFS -> DFSClient -> UGI -> Tokens, you can see the 
delegation tokens which let us confirm that Bulk Load was the culprit 
creating/leaking these.
 * For the user, it was an extremely slow growth. One RegionServer would get 
into this state every ~20 days across about that many RegionServers. So, in 
reality, we were probably looking at even longer than 20days for a single 
RegionServer to leak enough of these to cause issues.
 * UGI being a part of FileSystem caching is crap, and we should avoid it for 
anything other than "just the HBase user". It's extremely obtuse and error 
prone, likely to just give us more heartache (lots of issues around this in the 
past).

> FileSystem instance leaks due to bulk loads with Kerberos enabled
> -
>
> Key: HBASE-23679
> URL: https://issues.apache.org/jira/browse/HBASE-23679
> Project: HBase
>  Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
>
> Spent the better part of a week chasing an issue on HBase 2.x where the 
> number of DistributedFileSystem instances on the heap of a RegionServer would 
> grow unbounded. Looking at multiple heap-dumps, it was obvious to see that we 
> had an immense number of DFS instances cached (in FileSystem$Cache) for the 
> same user, with the unique number of Tokens contained in that DFS's UGI 
> member (one hbase delegation token, and two HDFS delegation tokens – we only 
> do this for bulk loads). For the user's clusters, they eventually experienced 
> 10x perf degradation as RegionServers spent all of their time in JVM GC (they 
> were unlucky to not have RegionServers crash outright, as this would've, 
> albeit temporarily, fixed the issue).
> The problem seems to be two-fold with changes by HBASE-15291 being largely 
> the cause. This issue tried to close FileSystem instances which were being 
> leaked – however, it did this by instrumenting the method 
> {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this 
> approach:
>  # It relies on clients to call this method (client's hanging up will leak 
> resources in RegionServers)
>  # This method is only called on the RegionServer hosting the first Region of 
> the table which was bulk-loaded into. For multiple RegionServers, they are 
> left to leak resources.
> HBASE-21342 later tried to fix an issue where FS objects were now being 
> closed prematurely via reference-counting (which appears to work fine), but 
> does not address the other two issues above. Point #2 makes debugging this 
> issue harder than normal because it doesn't manifest on a single node 
> instance :)
> Through all of this, I (re)learned the dirty history of UGI and how its 
> caching doesn't work so great HADOOP-6670. I see trying to continue to 
> leverage the FileSystem$CACHE as a potentially dangerous thing (we've been 
> back here multiple times already). My opinion at this point is that we should 
> cleanly create a new FileSystem instance during the call to 
> {{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally 
> block in that same method. This both simplifies the lifecycle of a FileSystem 
> instance in the bulk-load codepath but also helps us avoid future problems 
> with UGI and FS caching. The one downside is that we pay the penalty to 
> create a new FileSystem instance, but I'm of the opinion that we cross that 
> bridge when we get there.
> Thanks for [~jdcryans] and [~busbey] for their help along the way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-23653) Expose content of meta table in web ui

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-23653:
-
Attachment: 23653.1.png
23653.0.png

> Expose content of meta table in web ui
> --
>
> Key: HBASE-23653
> URL: https://issues.apache.org/jira/browse/HBASE-23653
> Project: HBase
>  Issue Type: New Feature
>  Components: master, Operability, UI
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: 23653.0.png, 23653.1.png
>
>
> While testing improvements to hbck2, breaking and fixing a cluster's region 
> assignments, we've found that quite a few questions can only be answered by 
> dropping into a shell and scanning meta. This is in contrast to the nice web 
> view we have on procedures and their locks. It would be nice to have a 
> similar view into the content of meta. I've put together and extension to the 
> existing {{table.jsp}} that does this. It's had a couple rounds of operator 
> feedback. Will post up some screenshots along with the PR.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] ndimiduk opened a new pull request #1020: HBASE-23653 Expose content of meta table in web ui

[GitBox]

ndimiduk opened a new pull request #1020: HBASE-23653 Expose content of meta 
table in web ui
URL: https://github.com/apache/hbase/pull/1020
 
 
   Adds a display of the content of 'hbase:meta' to the Master's
   table.jsp, when that table is selected. Supports basic pagination,
   filtering, 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-23349) Low refCount preventing archival of compacted away files

[Andrew Kyle Purtell (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013255#comment-17013255
 ] 

Andrew Kyle Purtell commented on HBASE-23349:
-

{quote}
I think we should step back and remember why we have the ref counting in the 
first place. This came from a discussion started in HBASE-13082 and 
HBASE-10060, namely too much synchronization.

If any changes we make now needs new synchronization in the scanner.next(...) 
path we're back to where we started and in that case we should remove the ref 
counting and bring back the old notification and scanner switching we had 
before.
{quote}

I made a similar comment on an internal discussion yesterday. If we have to 
walk back the StoreScanner "lock coarsening" work, then let's not be afraid to 
do it. There is a nuanced decision we would have to make, but let's not be 
concerned about sunk costs. 

> Low refCount preventing archival of compacted away files
> 
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> We have observed that refCount on compacted away store files as low as 1 is 
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file 
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
>  because of either isCompactedAway=true or file has reference, 
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread) 
> gracefully resolve reader lock issue by resetting ongoing scanners to start 
> pointing to new store files instead of compacted away store files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013250#comment-17013250
 ] 

Hudson commented on HBASE-18095:


Results for branch HBASE-18095/client-locate-meta-no-zookeeper
[build #35 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35/]:
 (x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Provide an option for clients to find the server hosting META that does not 
> involve the ZooKeeper client
> 
>
> Key: HBASE-18095
> URL: https://issues.apache.org/jira/browse/HBASE-18095
> Project: HBase
>  Issue Type: New Feature
>  Components: Client
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
> Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch
>
>
> Clients are required to connect to ZooKeeper to find the location of the 
> regionserver hosting the meta table region. Site configuration provides the 
> client a list of ZK quorum peers and the client uses an embedded ZK client to 
> query meta location. Timeouts and retry behavior of this embedded ZK client 
> are managed orthogonally to HBase layer settings and in some cases the ZK 
> cannot manage what in theory the HBase client can, i.e. fail fast upon outage 
> or network partition.
> We should consider new configuration settings that provide a list of 
> well-known master and backup master locations, and with this information the 
> client can contact any of the master processes directly. Any master in either 
> active or passive state will track meta location and respond to requests for 
> it with its cached last known location. If this location is stale, the client 
> can ask again with a flag set that requests the master refresh its location 
> cache and return the up-to-date location. Every client interaction with the 
> cluster thus uses only HBase RPC as transport, with appropriate settings 
> applied to the connection. The configuration toggle that enables this 
> alternative meta location lookup should be false by default.
> This removes the requirement that HBase clients embed the ZK client and 
> contact the ZK service directly at the beginning of the connection lifecycle. 
> This has several benefits. ZK service need not be exposed to clients, and 
> their potential abuse, yet no benefit ZK provides the HBase server cluster is 
> compromised. Normalizing HBase client and ZK client timeout settings and 
> retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer 
> necessary. 
> And, from [~ghelmling]: There is an additional complication here for 
> token-based authentication. When a delegation token is used for SASL 
> authentication, the client uses the cluster ID obtained from Zookeeper to 
> select the token identifier to use. So there would also need to be some 
> Zookeeper-less, unauthenticated way to obtain the cluster ID as well. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] saintstack commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters…

[GitBox]

saintstack commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on 
meta table (fix bloomfilters…
URL: https://github.com/apache/hbase/pull/1012#issuecomment-573230752
 
 
   Ok. So changing blooms seems fine. Let me retry with the encoder.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion

[Michael Stack (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013235#comment-17013235
 ] 

Michael Stack commented on HBASE-23326:
---

[~zhangduo] I see that the regionprocedurestore WAL files are being moved to 
the /hbase/archive but I also see build up at /hbase/MasterProcs/archive of old 
procedure hfiles. I have 1500 files in it currently (running about 24 hours 
under load). Poking in PR I don't see obvious handling of this Region archive 
dir. What am I missing? Thanks.

> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also 
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is 
> an anti-pattern as we make the hbase-server module more overloaded. But I 
> think later we can first try to move the WAL stuff out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters…

[GitBox]

Apache-HBase commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on 
meta table (fix bloomfilters…
URL: https://github.com/apache/hbase/pull/1012#issuecomment-573226238
 
 
   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 37s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   6m 31s |  master passed  |
   | +1 :green_heart: |  compile  |   1m  0s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 44s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m 25s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 40s |  master passed  |
   | +0 :ok: |  spotbugs  |   5m  4s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   5m  0s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m 52s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  5s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m  5s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m 39s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  shadedjars  |   5m 38s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  18m 30s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 or 3.1.2.  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |   4m 52s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 156m  1s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  asflicense  |   0m 38s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 223m 37s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1012 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile |
   | uname | Linux 888eea02fdcf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1012/out/precommit/personality/provided.sh
 |
   | git revision | master / 8ca614857d |
   | Default Java | 1.8.0_181 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/2/testReport/
 |
   | Max. process+thread count | 5889 (vs. ulimit of 1) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/2/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (HBASE-23628) Replace Apache Commons Digest Base64 with JDK8 Base64

[Sean Busbey (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-23628:

Summary: Replace Apache Commons Digest Base64 with JDK8 Base64  (was: 
Remove Apache Commons Digest Base64)

> Replace Apache Commons Digest Base64 with JDK8 Base64
> -
>
> Key: HBASE-23628
> URL: https://issues.apache.org/jira/browse/HBASE-23628
> Project: HBase
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
>
> Use the native JDK Base64 implementation instead.  Most places are using the 
> JDK version, but a couple of spots were missed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-23628) Remove Apache Commons Digest Base64

[Sean Busbey (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-23628.
-
Fix Version/s: 2.3.0
   Resolution: Fixed

> Remove Apache Commons Digest Base64
> ---
>
> Key: HBASE-23628
> URL: https://issues.apache.org/jira/browse/HBASE-23628
> Project: HBase
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
>
> Use the native JDK Base64 implementation instead.  Most places are using the 
> JDK version, but a couple of spots were missed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-23628) Remove Apache Commons Digest Base64

[Sean Busbey (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reopened HBASE-23628:
-

reopening for backport(s)

> Remove Apache Commons Digest Base64
> ---
>
> Key: HBASE-23628
> URL: https://issues.apache.org/jira/browse/HBASE-23628
> Project: HBase
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.0.0
>
>
> Use the native JDK Base64 implementation instead.  Most places are using the 
> JDK version, but a couple of spots were missed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23676) Address feedback on HBASE-23055 Alter hbase:meta.

[Michael Stack (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013220#comment-17013220
 ] 

Michael Stack commented on HBASE-23676:
---

{code}
Duo Zhang 

5:33 AM (8 hours ago)

to State, apache/hbase, Michael

@Apache9 commented on this pull request.

In 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java:

> @@ -666,42 +667,55 @@ public void run(PRESP resp) {
   new DisableTableProcedureBiConsumer(tableName));
   }
 
+  /**
+   * Utility for completing passed TableState {@link CompletableFuture} 
future
+   * using passed parameters.
+   */
+  private static CompletableFuture completeCheckTableState(

What is the return value used for?

In 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java:

> @@ -58,8 +62,19 @@
 
   private final ZNodePaths znodePaths;
 
+  /**
+   * A znode maintained by MirroringTableStateManager.
+   * MirroringTableStateManager is deprecated to be removed in hbase3. It can 
also be disabled.
+   * Make sure it is enabled if you want to alter hbase:meta table in hbase2. 
In hbase3,
+   * TBD how metatable state will be hosted; likely on active hbase master.

I think the state will either be on zk or on hdfs? HMaster itself is state 
less...

In general, in the current architecture, I think it the state should be placed 
on zk, of course you could cache it in master and let client go to master to 
ask for the state.

In 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java:

> @@ -58,8 +62,19 @@
 
   private final ZNodePaths znodePaths;
 
+  /**
+   * A znode maintained by MirroringTableStateManager.
+   * MirroringTableStateManager is deprecated to be removed in hbase3. It can 
also be disabled.
+   * Make sure it is enabled if you want to alter hbase:meta table in hbase2. 
In hbase3,
+   * TBD how metatable state will be hosted; likely on active hbase master.

So I do not think we should name this as mirrored, it is the original data. The 
state in master's memory, is a cache, actually.

In 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java:

> @@ -229,6 +244,43 @@ private void 
> getMetaRegionLocation(CompletableFuture future,
 });
   }
 
+  @Override
+  public CompletableFuture getMetaTableState() {
+return getAndConvert(this.znodeMirroredMetaTableState, 
ZKAsyncRegistry::getTableState).
+  thenApply(state -> {
+return state == null || 
state.equals(ENABLED_META_TABLE_STATE.getState())?
+  ENABLED_META_TABLE_STATE: new TableState(TableName.META_TABLE_NAME, 
state);
+  }).exceptionally(e -> {

Currently in HBase, usually we will create a new CompletableFuture and use 
FutureUtils.addListener to complete it. The code in exceptionally are a bit 
tricky, where we throw a CompletionException...

In 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java:

>private static final Logger LOG = 
> LoggerFactory.getLogger(TableStateManager.class);
+
+  /**
+   * All table state is kept in hbase:meta except that of hbase:meta itself.

In general I do not think this is a good design. We should persist it 
somewhere, and just cache it in master's memory. And we do not need to disable 
a table when altering any more? And the solution here just assume that we only 
have one meta region? So the altering operation can be atomic? What if we have 
multiple meta regions and we crash in the middle? I think this patch also aims 
to implement splittable meta in the future? No?

In 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RejectReplicationRequestStateChecker.java:

> @@ -1,4 +1,6 @@
-/**
+/*
+ * Copyright The Apache Software Foundation

What's this?

In hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseMetaEdit.java:

> +Admin admin = UTIL.getAdmin();
+admin.tableExists(TableName.META_TABLE_NAME);
+admin.disableTable(TableName.META_TABLE_NAME);
+assertTrue(admin.isTableDisabled(TableName.META_TABLE_NAME));
+TableDescriptor descriptor = 
admin.getDescriptor(TableName.META_TABLE_NAME);
+ColumnFamilyDescriptor cfd = 
descriptor.getColumnFamily(HConstants.CATALOG_FAMILY);
+byte [] extraColumnFamilyName = Bytes.toBytes("xtra");
+ColumnFamilyDescriptor newCfd =
+
ColumnFamilyDescriptorBuilder.newBuilder(extraColumnFamilyName).build();
+int oldVersions = cfd.getMaxVersions();
+// Add '1' to current versions count.
+cfd = 
ColumnFamilyDescriptorBuilder.newBuilder(cfd).setMaxVersions(oldVersions + 1).
+setConfiguration(ColumnFamilyDescriptorBuilder.DATA_BLOCK_ENCODING,
+DataBlockEncoding.ROW_INDEX_V1.toString()).build();
+admin.modifyColumnFamily(TableName.META_TABLE_NAME, cfd);
+admin.addColumnFamily(TableName.META_TABLE_NAME, newCfd);

Now we allow users to add/remove families of meta 

[jira] [Updated] (HBASE-23628) Remove Apache Commons Digest Base64

[Sean Busbey (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-23628:

Component/s: dependencies

> Remove Apache Commons Digest Base64
> ---
>
> Key: HBASE-23628
> URL: https://issues.apache.org/jira/browse/HBASE-23628
> Project: HBase
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.0.0
>
>
> Use the native JDK Base64 implementation instead.  Most places are using the 
> JDK version, but a couple of spots were missed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] joshelser opened a new pull request #1019: HBASE-23679 Use new FileSystem objects during bulk loads

[GitBox]

joshelser opened a new pull request #1019: HBASE-23679 Use new FileSystem 
objects during bulk loads
URL: https://github.com/apache/hbase/pull/1019
 
 
   Undoes lots of fanciness about FileSystem caching because of
   an explicit bug that was present (FileSystem only closed on one
   out of N regionservers, and not every time), but also because
   the FS caching itself is dodgy and prone to error.
   
   Each BulkLoad request will now get its own FileSystem instance
   that it is responsible for closing.
   
   The change to HRegion is because ${hbase.rootdir}/data is `chmod 700` which 
means that a normal user cannot get the size of those files (you'll see lots of 
AccessDenied exceptions in the RS log). Using HRegionFilesystem instead keeps 
this from being an issue (reading the filesize as HBase instead of the user).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (HBASE-23664) Upgrade JUnit to 4.13

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-23664.
--
Resolution: Fixed

> Upgrade JUnit to 4.13
> -
>
> Key: HBASE-23664
> URL: https://issues.apache.org/jira/browse/HBASE-23664
> Project: HBase
>  Issue Type: Task
>  Components: integration tests, test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> New JUnit released a week ago. Let's give it a spin.
> https://github.com/junit-team/junit4/blob/master/doc/ReleaseNotes4.13.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] ndimiduk merged pull request #1010: HBASE-23664 Upgrade JUnit to 4.13

[GitBox]

ndimiduk merged pull request #1010: HBASE-23664 Upgrade JUnit to 4.13
URL: https://github.com/apache/hbase/pull/1010
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (HBASE-23679) FileSystem instance leaks due to bulk loads with Kerberos enabled

[Josh Elser (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-23679:
---
Description: 
Spent the better part of a week chasing an issue on HBase 2.x where the number 
of DistributedFileSystem instances on the heap of a RegionServer would grow 
unbounded. Looking at multiple heap-dumps, it was obvious to see that we had an 
immense number of DFS instances cached (in FileSystem$Cache) for the same user, 
with the unique number of Tokens contained in that DFS's UGI member (one hbase 
delegation token, and two HDFS delegation tokens – we only do this for bulk 
loads). For the user's clusters, they eventually experienced 10x perf 
degradation as RegionServers spent all of their time in JVM GC (they were 
unlucky to not have RegionServers crash outright, as this would've, albeit 
temporarily, fixed the issue).

The problem seems to be two-fold with changes by HBASE-15291 being largely the 
cause. This issue tried to close FileSystem instances which were being leaked – 
however, it did this by instrumenting the method 
{{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this 
approach:
 # It relies on clients to call this method (client's hanging up will leak 
resources in RegionServers)
 # This method is only called on the RegionServer hosting the first Region of 
the table which was bulk-loaded into. For multiple RegionServers, they are left 
to leak resources.

HBASE-21342 later tried to fix an issue where FS objects were now being closed 
prematurely via reference-counting (which appears to work fine), but does not 
address the other two issues above. Point #2 makes debugging this issue harder 
than normal because it doesn't manifest on a single node instance :)

Through all of this, I (re)learned the dirty history of UGI and how its caching 
doesn't work so great HADOOP-6670. I see trying to continue to leverage the 
FileSystem$CACHE as a potentially dangerous thing (we've been back here 
multiple times already). My opinion at this point is that we should cleanly 
create a new FileSystem instance during the call to 
{{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally 
block in that same method. This both simplifies the lifecycle of a FileSystem 
instance in the bulk-load codepath but also helps us avoid future problems with 
UGI and FS caching. The one downside is that we pay the penalty to create a new 
FileSystem instance, but I'm of the opinion that we cross that bridge when we 
get there.

Thanks for [~jdcryans] and [~busbey] for their help along the way.

  was:
Spent the better part of a week chasing an issue on HBase 2.x where the number 
of DistributedFileSystem instances on the heap of a RegionServer would grow 
unbounded. Looking at multiple heap-dumps, it was obvious to see that we had an 
immense number of DFS instances cached (in FileSystem$Cache) for the same user, 
with the unique number of Tokens contained in that DFS's UGI member (one hbase 
delegation token, and two HDFS delegation tokens – we only do this for bulk 
loads). For the user's clusters, they eventually experienced 10x perf 
degradation as RegionServers spent all of their time in JVM GC (they were 
unlucky to not have RegionServers crash outright, as this would've, albeit 
temporarily, fixed the issue).

The problem seems to be two-fold with changes by HBASE-15291 being largely the 
cause. This issue tried to close
 FileSystem instances which were being leaked – however, it did this by 
instrumenting the method
 {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this 
approach:
 1. It relies on clients to call this method (client's hanging up will leak 
resources in RegionServers)
 2. This method is only called on the RegionServer hosting the first Region of 
the table which was bulk-loaded into. For
 multiple RegionServers, they are left to leak resources.

HBASE-21342 later tried to fix an issue where FS objects were now being closed 
prematurely via reference-counting (which appears to work fine), but does not 
address the other two issues above.

Through all of this, I (re)learned the dirty history of UGI and how its caching 
doesn't work so great HADOOP-6670. I see trying to continue to leverage the 
FileSystem$CACHE as a potentially dangerous thing (we've been back here 
multiple times already). My opinion at this point is that we should cleanly 
create a new FileSystem instance during the call to 
{{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally 
block in that same method. This both simplifies the lifecycle of a FileSystem 
instance in the bulk-load codepath but also helps us avoid future problems with 
UGI and FS caching. The one downside is that we pay the penalty to create a new 
FileSystem instance, but I'm of the opinion that we cross that bridge when we 
get there.

Thanks for [~jdcryans] and [~busbey] for 

[jira] [Created] (HBASE-23679) FileSystem instance leaks due to bulk loads with Kerberos enabled

[Josh Elser (Jira)]

Josh Elser created HBASE-23679:
--

 Summary: FileSystem instance leaks due to bulk loads with Kerberos 
enabled
 Key: HBASE-23679
 URL: https://issues.apache.org/jira/browse/HBASE-23679
 Project: HBase
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser


Spent the better part of a week chasing an issue on HBase 2.x where the number 
of DistributedFileSystem instances on the heap of a RegionServer would grow 
unbounded. Looking at multiple heap-dumps, it was obvious to see that we had an 
immense number of DFS instances cached (in FileSystem$Cache) for the same user, 
with the unique number of Tokens contained in that DFS's UGI member (one hbase 
delegation token, and two HDFS delegation tokens – we only do this for bulk 
loads). For the user's clusters, they eventually experienced 10x perf 
degradation as RegionServers spent all of their time in JVM GC (they were 
unlucky to not have RegionServers crash outright, as this would've, albeit 
temporarily, fixed the issue).

The problem seems to be two-fold with changes by HBASE-15291 being largely the 
cause. This issue tried to close
 FileSystem instances which were being leaked – however, it did this by 
instrumenting the method
 {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this 
approach:
 1. It relies on clients to call this method (client's hanging up will leak 
resources in RegionServers)
 2. This method is only called on the RegionServer hosting the first Region of 
the table which was bulk-loaded into. For
 multiple RegionServers, they are left to leak resources.

HBASE-21342 later tried to fix an issue where FS objects were now being closed 
prematurely via reference-counting (which appears to work fine), but does not 
address the other two issues above.

Through all of this, I (re)learned the dirty history of UGI and how its caching 
doesn't work so great HADOOP-6670. I see trying to continue to leverage the 
FileSystem$CACHE as a potentially dangerous thing (we've been back here 
multiple times already). My opinion at this point is that we should cleanly 
create a new FileSystem instance during the call to 
{{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally 
block in that same method. This both simplifies the lifecycle of a FileSystem 
instance in the bulk-load codepath but also helps us avoid future problems with 
UGI and FS caching. The one downside is that we pay the penalty to create a new 
FileSystem instance, but I'm of the opinion that we cross that bridge when we 
get there.

Thanks for [~jdcryans] and [~busbey] for their help along the way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema

[Andrew Kyle Purtell (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-23678:

Description: 
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions after that interval 
elapses. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, 
KEEP_DELETED_CELLS=TTL, MIN_VERSIONS=_num_versions_ . This is not intuitive 
though because VERSIONS has been used to specify the number of versions to 
retain (_num_versions_ in this example) since HBase version 0.1, so this is 
going to be a source of confusion - I've seen it in practice. 

A literate builder API, by way of the way we design its method names, could let 
a user describe more or less in speaking language how they want version 
retention to work, and internally the builder API could set the low level 
schema attributes. 

  was:
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions after that interval 
elapses. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, 
KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive 
though because VERSIONS has been used to specify the number of versions to 
retain (_num_versions_ in this example) since HBase version 0.1, so this is 
going to be a source of confusion - I've seen it in practice. 

A literate builder API, by way of the way we design its method names, could let 
a user describe more or less in speaking language how they want version 
retention to work, and internally the builder API could set the low level 
schema attributes. 


> Literate builder API for version management in schema
> -
>
> Key: HBASE-23678
> URL: https://issues.apache.org/jira/browse/HBASE-23678
> Project: HBase
>  Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
> KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
> regarding their usage. Almost all combinations of these four settings make 
> sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
> KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the 
> behavior with TTL easier to conceive when creating the schema. This could 
> take the form of a literate builder API for ColumnDescriptor or an extension 
> to an existing one. 
> Let me give you a motivating example: We may want to retain all versions for 
> a given TTL, and then only a specific number of versions after that interval 
> elapses. This can be achieved with VERSIONS=INT_MAX, 
> TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSIONS=_num_versions_ 
> . This is not intuitive though because VERSIONS has been used to specify the 
> number of versions to retain (_num_versions_ in this example) since HBase 
> version 0.1, so this is going to be a source of confusion - I've seen it in 
> practice. 
> A literate builder API, by way of the way we design its method names, could 
> let a user describe more or less in speaking language how they want version 
> retention to work, and internally the builder API could set the low level 
> schema attributes. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema

[Andrew Kyle Purtell (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-23678:

Description: 
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions after that interval 
elapses. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, 
KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive 
though because VERSIONS has been used to specify the number of versions to 
retain (_num_versions_ in this example) since HBase version 0.1, so this is 
going to be a source of confusion - I've seen it in practice. 

A literate builder API, by way of the way we design its method names, could let 
a user describe more or less in speaking language how they want version 
retention to work, and internally the builder API could set the low level 
schema attributes. 

  was:
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions. This can be achieved 
with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, 
MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has 
been used to specify the number of versions to retain (_num_versions_ in this 
example) since HBase version 0.1, so this is going to be a source of confusion 
- I've seen it in practice. 

A literate builder API, by way of the way we design its method names, could let 
a user describe more or less in speaking language how they want version 
retention to work, and internally the builder API could set the low level 
schema attributes. 


> Literate builder API for version management in schema
> -
>
> Key: HBASE-23678
> URL: https://issues.apache.org/jira/browse/HBASE-23678
> Project: HBase
>  Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
> KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
> regarding their usage. Almost all combinations of these four settings make 
> sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
> KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the 
> behavior with TTL easier to conceive when creating the schema. This could 
> take the form of a literate builder API for ColumnDescriptor or an extension 
> to an existing one. 
> Let me give you a motivating example: We may want to retain all versions for 
> a given TTL, and then only a specific number of versions after that interval 
> elapses. This can be achieved with VERSIONS=INT_MAX, 
> TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ 
> . This is not intuitive though because VERSIONS has been used to specify the 
> number of versions to retain (_num_versions_ in this example) since HBase 
> version 0.1, so this is going to be a source of confusion - I've seen it in 
> practice. 
> A literate builder API, by way of the way we design its method names, could 
> let a user describe more or less in speaking language how they want version 
> retention to work, and internally the builder API could set the low level 
> schema attributes. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema

[Andrew Kyle Purtell (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-23678:

Description: 
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions. This can be achieved 
with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, 
MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has 
been used to specify the number of versions to retain (_num_versions_ in this 
example) since HBase version 0.1, so this is going to be a source of confusion 
- I've seen it in practice. 

A literate builder API, by way of the way we design its method names, could let 
a user describe more or less in speaking language how they want version 
retention to work, and internally the builder API could set the low level 
schema attributes. 

  was:
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions. This can be achieved 
with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, 
MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has 
been used to specify the number of versions to retain (_num_versions_ in this 
example) since HBase version 0.1, so this is going to be a source of confusion 
- I've seen it in practice. 

A literate builder API, by way if its method names, could let a user describe 
more or less in speaking language how they want version retention to work, and 
internally the builder API could set the low level schema attributes. 


> Literate builder API for version management in schema
> -
>
> Key: HBASE-23678
> URL: https://issues.apache.org/jira/browse/HBASE-23678
> Project: HBase
>  Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
> KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
> regarding their usage. Almost all combinations of these four settings make 
> sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
> KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the 
> behavior with TTL easier to conceive when creating the schema. This could 
> take the form of a literate builder API for ColumnDescriptor or an extension 
> to an existing one. 
> Let me give you a motivating example: We may want to retain all versions for 
> a given TTL, and then only a specific number of versions. This can be 
> achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, 
> KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive 
> though because VERSIONS has been used to specify the number of versions to 
> retain (_num_versions_ in this example) since HBase version 0.1, so this is 
> going to be a source of confusion - I've seen it in practice. 
> A literate builder API, by way of the way we design its method names, could 
> let a user describe more or less in speaking language how they want version 
> retention to work, and internally the builder API could set the low level 
> schema attributes. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema

[Andrew Kyle Purtell (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-23678:

Description: 
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions. This can be achieved 
with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, 
MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has 
been used to specify the number of versions to retain (_num_versions_ in this 
example) since HBase version 0.1, so this is going to be a source of confusion 
- I've seen it in practice. 

A literate builder API, by way if its method names, could let a user describe 
more or less in speaking language how they want version retention to work, and 
internally the builder API could set the low level schema attributes. 

  was:
Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions. This can be achieved 
with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, 
MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has 
been used to specify _num_versions_ in this example since version 0.1.

A literate builder API, by way if its method names, could let a user describe 
more or less in speaking language how they want version retention to work, and 
internally the builder API could set the low level schema attributes. 


> Literate builder API for version management in schema
> -
>
> Key: HBASE-23678
> URL: https://issues.apache.org/jira/browse/HBASE-23678
> Project: HBase
>  Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
> KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
> regarding their usage. Almost all combinations of these four settings make 
> sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
> KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the 
> behavior with TTL easier to conceive when creating the schema. This could 
> take the form of a literate builder API for ColumnDescriptor or an extension 
> to an existing one. 
> Let me give you a motivating example: We may want to retain all versions for 
> a given TTL, and then only a specific number of versions. This can be 
> achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, 
> KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive 
> though because VERSIONS has been used to specify the number of versions to 
> retain (_num_versions_ in this example) since HBase version 0.1, so this is 
> going to be a source of confusion - I've seen it in practice. 
> A literate builder API, by way if its method names, could let a user describe 
> more or less in speaking language how they want version retention to work, 
> and internally the builder API could set the low level schema attributes. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23678) Literate builder API for version management in schema

[Andrew Kyle Purtell (Jira)]

Andrew Kyle Purtell created HBASE-23678:
---

 Summary: Literate builder API for version management in schema
 Key: HBASE-23678
 URL: https://issues.apache.org/jira/browse/HBASE-23678
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Kyle Purtell


Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and 
KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance 
regarding their usage. Almost all combinations of these four settings make 
sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and 
KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior 
with TTL easier to conceive when creating the schema. This could take the form 
of a literate builder API for ColumnDescriptor or an extension to an existing 
one. 

Let me give you a motivating example: We may want to retain all versions for a 
given TTL, and then only a specific number of versions. This can be achieved 
with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, 
MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has 
been used to specify _num_versions_ in this example since version 0.1.

A literate builder API, by way if its method names, could let a user describe 
more or less in speaking language how they want version retention to work, and 
internally the builder API could set the low level schema attributes. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23279) Switch default block encoding to ROW_INDEX_V1

[Michael Stack (Jira)]

[ 
https://issues.apache.org/jira/browse/HBASE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013140#comment-17013140
 ] 

Michael Stack commented on HBASE-23279:
---

How we doing here [~vjasani]? [~anoop.hbase] reminded me this was outstanding 
in his review of HBASE-21065. In my test runs enabling ROW_INDEX_V1 on 
hbase:meta, the TestGetClosestAtOrBefore fails (commented out here) and then 
the TestProcedurePriority fails too as it does for you here. What you thinking? 
I was going to dig in but asking here first.

> Switch default block encoding to ROW_INDEX_V1
> -
>
> Key: HBASE-23279
> URL: https://issues.apache.org/jira/browse/HBASE-23279
> Project: HBase
>  Issue Type: Wish
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Lars Hofhansl
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-23279.master.000.patch, 
> HBASE-23279.master.001.patch, HBASE-23279.master.002.patch, 
> HBASE-23279.master.003.patch
>
>
> Currently we set both block encoding and compression to NONE.
> ROW_INDEX_V1 has many advantages and (almost) no disadvantages (the hfiles 
> are slightly larger about 3% or so). I think that would a better default than 
> NONE.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on issue #1010: HBASE-23664 Upgrade JUnit to 4.13

[GitBox]

Apache-HBase commented on issue #1010: HBASE-23664 Upgrade JUnit to 4.13
URL: https://github.com/apache/hbase/pull/1010#issuecomment-573157065
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   7m 11s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch.  |
   ||| _ branch-1 Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   8m 43s |  branch-1 passed  |
   | +1 :green_heart: |  compile  |   1m 42s |  branch-1 passed with JDK 
v1.8.0_232  |
   | +1 :green_heart: |  compile  |   1m 49s |  branch-1 passed with JDK 
v1.7.0_242  |
   | +1 :green_heart: |  shadedjars  |   2m 49s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   2m 13s |  branch-1 passed with JDK 
v1.8.0_232  |
   | +1 :green_heart: |  javadoc  |   3m 46s |  branch-1 passed with JDK 
v1.7.0_242  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   2m  1s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 42s |  the patch passed with JDK 
v1.8.0_232  |
   | +1 :green_heart: |  javac  |   1m 42s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 45s |  the patch passed with JDK 
v1.7.0_242  |
   | +1 :green_heart: |  javac  |   1m 45s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | -1 :x: |  xml  |   0m  1s |  The patch has 1 ill-formed XML file(s).  |
   | +1 :green_heart: |  shadedjars  |   2m 43s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |   5m  0s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2.  |
   | +1 :green_heart: |  javadoc  |   2m  7s |  the patch passed with JDK 
v1.8.0_232  |
   | +1 :green_heart: |  javadoc  |   3m 48s |  the patch passed with JDK 
v1.7.0_242  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 147m 38s |  root in the patch failed.  |
   | +1 :green_heart: |  asflicense  |   0m 38s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 198m 52s |   |
   
   
   | Reason | Tests |
   |---:|:--|
   | XML | Parsing Error(s): |
   |   | pom.xml |
   | Failed junit tests | hadoop.hbase.client.TestAdmin2 |
   |   | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas |
   |   | hadoop.hbase.client.TestSnapshotDFSTemporaryDirectory |
   |   | hadoop.hbase.regionserver.TestCompactionState |
   |   | hadoop.hbase.security.token.TestGenerateDelegationToken |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.5 Server=19.03.5 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/1010 |
   | Optional Tests | dupname asflicense javac javadoc unit shadedjars 
hadoopcheck xml compile |
   | uname | Linux 84529f3a5ea7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1010/out/precommit/personality/provided.sh
 |
   | git revision | branch-1 / 6b940ae |
   | Default Java | 1.7.0_242 |
   | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_232 
/usr/lib/jvm/zulu-7-amd64:1.7.0_242 |
   | xml | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/artifact/out/xml.txt
 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/artifact/out/patch-unit-root.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/testReport/
 |
   | Max. process+thread count | 4560 (vs. ulimit of 1) |
   | modules | C: . U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/console |
   | versions | git=1.9.1 maven=3.0.5 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (HBASE-23668) Master log start filling with "Flush journal status" messages

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-23668.
---
Hadoop Flags: Reviewed
Assignee: Michael Stack
  Resolution: Fixed

Merged to branch-2+. Thanks for review [~zhangduo] (reopen to add your UT?).

> Master log start filling with "Flush journal status" messages
> -
>
> Key: HBASE-23668
> URL: https://issues.apache.org/jira/browse/HBASE-23668
> Project: HBase
>  Issue Type: Improvement
>  Components: proc-v2, RegionProcedureStore
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Takes a while to get into this condition. Not each to tell how because all 
> logs have rolled off and I only have logs filled w/ below:
> {code}
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226854 at 1578553261723
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226855 at 1578553261723
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226856 at 1578553261723
> 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
> Flush status journal:
>   Acquiring readlock on region at 1578553261723
>   Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, 
> failureReason:Nothing to flush,flush seq id45226857 at 1578553261723
> {code}
> ... I added the printing of flushresult... i.e. cannot flush because store is 
> empty.
> Digging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] saintstack commented on issue #1015: HBASE-23668 Master log start filling with "Flush journal status" messages

[GitBox]

saintstack commented on issue #1015: HBASE-23668  Master log start filling with 
"Flush journal status" messages
URL: https://github.com/apache/hbase/pull/1015#issuecomment-573152004
 
 
   Manually merged to branch-2 and master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hbase] saintstack closed pull request #1015: HBASE-23668 Master log start filling with "Flush journal status" messages

[GitBox]

saintstack closed pull request #1015: HBASE-23668  Master log start filling 
with "Flush journal status" messages
URL: https://github.com/apache/hbase/pull/1015
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services