date:20200723

[GitHub] [hbase] anoopsjohn commented on a change in pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll

2020-07-23 Thread GitBox



anoopsjohn commented on a change in pull request #2021:
URL: https://github.com/apache/hbase/pull/2021#discussion_r459256398



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java
##
@@ -58,31 +58,31 @@
 
   protected static final String WAL_ROLL_PERIOD_KEY = 
"hbase.regionserver.logroll.period";
 
-  protected final ConcurrentMap walNeedsRoll = new 
ConcurrentHashMap<>();
+  protected final ConcurrentMap wals = new 
ConcurrentHashMap<>();

Review comment:
   This will have all wal entries once the wal instance is created.  Not 
just when it needs a roll. So ya these were wals which needed roll at some 
point.  So wals name also fine IMO.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache9 commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



Apache9 commented on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662859269


   OK, just saw this on jira
   
   > They were in a separate WAL directory. The assumption here is that all the 
data has been flushed and there is no data needing to be replayed from the WAL.
   
   So this is not a normal operation. Typically, you can just disable all the 
tables, restart the whole cluster on a new environment, and then enable all the 
tables. The assumption of our implementation is that, a region server should 
have a WAL directory for it, unless it is dead and has already been processed 
by SCP.
   
   And I prefer we just add a new operation in HBCK2, to scan for unknown 
servers and schedule SCP for them? Or maybe we already have one in place? 
@saintstack Can you recall sir? I'm not very familiar with all the operations 
in HBCK2.
   
   Thanks.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] virajjasani commented on pull request #2119: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.2

2020-07-23 Thread GitBox



virajjasani commented on pull request #2119:
URL: https://github.com/apache/hbase/pull/2119#issuecomment-662862203


   Triggered one more run.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] virajjasani commented on pull request #2120: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.1

2020-07-23 Thread GitBox



virajjasani commented on pull request #2120:
URL: https://github.com/apache/hbase/pull/2120#issuecomment-662862754


   I think we don't have active builds for branch-2.1.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2122: HBASE-24743 Reject to add a peer which replicate to itself earlier

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2122:
URL: https://github.com/apache/hbase/pull/2122#issuecomment-662864609


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m  6s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  2s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 21s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   4m 41s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 31s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   6m 24s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 41s |  hbase-server in master failed.  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 14s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 32s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 30s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 30s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m 23s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 41s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 194m 57s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  unit  |   7m  3s |  hbase-shell in the patch passed.  |
   |  |   | 232m 28s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2122 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 9b5e2d90916e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/testReport/
 |
   | Max. process+thread count | 3542 (vs. ulimit of 12500) |
   | modules | C: hbase-server hbase-shell U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default

2020-07-23 Thread Anoop Sam John (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163290#comment-17163290
 ] 

Anoop Sam John commented on HBASE-23634:


bq.One question, if we only partially write an HFile and we don't complete 
(because crash splitting the WAL say), does it get sidelined, cleaned up? Just 
wondering. Thanks.
You mean when the next attempt of WAL split happens (by another RS).  Ya this 
has to happen.  I doubt whether we do this. Good point.

> Enable "Split WAL to HFile" by default
> --
>
> Key: HBASE-23634
> URL: https://issues.apache.org/jira/browse/HBASE-23634
> Project: HBase
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Guanghao Zhang
>Priority: Blocker
> Fix For: 3.0.0-alpha-1
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] taklwu commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



taklwu commented on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662866824


   yup, it should be only the case of ALL WALs (user table WAL and MasterRegion 
WALs) has been flushed but removed, it should have no data loss. You're right 
that the new set of region servers start on new environment that they are 
identified as 'Unknown server'. although it's not common on long running 
cluster, this is very common on a one-time cluster in a dynamic environment but 
StoreFiles and Tables has been persisted. 
   
   HBCKSCP and `scheduleRecoveries` were the manual recovery, but our point 
were comparing to branch-1.4+ that didn't have this issues where `dead` and 
`unknown` servers were expired and regions can be assigned (and if I found the 
history correctly, that logic were removed from 
[processofflineServersWithOnlineRegions -> processOfflineRegions in 
HBASE-20708](https://github.com/apache/hbase/commit/3e33aecea2da1071d7336584ed29448c233fe3be#diff-322aa95013b832abcc3fa2a396a70748L1310-L1357).
  



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] taklwu edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



taklwu edited a comment on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662866824


   yup, it should be only the case of ALL WALs (user table WAL and MasterRegion 
WALs) has been flushed but removed, it should have no data loss. You're right 
that the new set of region servers start on new environment that they are 
identified as 'Unknown server'. although it's not common on long running 
cluster, this is very common on a one-time cluster in a dynamic environment but 
StoreFiles and Tables has been persisted. 
   
   HBCKSCP and `scheduleRecoveries` were the manual recovery, but our point 
were comparing to branch-1.4+ that didn't have this issues where `dead` and 
`unknown` servers were expired and regions can be assigned (and if I found the 
history correctly, that logic were removed from 
[processofflineServersWithOnlineRegions -> processOfflineRegions in 
HBASE-20708](https://github.com/apache/hbase/commit/3e33aecea2da1071d7336584ed29448c233fe3be#diff-322aa95013b832abcc3fa2a396a70748L1310-L1357).
  
   
   So, I added this flag as a optional feature in the last commit to avoid the 
confusion to operator that has `WAL` covered, what do you guys think?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Sun Xin (Jira)

Sun Xin created HBASE-24759:
---

 Summary: Persisting configuration of default rsgroup
 Key: HBASE-24759
 URL: https://issues.apache.org/jira/browse/HBASE-24759
 Project: HBase
  Issue Type: New Feature
  Components: rsgroup
Affects Versions: 3.0.0-alpha-1
Reporter: Sun Xin
Assignee: Sun Xin
 Fix For: 3.0.0-alpha-1


In the current scenario, we didn't store the default rsgroup information. But 
after HBASE-24431 , we have added a config map, which need to be persisted to 
avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2077: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2077:
URL: https://github.com/apache/hbase/pull/2077#issuecomment-662867634


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   3m  4s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-24666 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 27s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 45s |  HBASE-24666 passed  |
   | +1 :green_heart: |  compile  |   2m 39s |  HBASE-24666 passed  |
   | +1 :green_heart: |  shadedjars  |   6m  3s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 42s |  HBASE-24666 passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 13s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 47s |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 38s |  the patch passed  |
   | +1 :green_heart: |  javac  |   2m 38s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  3s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 43s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   0m 47s |  hbase-protocol-shaded in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 13s |  hbase-client in the patch passed.  
|
   | +1 :green_heart: |  unit  | 209m 17s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  unit  |   5m 27s |  hbase-thrift in the patch passed.  
|
   |  |   | 250m 26s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2077 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux cbd25848ac76 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-24666 / fae9f0cd51 |
   | Default Java | 1.8.0_232 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/testReport/
 |
   | Max. process+thread count | 3898 (vs. ulimit of 12500) |
   | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift 
U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Sun Xin (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sun Xin updated HBASE-24759:

Issue Type: Bug  (was: New Feature)

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] taklwu edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



taklwu edited a comment on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662866824


   yup, it should be only the case of ALL WALs (user table WAL and MasterRegion 
WALs) has been flushed but removed, it should have no data loss. You're right 
that the new set of region servers start on new environment that they are 
identified as 'Unknown server'. although it's not common on long running 
cluster, this is very common on a one-time cluster in a dynamic environment but 
StoreFiles and Tables has been persisted. 
   
   HBCKSCP and `scheduleRecoveries` were the manual recovery, but our point 
were comparing to branch-1.4+ that didn't have this issues where `dead` and 
`unknown` servers were expired and regions can be assigned (and if I found the 
history correctly, that logic were removed from 
[processofflineServersWithOnlineRegions -> processOfflineRegions in 
HBASE-20708](https://github.com/apache/hbase/commit/3e33aecea2da1071d7336584ed29448c233fe3be#diff-322aa95013b832abcc3fa2a396a70748L1310-L1357).
 Other than this missing `WAL` situation, somehow current unknown servers are 
showing up in the case of SCP or other cases. although we were from the `no 
WAL` scenario, should this change heal the cluster itself? 
   
   as a second thought, I added this flag 
`hbase.catalogjanitor.repair.unknown.servers` as a optional feature in the last 
commit to avoid the confusion to operator that has `WAL` covered, what do you 
guys think?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default

2020-07-23 Thread ramkrishna.s.vasudevan (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163315#comment-17163315
 ] 

ramkrishna.s.vasudevan commented on HBASE-23634:


We don't do that. If 'hbase.skip.errors' set to true we will atleast make the 
region open go ahead with out those files. If we don 't set it we will fail the 
region open. 

> Enable "Split WAL to HFile" by default
> --
>
> Key: HBASE-23634
> URL: https://issues.apache.org/jira/browse/HBASE-23634
> Project: HBase
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Guanghao Zhang
>Priority: Blocker
> Fix For: 3.0.0-alpha-1
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2077: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2077:
URL: https://github.com/apache/hbase/pull/2077#issuecomment-662874012


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 46s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-24666 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 31s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   5m 10s |  HBASE-24666 passed  |
   | +1 :green_heart: |  compile  |   4m  6s |  HBASE-24666 passed  |
   | +1 :green_heart: |  shadedjars  |   7m 25s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 29s |  hbase-client in HBASE-24666 failed. 
 |
   | -0 :warning: |  javadoc  |   0m 50s |  hbase-server in HBASE-24666 failed. 
 |
   | -0 :warning: |  javadoc  |   1m 10s |  hbase-thrift in HBASE-24666 failed. 
 |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 15s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 27s |  the patch passed  |
   | +1 :green_heart: |  compile  |   4m 18s |  the patch passed  |
   | +1 :green_heart: |  javac  |   4m 18s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   7m 19s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 36s |  hbase-client in the patch failed.  |
   | -0 :warning: |  javadoc  |   0m 57s |  hbase-server in the patch failed.  |
   | -0 :warning: |  javadoc  |   1m 18s |  hbase-thrift in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 18s |  hbase-protocol-shaded in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 42s |  hbase-client in the patch passed.  
|
   | +1 :green_heart: |  unit  | 212m 50s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  unit  |   5m  3s |  hbase-thrift in the patch passed.  
|
   |  |   | 265m 37s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2077 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 0025876bdfbf 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-24666 / fae9f0cd51 |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/testReport/
 |
   | Max. process+thread count | 3670 (vs. ulimit of 12500) |
   | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift 
U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache9 commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



Apache9 commented on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662876047


   > You're right that the new set of region servers start on new environment 
that they are identified as 'Unknown server'.
   
   I think the old server will be unknown servers, not the new servers?
   
   I do not think there is a guarantee that you change the filesystem layout of 
HBase internal, and HBase cluster will still be functional. Even if sometimes 
it could, as you said, on 1.4, it does not mean that we will always keep this 
behavior in new versions.
   
   For your scenario, it is OK as you can confirm that there is no data loss as 
you manually flushed all the data. But what if another user just configs the 
wrong WAL directory? In this case, if we schedule SCPs automatically, there 
will be data loss.
   
   In general, in HBase, we only know that something strange happen, but we do 
not know how we come into this situation so it is not safe for us to just 
recover automatically. Only end user knows, so I still prefer we just provide 
tools for them to recover, not add some 'unsafe' options in HBase...
   
   Thanks.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2122: HBASE-24743 Reject to add a peer which replicate to itself earlier

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2122:
URL: https://github.com/apache/hbase/pull/2122#issuecomment-662878052


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   2m 45s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  2s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 21s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 58s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   6m  3s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 50s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 15s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 48s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 17s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 17s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  3s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 229m  8s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  unit  |   7m 15s |  hbase-shell in the patch passed.  |
   |  |   | 265m 51s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2122 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 707aab7f8377 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 
11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 1.8.0_232 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/testReport/
 |
   | Max. process+thread count | 3358 (vs. ulimit of 12500) |
   | modules | C: hbase-server hbase-shell U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Duo Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1716#comment-1716
 ] 

Duo Zhang commented on HBASE-24759:
---

Ah, this is a pain, hope we do not need to persist the region servers in 
default rs group?

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Sun Xin (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163342#comment-17163342
 ] 

Sun Xin commented on HBASE-24759:
-

{quote}this is a pain, hope we do not need to persist the region servers in 
default rs group?
{quote}
If we persist the default rsgroup's information except servers and tables, for 
now, only persist config of default rs group.

Is that OK?

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2124:
URL: https://github.com/apache/hbase/pull/2124#issuecomment-662888285


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 31s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  7s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ branch-2 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 17s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   4m 40s |  branch-2 passed  |
   | +1 :green_heart: |  compile  |   1m 35s |  branch-2 passed  |
   | +1 :green_heart: |  shadedjars  |   6m 24s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 57s |  branch-2 passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m  4s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 32s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 32s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m 11s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 57s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 223m 33s |  hbase-server in the patch passed.  
|
   | -1 :x: |  unit  |   7m  8s |  hbase-shell in the patch failed.  |
   |  |   | 261m 21s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2124 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 00d68a21d13d 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 
11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | branch-2 / ce4e692699 |
   | Default Java | 1.8.0_232 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-shell.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/testReport/
 |
   | Max. process+thread count | 2534 (vs. ulimit of 12500) |
   | modules | C: hbase-server hbase-shell U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] ddupg opened a new pull request #2126: HBASE-24759 Persisting configuration of default rsgroup

2020-07-23 Thread GitBox



ddupg opened a new pull request #2126:
URL: https://github.com/apache/hbase/pull/2126


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Pankaj Kumar (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163346#comment-17163346
 ] 

Pankaj Kumar commented on HBASE-24759:
--

Any use case scenario to store additional info in default rsgroup? 

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Pankaj Kumar (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163346#comment-17163346
 ] 

Pankaj Kumar edited comment on HBASE-24759 at 7/23/20, 8:45 AM:


Any use case scenario to store additional info in default rsgroup?  HBASE-22820 
do not persist default group in rsgroup table.


was (Author: pankajkumar):
Any use case scenario to store additional info in default rsgroup? 

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Duo Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163350#comment-17163350
 ] 

Duo Zhang commented on HBASE-24759:
---

[~pankajkumar] HBASE-22820 is the time where the only thing in RSGroup is 
region servers, so we do not need to persist anything for the default rs group, 
as all the region servers do not belong to any rs groups will be in default rs 
group.

But in HBASE-24431 we introduced a config map for rs group, so we need to 
persist at least the config map of the default rs group.

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Pankaj Kumar (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163356#comment-17163356
 ] 

Pankaj Kumar commented on HBASE-24759:
--

Yeah [~zhangduo], I'm inline to HBASE-24431. Persisting additional info for 
user rsgroup make sense to me but do we allow users to set extra details for 
default rsgroup?

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default

2020-07-23 Thread Guanghao Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163364#comment-17163364
 ] 

Guanghao Zhang commented on HBASE-23634:


{quote}We don't do that. If 'hbase.skip.errors' set to true we will atleast 
make the region open go ahead with out those files. If we don 't set it we will 
fail the region open.
{quote}
Yes. But I thought these should be cleanup directly when open region.
{quote}Here if we do write the HFiles directly under thd CF, we have a chance 
that we will get to see a partially written HFile. 
{quote}
What is the affect for other systems(offline compaction tool)?

> Enable "Split WAL to HFile" by default
> --
>
> Key: HBASE-23634
> URL: https://issues.apache.org/jira/browse/HBASE-23634
> Project: HBase
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Guanghao Zhang
>Priority: Blocker
> Fix For: 3.0.0-alpha-1
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-22561) modify HFilePrettyPrinter to accept non-hbase.rootdir directories

2020-07-23 Thread gurmukh singh (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-22561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163371#comment-17163371
 ] 

gurmukh singh commented on HBASE-22561:
---

Thanks [~wchevreuil] , I am on 2.1.0. Will upgrade in the coming nothing.

> modify HFilePrettyPrinter to accept non-hbase.rootdir directories
> -
>
> Key: HBASE-22561
> URL: https://issues.apache.org/jira/browse/HBASE-22561
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.2.0, 2.3.0, 2.1.5
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.0.6, 2.2.1, 2.1.6
>
> Attachments: HBASE-22561.v06.patch, HBASE-22561.v07.patch, 
> HBASE-22561.v11.patch, HBASE-22561.v12.patch, non_root_dir.png, root_dir.png
>
>
> Today {{hbase hfile}} utility accepts only _hbase.rootDir_ directory. It is 
> more useful to pass other locations with HFile as well.
> {code:java}
> hbase org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter hfileoutput
> ...
> ERROR, file (hfileoutput) is not in HBase's root directory 
> (hdfs://hadoop.example.com:9000/apps/hbase/)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2126: HBASE-24759 Persisting configuration of default rsgroup

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2126:
URL: https://github.com/apache/hbase/pull/2126#issuecomment-662907673


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 30s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 23s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 36s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 29s |  master passed  |
   | +1 :green_heart: |  spotbugs  |   2m 40s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 13s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 21s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m 26s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  hadoopcheck  |  11m 19s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   2m 59s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 27s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  35m 48s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2126 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti 
checkstyle |
   | uname | Linux babc727a0286 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Max. process+thread count | 94 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) 
spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Sun Xin (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163383#comment-17163383
 ] 

Sun Xin commented on HBASE-24759:
-

In HBASE-24431 , an idea was mentioned
{quote}Use special RSGroup config to overright the cluster config. Then we can 
config different RSGroup to use different config in future.
{quote}
[~pankajkumar] If this idea is implemented, could region servers in default rs 
group use different config based on this issue?

 

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] ramkrish86 commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751

2020-07-23 Thread GitBox



ramkrish86 commented on pull request #2125:
URL: https://github.com/apache/hbase/pull/2125#issuecomment-662916393


   Seems the null check was causing some issues as the finally was not getting 
executed. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Pankaj Kumar (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163396#comment-17163396
 ] 

Pankaj Kumar commented on HBASE-24759:
--

Ok, got it.. 

Recently we met a blocking issue (in HBase-2.2.3) where all system tables were 
assigned to default rsgroup and due to some reason all RSs (only 1 RS is in 
default rsgroup) of default rsgroup went down. Hmaster stuck while persisting 
default rsgroup info to rsgroup table (since no RS in deafult rsgroup, so meta 
and other system region were not assigned). Even new RS join the default 
rsgroup, AM couldn't pick them as candidate since rsGroupMap will be updated 
after persisting default rsgroup to rsgroup table.  HBASE-22820 addressed the 
problem.

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-11288) Splittable Meta

2020-07-23 Thread Francis Christopher Liu (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163398#comment-17163398
 ] 

Francis Christopher Liu commented on HBASE-11288:
-

[~stack] and [~zhangduo] thank you for your suggestions of avoiding master 
branch and just rebasing on branch-2. I will go ahead and do that then. I will 
try to get as much information of course. I’ll do my best to get ITBLL to pass. 

[~zhangduo] Good to hear you are willing to change your mind with a reasonable 
result. As my intention has always been not only to prove/disprove it to myself 
but to you and [~stack] as well to help come to a consensus on a path forward.

Looking at [~stack]’s recent comment the consensus seems to be that there was 
no consensus on a path forward. Although as [~stack] pointed out [~apurtell] 
did suggest we just do a reset so I missed the signs that we were doing that so 
my mistake. 

> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
>  Issue Type: Umbrella
>  Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2095: HBASE-24459 Move the locateMeta logic from AsyncMetaRegionTableLocato…

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2095:
URL: https://github.com/apache/hbase/pull/2095#issuecomment-662926329


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 15s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ HBASE-11288.splittable-meta Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 29s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 50s |  HBASE-11288.splittable-meta 
passed  |
   | +1 :green_heart: |  checkstyle  |   2m  4s |  HBASE-11288.splittable-meta 
passed  |
   | +1 :green_heart: |  spotbugs  |   3m 45s |  HBASE-11288.splittable-meta 
passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 12s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 43s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   0m 24s |  The patch passed checkstyle 
in hbase-common  |
   | +1 :green_heart: |  checkstyle  |   0m 27s |  The patch passed checkstyle 
in hbase-client  |
   | +1 :green_heart: |  checkstyle  |   1m 14s |  hbase-server: The patch 
generated 0 new + 148 unchanged - 1 fixed = 148 total (was 149)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  hadoopcheck  |  12m 20s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   4m 16s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 33s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  42m 50s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2095 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti 
checkstyle |
   | uname | Linux c42110f3288d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-11288.splittable-meta / 1e358188ac |
   | Max. process+thread count | 84 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-client hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) 
spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2125:
URL: https://github.com/apache/hbase/pull/2125#issuecomment-662927465


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m  2s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m  2s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 13s |  master passed  |
   | +1 :green_heart: |  spotbugs  |   2m  8s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 44s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m 11s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  1s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  hadoopcheck  |  12m 20s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   2m 17s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 12s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  35m 47s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2125 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti 
checkstyle |
   | uname | Linux 36fbf11808fd 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Max. process+thread count | 84 (vs. ulimit of 12500) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) 
spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-11288) Splittable Meta

2020-07-23 Thread Duo Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163415#comment-17163415
 ] 

Duo Zhang commented on HBASE-11288:
---

{quote}
I’ll do my best to get ITBLL to pass.
{quote}

You do not need to get ITBLL to pass. I do not think the work can be done 
within a reasonable time. [~ndimiduk] has spent several month to archive this.

Instead, set a deadline, 1 week or 2 weeks, then we come back to analyze the 
result, on how to explain the failures, and also the complexity of the code to 
archive the current state. Of course it will be good if you can pass the ITBLL.

And an interesting thing is that, I offer that, I will change my mind if you 
can get a reasonable result, then what can make you change your mind? We both 
have a solution, I do not think it is fair to me that, only I offer how to 
change my mind, but you do not offer anything to change your mind.

Thanks.

> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
>  Issue Type: Umbrella
>  Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24696) Include JVM information on Web UI under "Software Attributes"

2020-07-23 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163417#comment-17163417
 ] 

Hudson commented on HBASE-24696:


Results for branch branch-1
[build #1332 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332//JDK8_Nightly_Build_Report_(Hadoop2)/]




(x) {color:red}-1 source release artifact{color}
-- See build output for details.


> Include JVM information on Web UI under "Software Attributes"
> -
>
> Key: HBASE-24696
> URL: https://issues.apache.org/jira/browse/HBASE-24696
> Project: HBase
>  Issue Type: Improvement
>  Components: UI
>Reporter: Nick Dimiduk
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
> Attachments: Screen Shot 2020-07-17 at 10.55.56 PM.png
>
>
> It's a small thing, but seems like an omission.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default

2020-07-23 Thread Anoop Sam John (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163420#comment-17163420
 ] 

Anoop Sam John commented on HBASE-23634:


The incomplete HFiles will be under the cf/recovered.edits dir then.  When we 
try load those files to region/cf, we will have the validation and it will fail 
for such incomplete files. What will happen for such fail? This says we can not 
really avoid that validation step while loading the recovered.edits HFiles.
bq.What is the affect for other systems(offline compaction tool)?
It will see such files are corrupt. The files might not have a proper trailer. 
So every such external systems to handle such cases. If such files, just 
ignore.  Then one issue is how the system can know whether it is a partially 
written file or a real corruption?

> Enable "Split WAL to HFile" by default
> --
>
> Key: HBASE-23634
> URL: https://issues.apache.org/jira/browse/HBASE-23634
> Project: HBase
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Guanghao Zhang
>Priority: Blocker
> Fix For: 3.0.0-alpha-1
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup

2020-07-23 Thread Sun Xin (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163426#comment-17163426
 ] 

Sun Xin commented on HBASE-24759:
-

I think the core problem you [~pankajkumar] had is that the system tables went 
unavailable.

In HBASE-22738 we allow tables fallback to specific rs groups, but for system 
tables, maybe we should allow them fallback to any rsgroup to keep always 
available.

Let me do this on a new jira.

> Persisting configuration of default rsgroup
> ---
>
> Key: HBASE-24759
> URL: https://issues.apache.org/jira/browse/HBASE-24759
> Project: HBase
>  Issue Type: Bug
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In the current scenario, we didn't store the default rsgroup information. But 
> after HBASE-24431 , we have added a config map, which need to be persisted to 
> avoid lossing config of default rsgroup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HBASE-24752) NPE/500 accessing webui on master startup

2020-07-23 Thread wenfeiyi666 (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-24752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wenfeiyi666 reassigned HBASE-24752:
---

Assignee: wenfeiyi666

> NPE/500 accessing webui on master startup
> -
>
> Key: HBASE-24752
> URL: https://issues.apache.org/jira/browse/HBASE-24752
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: wenfeiyi666
>Priority: Minor
>
> I got a 500 error while accessing the master web ui (non-backup master) 
> during startup.
> {noformat}
> 2020-07-21 18:56:39,762 WARN org.eclipse.jetty.servlet.ServletHandler: 
> /master-status
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:339)
> at 
> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:397)
> at 
> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:388)
> at 
> org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:79)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> ...
> {noformat}
> {{MasterStatusTmplImpl.java:339}} appears to align with 
> {{MasterStatusTmpl.jamon:227}}:
> {noformat}
>   <%if 
> master.getMasterCoprocessorHost().findCoprocessor("RSGroupAdminEndpoint") != 
> null &&
> serverManager.getOnlineServersList().size() > 0 %>
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24760) Allow system tables fallback to any rs groups

2020-07-23 Thread Sun Xin (Jira)

Sun Xin created HBASE-24760:
---

 Summary: Allow system tables fallback to any rs groups
 Key: HBASE-24760
 URL: https://issues.apache.org/jira/browse/HBASE-24760
 Project: HBase
  Issue Type: New Feature
  Components: rsgroup
Affects Versions: 3.0.0-alpha-1
Reporter: Sun Xin
Assignee: Sun Xin
 Fix For: 3.0.0-alpha-1


In HBASE-22738 we allow tables fallback to specific rs groups, If there is no 
online servers in the table's rsgroup.

But for system tables, It is necessary to allow system tables fallback to any 
rsgroup in order to keey available at all times.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] ramkrish86 commented on a change in pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll

2020-07-23 Thread GitBox



ramkrish86 commented on a change in pull request #2021:
URL: https://github.com/apache/hbase/pull/2021#discussion_r459374807



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java
##
@@ -58,31 +58,31 @@
 
   protected static final String WAL_ROLL_PERIOD_KEY = 
"hbase.regionserver.logroll.period";
 
-  protected final ConcurrentMap walNeedsRoll = new 
ConcurrentHashMap<>();
+  protected final ConcurrentMap wals = new 
ConcurrentHashMap<>();

Review comment:
   Ok





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (HBASE-24760) Allow system tables fallback to any rs groups

2020-07-23 Thread Sun Xin (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-24760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sun Xin updated HBASE-24760:

Description: 
In HBASE-22738 we allow tables fallback to specific rs groups, If there is no 
online servers in the table's rsgroup.

But for system tables, if there is no specified fallback rsgroup or the servers 
in the fallback rsgroup all went down, It is necessary to allow system tables 
fallback to any rsgroup in order to keey available at all times.

  was:
In HBASE-22738 we allow tables fallback to specific rs groups, If there is no 
online servers in the table's rsgroup.

But for system tables, It is necessary to allow system tables fallback to any 
rsgroup in order to keey available at all times.


> Allow system tables fallback to any rs groups
> -
>
> Key: HBASE-24760
> URL: https://issues.apache.org/jira/browse/HBASE-24760
> Project: HBase
>  Issue Type: New Feature
>  Components: rsgroup
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> In HBASE-22738 we allow tables fallback to specific rs groups, If there is no 
> online servers in the table's rsgroup.
> But for system tables, if there is no specified fallback rsgroup or the 
> servers in the fallback rsgroup all went down, It is necessary to allow 
> system tables fallback to any rsgroup in order to keey available at all times.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default

2020-07-23 Thread Anoop Sam John (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163438#comment-17163438
 ] 

Anoop Sam John commented on HBASE-23634:


bq.Then one issue is how the system can know whether it is a partially written 
file or a real corruption?
This is applicable when we do split wals to HFiles and during that there was 
some wal file split failed in between and reattempted.  End of the day there 
might be some HFiles which are duplicate.  Some HFiles may be incomplete.  Now 
while reading back the file (verification while loading to cf) we dont know 
whether this is a failed attempt's partial HFile or a real corrupted file once 
it was written to FS.So on wal file split failure, a cleanup before next 
attempt is imp. Now one problem is there might be N WAL files and all splits 
will create the HFiles for a region:cf under same dir 
region/cf/recovered.edits.   Now if we wanted this cleanup, these files should 
have been generated with some way to identify them as result of which wal 
file's split.  Say if HFile was placed under region/cf/recovered.edits/ dir, we could have cleaned it up before doing the next attempt. 
Thoughts?

> Enable "Split WAL to HFile" by default
> --
>
> Key: HBASE-23634
> URL: https://issues.apache.org/jira/browse/HBASE-23634
> Project: HBase
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Guanghao Zhang
>Priority: Blocker
> Fix For: 3.0.0-alpha-1
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2126: HBASE-24759 Persisting configuration of default rsgroup

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2126:
URL: https://github.com/apache/hbase/pull/2126#issuecomment-662955563


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 31s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 22s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   4m  9s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 28s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m 49s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 21s |  hbase-common in master failed.  |
   | -0 :warning: |  javadoc  |   0m 39s |  hbase-server in master failed.  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m  4s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 29s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 29s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   5m 45s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 19s |  hbase-common in the patch failed.  |
   | -0 :warning: |  javadoc  |   0m 39s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 31s |  hbase-common in the patch passed.  
|
   | +1 :green_heart: |  unit  | 131m 50s |  hbase-server in the patch passed.  
|
   |  |   | 161m 32s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2126 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux a28da9d3a1e3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/testReport/
 |
   | Max. process+thread count | 4299 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] ramkrish86 commented on a change in pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll

2020-07-23 Thread GitBox



ramkrish86 commented on a change in pull request #2021:
URL: https://github.com/apache/hbase/pull/2021#discussion_r459384595



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java
##
@@ -148,53 +147,49 @@ private void abort(String reason, Throwable cause) {
   @Override
   public void run() {
 while (running) {
-  boolean periodic = false;
   long now = System.currentTimeMillis();
   checkLowReplication(now);
-  periodic = (now - this.lastRollTime) > this.rollPeriod;
-  if (periodic) {
-// Time for periodic roll, fall through
-LOG.debug("WAL roll period {} ms elapsed", this.rollPeriod);
-  } else {
-synchronized (this) {
-  if (walNeedsRoll.values().stream().anyMatch(Boolean::booleanValue)) {
-// WAL roll requested, fall through
-LOG.debug("WAL roll requested");
-  } else {
-try {
-  wait(this.threadWakeFrequency);
-} catch (InterruptedException e) {
-  // restore the interrupt state
-  Thread.currentThread().interrupt();
-}
-// goto the beginning to check whether again whether we should 
fall through to roll
-// several WALs, and also check whether we should quit.
-continue;
+  synchronized (this) {
+if (wals.values().stream().noneMatch(rc -> rc.needsRoll(now))) {
+  try {
+wait(this.threadWakeFrequency);
+  } catch (InterruptedException e) {
+// restore the interrupt state
+Thread.currentThread().interrupt();
   }
+  // goto the beginning to check whether again whether we should fall 
through to roll
+  // several WALs, and also check whether we should quit.
+  continue;
 }
   }
   try {
-this.lastRollTime = System.currentTimeMillis();
-for (Iterator> iter = 
walNeedsRoll.entrySet().iterator(); iter
-  .hasNext();) {
-  Entry entry = iter.next();
+for (Iterator> iter = 
wals.entrySet().iterator();
+ iter.hasNext();) {
+  Entry entry = iter.next();
   WAL wal = entry.getKey();
-  // reset the flag in front to avoid missing roll request before we 
return from rollWriter.
-  walNeedsRoll.put(wal, Boolean.FALSE);
-  Map> regionsToFlush = null;
+  RollController controller = entry.getValue();
+  if (controller.isRollRequested()) {
+// WAL roll requested, fall through
+LOG.debug("WAL {} roll requested", wal);
+  } else if (controller.needsPeriodicRoll(now)){
+// Time for periodic roll, fall through
+LOG.debug("WAL {} roll period {} ms elapsed", wal, 
this.rollPeriod);
+  } else {
+continue;

Review comment:
   This is they key. We ensure we only roll if one of the condition is met 
(if either size reach caused a log roll ) or the time elapsed. That is also 
tracked per wal. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] anoopsjohn commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



anoopsjohn commented on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329


   >Typically, you can just disable all the tables, restart the whole cluster 
on a new environment, and then enable all the tables.
   Agree to Duo. That might be a better path for such use case where the 
cluster is deleted (after persisting all data) and later create pointing to 
existing FS.
   >In general, in HBase, we only know that something strange happen, but we do 
not know how we come into this situation so it is not safe for us to just 
recover automatically
   Ya I am also agreeing to this point.
   In 2.x we store the table's disable/enable state in META.  So we will have 
that info available in the newly recreated cluster also. (The zk data from old 
cluster also wont be there in new cluster I believe)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] anoopsjohn edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



anoopsjohn edited a comment on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329


   >Typically, you can just disable all the tables, restart the whole cluster 
on a new environment, and then enable all the tables.<
   Agree to Duo. That might be a better path for such use case where the 
cluster is deleted (after persisting all data) and later create pointing to 
existing FS.
   >In general, in HBase, we only know that something strange happen, but we do 
not know how we come into this situation so it is not safe for us to just 
recover automatically
   Ya I am also agreeing to this point.
   In 2.x we store the table's disable/enable state in META.  So we will have 
that info available in the newly recreated cluster also. (The zk data from old 
cluster also wont be there in new cluster I believe)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] anoopsjohn edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



anoopsjohn edited a comment on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329


   >Typically, you can just disable all the tables, restart the whole cluster 
on a new environment, and then enable all the tables.
   
   Agree to Duo. That might be a better path for such use case where the 
cluster is deleted (after persisting all data) and later create pointing to 
existing FS.
   >In general, in HBase, we only know that something strange happen, but we do 
not know how we come into this situation so it is not safe for us to just 
recover automatically
   Ya I am also agreeing to this point.
   In 2.x we store the table's disable/enable state in META.  So we will have 
that info available in the newly recreated cluster also. (The zk data from old 
cluster also wont be there in new cluster I believe)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] anoopsjohn edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…

2020-07-23 Thread GitBox



anoopsjohn edited a comment on pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329


   >Typically, you can just disable all the tables, restart the whole cluster 
on a new environment, and then enable all the tables.
   
   Agree to Duo. That might be a better path for such use case where the 
cluster is deleted (after persisting all data) and later create pointing to 
existing FS.
   >In general, in HBase, we only know that something strange happen, but we do 
not know how we come into this situation so it is not safe for us to just 
recover automatically
   
   Ya I am also agreeing to this point.
   In 2.x we store the table's disable/enable state in META.  So we will have 
that info available in the newly recreated cluster also. (The zk data from old 
cluster also wont be there in new cluster I believe)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24665) all wal of RegionGroupingProvider together roll

2020-07-23 Thread Anoop Sam John (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163464#comment-17163464
 ] 

Anoop Sam John commented on HBASE-24665:


Will wait till tomorrow if we have any more comments.  If none, will commit the 
PR.

> all wal of RegionGroupingProvider together roll
> ---
>
> Key: HBASE-24665
> URL: https://issues.apache.org/jira/browse/HBASE-24665
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.3.0, master, 2.1.10, 1.4.14, 2.2.6
>Reporter: wenfeiyi666
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.1.10, 1.4.14, 2.2.7
>
>
> when use multiwal, any a wal request roll, all wal will be together roll.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-24665) MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll

2020-07-23 Thread Anoop Sam John (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-24665:
---
Summary: MultiWAL :  Avoid rolling of ALL WALs when one of the WAL needs a 
roll  (was: all wal of RegionGroupingProvider together roll)

> MultiWAL :  Avoid rolling of ALL WALs when one of the WAL needs a roll
> --
>
> Key: HBASE-24665
> URL: https://issues.apache.org/jira/browse/HBASE-24665
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.3.0, master, 2.1.10, 1.4.14, 2.2.6
>Reporter: wenfeiyi666
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.1.10, 1.4.14, 2.2.7
>
>
> when use multiwal, any a wal request roll, all wal will be together roll.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2126: HBASE-24759 Persisting configuration of default rsgroup

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2126:
URL: https://github.com/apache/hbase/pull/2126#issuecomment-662962818


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 28s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 23s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 34s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m 35s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 59s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 26s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 16s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 16s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   5m 34s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 58s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 22s |  hbase-common in the patch passed.  
|
   | +1 :green_heart: |  unit  | 153m 18s |  hbase-server in the patch passed.  
|
   |  |   | 180m 43s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2126 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 08eb90886ca5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 1.8.0_232 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/testReport/
 |
   | Max. process+thread count | 3997 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2119: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.2

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2119:
URL: https://github.com/apache/hbase/pull/2119#issuecomment-662966432


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 49s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   | -0 :warning: |  test4tests  |   0m  0s |  The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch.  |
   ||| _ branch-2.2 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 15s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   5m 33s |  branch-2.2 passed  |
   | +1 :green_heart: |  compile  |   2m  3s |  branch-2.2 passed  |
   | +1 :green_heart: |  checkstyle  |   2m 25s |  branch-2.2 passed  |
   | +1 :green_heart: |  shadedjars  |   4m 24s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 26s |  branch-2.2 passed  |
   | +0 :ok: |  spotbugs  |   1m 37s |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   6m  3s |  branch-2.2 passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 14s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m  8s |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m  4s |  the patch passed  |
   | +1 :green_heart: |  javac  |   2m  4s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   2m 23s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  shadedjars  |   4m 25s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  hadoopcheck  |  27m 22s |  Patch does not cause any 
errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1.  |
   | +1 :green_heart: |  javadoc  |   1m 27s |  the patch passed  |
   | +1 :green_heart: |  findbugs  |   5m 58s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 173m 50s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  unit  |   4m  4s |  hbase-thrift in the patch passed.  
|
   | +1 :green_heart: |  unit  |   6m 10s |  hbase-rest in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 16s |  The patch does not generate 
ASF License warnings.  |
   |  |   | 264m 27s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2119/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2119 |
   | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs 
shadedjars hadoopcheck hbaseanti checkstyle compile |
   | uname | Linux 8d7614015d03 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-2119/out/precommit/personality/provided.sh
 |
   | git revision | branch-2.2 / 2c08876d70 |
   | Default Java | 1.8.0_181 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2119/2/testReport/
 |
   | Max. process+thread count | 5344 (vs. ulimit of 1) |
   | modules | C: hbase-server hbase-thrift hbase-rest U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2119/2/console |
   | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2124:
URL: https://github.com/apache/hbase/pull/2124#issuecomment-662967670


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 13s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ branch-2 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 14s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   4m 15s |  branch-2 passed  |
   | +1 :green_heart: |  checkstyle  |   1m 27s |  branch-2 passed  |
   | +1 :green_heart: |  spotbugs  |   2m  6s |  branch-2 passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 12s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 32s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m 27s |  the patch passed  |
   | -0 :warning: |  rubocop  |   0m  7s |  The patch generated 62 new + 294 
unchanged - 17 fixed = 356 total (was 311)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  hadoopcheck  |  12m 36s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   2m 13s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 23s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  37m 40s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2124 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti 
checkstyle rubocop |
   | uname | Linux 3c2c145f9b15 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | branch-2 / ce4e692699 |
   | rubocop | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt
 |
   | Max. process+thread count | 84 (vs. ulimit of 12500) |
   | modules | C: hbase-server hbase-shell U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) 
spotbugs=3.1.12 rubocop=0.80.0 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2095: HBASE-24459 Move the locateMeta logic from AsyncMetaRegionTableLocato…

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2095:
URL: https://github.com/apache/hbase/pull/2095#issuecomment-662975082


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 30s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  4s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-11288.splittable-meta Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 22s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 37s |  HBASE-11288.splittable-meta 
passed  |
   | +1 :green_heart: |  compile  |   1m 42s |  HBASE-11288.splittable-meta 
passed  |
   | +1 :green_heart: |  shadedjars  |   5m 38s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 24s |  HBASE-11288.splittable-meta 
passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 17s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 26s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 44s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 44s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   5m 30s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 21s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 25s |  hbase-common in the patch passed.  
|
   | +1 :green_heart: |  unit  |   1m  3s |  hbase-client in the patch passed.  
|
   | -1 :x: |  unit  | 139m 17s |  hbase-server in the patch failed.  |
   |  |   | 170m  7s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2095 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 6b1036ef1a82 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-11288.splittable-meta / 1e358188ac |
   | Default Java | 1.8.0_232 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/testReport/
 |
   | Max. process+thread count | 4066 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-client hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2125:
URL: https://github.com/apache/hbase/pull/2125#issuecomment-662978376


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 29s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  2s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 25s |  master passed  |
   | +1 :green_heart: |  compile  |   0m 52s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m 36s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 35s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 23s |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 56s |  the patch passed  |
   | +1 :green_heart: |  javac  |   0m 56s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   5m 36s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 145m  6s |  hbase-server in the patch failed.  |
   |  |   | 168m 31s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2125 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux b302ab71068d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 1.8.0_232 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/testReport/
 |
   | Max. process+thread count | 4245 (vs. ulimit of 12500) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2095: HBASE-24459 Move the locateMeta logic from AsyncMetaRegionTableLocato…

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2095:
URL: https://github.com/apache/hbase/pull/2095#issuecomment-662979595


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 52s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ HBASE-11288.splittable-meta Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 24s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   5m 29s |  HBASE-11288.splittable-meta 
passed  |
   | +1 :green_heart: |  compile  |   2m 19s |  HBASE-11288.splittable-meta 
passed  |
   | +1 :green_heart: |  shadedjars  |   7m 44s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 32s |  hbase-client in 
HBASE-11288.splittable-meta failed.  |
   | -0 :warning: |  javadoc  |   0m 20s |  hbase-common in 
HBASE-11288.splittable-meta failed.  |
   | -0 :warning: |  javadoc  |   0m 48s |  hbase-server in 
HBASE-11288.splittable-meta failed.  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   5m 23s |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 27s |  the patch passed  |
   | +1 :green_heart: |  javac  |   2m 27s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   7m 34s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 19s |  hbase-common in the patch failed.  |
   | -0 :warning: |  javadoc  |   0m 29s |  hbase-client in the patch failed.  |
   | -0 :warning: |  javadoc  |   0m 49s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 49s |  hbase-common in the patch passed.  
|
   | +1 :green_heart: |  unit  |   1m 20s |  hbase-client in the patch passed.  
|
   | -1 :x: |  unit  | 139m 20s |  hbase-server in the patch failed.  |
   |  |   | 181m  7s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2095 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 1a2c5b15603e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | HBASE-11288.splittable-meta / 1e358188ac |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/testReport/
 |
   | Max. process+thread count | 4406 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-client hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2125:
URL: https://github.com/apache/hbase/pull/2125#issuecomment-662980878


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 45s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 52s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 17s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   6m 57s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 50s |  hbase-server in master failed.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 52s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 17s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 17s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   7m  9s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 49s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 143m 15s |  hbase-server in the patch failed.  |
   |  |   | 174m 23s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2125 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 6fe797610be1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/testReport/
 |
   | Max. process+thread count | 4868 (vs. ulimit of 12500) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking

2020-07-23 Thread Anoop Sam John (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163542#comment-17163542
 ] 

Anoop Sam John commented on HBASE-24749:


bq.On events such as flush and compaction, we write markers to the WAL w/ notes 
listing files that participated in the event. On recovery, we read these events 
completing compactions if all participants present and it looked like we 
crashed after compaction completed but before we got to slot the new files into 
place and remove the old.
This is not only for META (or ROOT) table you suggesting stack? (But 
generically)  Ya during flush and compaction we have start markers with files 
involved in that op and on completion another marker.  During region recovery, 
we can use these markers to identify uncommitted files right? Need to see 
whether there is chance that we wall gets rolled and miss the start markers. 

> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
>  Issue Type: Umbrella
>  Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
>  Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct 
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}} 
> directory used in the commit stage for common HFile operations such as flush 
> and compaction to improve the write throughput and latency on object stores. 
> Specifically for S3 filesystems, this will also mitigate read-after-write 
> inconsistencies caused by immediate HFiles validation after moving the 
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with 
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, 
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed 
> improvement on the object stores use case makes senses and if we miss 
> anything should be included.
> Improvement Highlights
>  1. Lower write latency, especially the p99+
>  2. Higher write throughput on flush and compaction 
>  3. Lower MTTR on region (re)open or assignment 
>  4. Remove consistent check dependencies (e.g. DynamoDB) supported by file 
> system implementation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking

2020-07-23 Thread ramkrishna.s.vasudevan (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163552#comment-17163552
 ] 

ramkrishna.s.vasudevan commented on HBASE-24749:


I think in flushes this is possible to recover from a start marker (assuming we 
will abort on a flush failure). But for compactions I think the markers in the 
WAL may get rolled over, right?

> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
>  Issue Type: Umbrella
>  Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
>  Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct 
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}} 
> directory used in the commit stage for common HFile operations such as flush 
> and compaction to improve the write throughput and latency on object stores. 
> Specifically for S3 filesystems, this will also mitigate read-after-write 
> inconsistencies caused by immediate HFiles validation after moving the 
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with 
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, 
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed 
> improvement on the object stores use case makes senses and if we miss 
> anything should be included.
> Improvement Highlights
>  1. Lower write latency, especially the p99+
>  2. Higher write throughput on flush and compaction 
>  3. Lower MTTR on region (re)open or assignment 
>  4. Remove consistent check dependencies (e.g. DynamoDB) supported by file 
> system implementation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24632) Enable procedure-based log splitting as default in hbase3

2020-07-23 Thread Anoop Sam John (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163551#comment-17163551
 ] 

Anoop Sam John commented on HBASE-24632:


So for every WAL file, it will add a sub procedure in ProcedureExecutor and 
that in turn add entries for RemoteProcExecutor correct?  So at this stage also 
these Sub proc executing threads will yield once the split proc is submitted to 
RemoteProcExecutor ?  And there in that executor we have 128 threads by 
default. Will that wait once it issue split request to RS for split?  Or will 
that also yield? If former, then we will get big backlog in a bigger cluster 
with many RS down and every having many WALs to split

> Enable procedure-based log splitting as default in hbase3
> -
>
> Key: HBASE-24632
> URL: https://issues.apache.org/jira/browse/HBASE-24632
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Reporter: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Means changing this value in HConstants to false:
>public static final boolean DEFAULT_HBASE_SPLIT_COORDINATED_BY_ZK = true;
> Should probably also deprecate the current zk distributed split too so we can 
> clear out those classes to.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2125:
URL: https://github.com/apache/hbase/pull/2125#issuecomment-662996848


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 29s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 47s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m  6s |  master passed  |
   | +1 :green_heart: |  spotbugs  |   1m 59s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   3m 26s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   1m  3s |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  hadoopcheck  |  11m 15s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   2m  8s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 16s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  32m 31s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2125 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti 
checkstyle |
   | uname | Linux 9b05678ad07a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Max. process+thread count | 94 (vs. ulimit of 12500) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) 
spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking

2020-07-23 Thread Duo Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163558#comment-17163558
 ] 

Duo Zhang commented on HBASE-24749:
---

IIRC there is a issue for storing the compacted files in HFile's metadata, to 
solve the problem that the wal file contains the compaction marker may be 
deleted before wal splitting.

> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
>  Issue Type: Umbrella
>  Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
>  Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct 
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}} 
> directory used in the commit stage for common HFile operations such as flush 
> and compaction to improve the write throughput and latency on object stores. 
> Specifically for S3 filesystems, this will also mitigate read-after-write 
> inconsistencies caused by immediate HFiles validation after moving the 
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with 
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, 
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed 
> improvement on the object stores use case makes senses and if we miss 
> anything should be included.
> Improvement Highlights
>  1. Lower write latency, especially the p99+
>  2. Higher write throughput on flush and compaction 
>  3. Lower MTTR on region (re)open or assignment 
>  4. Remove consistent check dependencies (e.g. DynamoDB) supported by file 
> system implementation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-23285) walArchiveDir does not exist when separate the old WALs into different regionserver directories

2020-07-23 Thread xuqinya (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xuqinya updated HBASE-23285:

Summary: walArchiveDir does not exist when separate the old WALs into 
different regionserver directories  (was: Sometimes walArchiveDir does not 
exist when separate the old WALs into different regionserver directories)

> walArchiveDir does not exist when separate the old WALs into different 
> regionserver directories
> ---
>
> Key: HBASE-23285
> URL: https://issues.apache.org/jira/browse/HBASE-23285
> Project: HBase
>  Issue Type: Bug
>Affects Versions: master
>Reporter: xuqinya
>Assignee: xuqinya
>Priority: Minor
>
> [HBASE-14247|https://issues.apache.org/jira/browse/HBASE-14247] Add a new 
> config hbase.separate.oldlogdir.by.regionserver, the old wal dir will be 
> separated by regionservers. 
> But,if there are no files, the regionserver directory will be cleared. 
> CommonFSUtils.renameAndSetModifyTime(this.fs, p, newPath) will throw 
> exception.
> {code:java}
> java.io.IOException: Unable to rename 
> hdfs://127.0.0.1:59066/user/xuqinya/test-data/0836eb3c-cf89-ffdd-5ca1-54049cb50c15/WALs/169.254.31.146,59229,1573636627328/169.254.31.146%2C59229%2C1573636627328.1573636642613
>  to 
> hdfs://127.0.0.1:59066/user/xuqinya/test-data/0836eb3c-cf89-ffdd-5ca1-54049cb50c15/oldWALs/169.254.31.146,59229,1573636627328/169.254.31.146%2C59229%2C1573636627328.1573636642613
>   at 
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:735)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.cleanOldLogs(AbstractFSWAL.java:706)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:849)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:540)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] virajjasani opened a new pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold

2020-07-23 Thread GitBox



virajjasani opened a new pull request #2127:
URL: https://github.com/apache/hbase/pull/2127


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Work started] (HBASE-24757) ReplicationSink should limit the batch size for batch mutations based on hbase.rpc.rows.warning.threshold

2020-07-23 Thread Viraj Jasani (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-24757 started by Viraj Jasani.

> ReplicationSink should limit the batch size for batch mutations based on 
> hbase.rpc.rows.warning.threshold
> -
>
> Key: HBASE-24757
> URL: https://issues.apache.org/jira/browse/HBASE-24757
> Project: HBase
>  Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>
> At times there are quite a large no of WAL Edits to ship as part of 
> Replication and sometimes replication queues accumulate huge list of Edits to 
> process. ReplicationSink at the sink server usually goes through all Edits 
> and creates map of table -> list of rows grouped by clusterIds, and performs 
> batch mutation of all rows per table level. However, there is no limit to no 
> of Rows that are sent as part of batch mutate call. If no of rows > limit 
> threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn 
> "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is 
> turned on, RS will reject the whole batch without processing.
> We should let Replication Sink honour this threshold value and accordingly 
> keep the size lower per batch mutation call.
> Replication triggered batch mutations should always be consumed but keeping 
> limit of mutation low enough will let the system function at the same pace 
> and without triggering warnings. This will also restrict exploitation of heap 
> and cpu cycles at the destination.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking

2020-07-23 Thread Anoop Sam John (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163565#comment-17163565
 ] 

Anoop Sam John commented on HBASE-24749:


Ya that will help for deleting the compacted away files while opening the 
result file.
But the other case where we write compaction result file directly under 
region/cf and we had the start compaction marker in wal. The compaction did not 
finish and RS crashed. By that time the wal file which had the start compaction 
marker got rolled and deleted also.  Now while replay of wal, we dont have 
start compaction marker for this wal file. So we think this is an old valid 
file but that is wrong. This is  a partial file.  This is possible.  On that 
approach of solving the issue with WAL marker, this is a possible issue (?) 

> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
>  Issue Type: Umbrella
>  Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
>  Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct 
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}} 
> directory used in the commit stage for common HFile operations such as flush 
> and compaction to improve the write throughput and latency on object stores. 
> Specifically for S3 filesystems, this will also mitigate read-after-write 
> inconsistencies caused by immediate HFiles validation after moving the 
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with 
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, 
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed 
> improvement on the object stores use case makes senses and if we miss 
> anything should be included.
> Improvement Highlights
>  1. Lower write latency, especially the p99+
>  2. Higher write throughput on flush and compaction 
>  3. Lower MTTR on region (re)open or assignment 
>  4. Remove consistent check dependencies (e.g. DynamoDB) supported by file 
> system implementation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24761) Support hadoop 3.3.0

2020-07-23 Thread Duo Zhang (Jira)

Duo Zhang created HBASE-24761:
-

 Summary: Support hadoop 3.3.0
 Key: HBASE-24761
 URL: https://issues.apache.org/jira/browse/HBASE-24761
 Project: HBase
  Issue Type: Umbrella
  Components: hadoop3
Reporter: Duo Zhang


A good thing is that, finally hadoop upgraded protobuf and also shaded the 
protobuf it uses.

Let's try to support it, and also we can purge our own dependencies on protobuf 
2.5.0.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] brfrn169 commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side

2020-07-23 Thread GitBox



brfrn169 commented on pull request #2094:
URL: https://github.com/apache/hbase/pull/2094#issuecomment-663010093


   @joshelser Thank you very much for reviewing this!
   
   I will wait for any other opinions/objections until tomorrow. If nothing, 
will commit this.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (HBASE-24762) Purge protobuf java 2.5.0 dependency

2020-07-23 Thread Duo Zhang (Jira)

Duo Zhang created HBASE-24762:
-

 Summary: Purge protobuf java 2.5.0 dependency
 Key: HBASE-24762
 URL: https://issues.apache.org/jira/browse/HBASE-24762
 Project: HBase
  Issue Type: Sub-task
  Components: dependencies, Protobufs
Reporter: Duo Zhang
Assignee: Duo Zhang
 Fix For: 3.0.0-alpha-1


On master branch, we have removed the hbase-protocol module so in general, we 
do not need to depend on protobuf 2.5.0 directl. Especially for hadoop 3.3.0, 
hadoop will not depend on 2.5.0 any more, we should make sure hbase do not 
introduce protobuf 2.5.0 too.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-24762) Purge protobuf java 2.5.0 dependency

2020-07-23 Thread Duo Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-24762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-24762:
--
Parent Issue: HBASE-22953  (was: HBASE-24761)

> Purge protobuf java 2.5.0 dependency
> 
>
> Key: HBASE-24762
> URL: https://issues.apache.org/jira/browse/HBASE-24762
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies, Protobufs
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> On master branch, we have removed the hbase-protocol module so in general, we 
> do not need to depend on protobuf 2.5.0 directl. Especially for hadoop 3.3.0, 
> hadoop will not depend on 2.5.0 any more, we should make sure hbase do not 
> introduce protobuf 2.5.0 too.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24761) Support hadoop 3.3.0

2020-07-23 Thread Duo Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-24761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-24761.
---
Resolution: Duplicate

> Support hadoop 3.3.0
> 
>
> Key: HBASE-24761
> URL: https://issues.apache.org/jira/browse/HBASE-24761
> Project: HBase
>  Issue Type: Umbrella
>  Components: hadoop3
>Reporter: Duo Zhang
>Priority: Major
>
> A good thing is that, finally hadoop upgraded protobuf and also shaded the 
> protobuf it uses.
> Let's try to support it, and also we can purge our own dependencies on 
> protobuf 2.5.0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2127:
URL: https://github.com/apache/hbase/pull/2127#issuecomment-663023627


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 37s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 23s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 42s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   1m 34s |  master passed  |
   | +1 :green_heart: |  spotbugs  |   2m 42s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 14s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 30s |  the patch passed  |
   | -0 :warning: |  checkstyle  |   1m  4s |  hbase-server: The patch 
generated 1 new + 80 unchanged - 0 fixed = 81 total (was 80)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  hadoopcheck  |  11m 38s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   3m 49s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 26s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  39m 22s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2127 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti 
checkstyle |
   | uname | Linux 698adefbb8b0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | checkstyle | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
 |
   | Max. process+thread count | 94 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) 
spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache9 commented on a change in pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side

2020-07-23 Thread GitBox



Apache9 commented on a change in pull request #2094:
URL: https://github.com/apache/hbase/pull/2094#discussion_r459471919



##
File path: 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
##
@@ -86,6 +86,12 @@
*/
   void updateCheckAndPut(long t);
 
+  /**
+   * Update checkAndMutate histogram
+   * @param t time it took
+   */
+  void updateCheckAndMutate(long t);

Review comment:
   After this change, do we still have checkAndPut and checkAndDelete?

##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -4272,43 +4275,96 @@ protected Durability getEffectiveDurability(Durability 
d) {
   }
 
   @Override
+  @Deprecated
   public boolean checkAndMutate(byte[] row, byte[] family, byte[] qualifier, 
CompareOperator op,
 ByteArrayComparable comparator, TimeRange timeRange, Mutation mutation) 
throws IOException {
-return doCheckAndRowMutate(row, family, qualifier, op, comparator, null, 
timeRange, null,
-  mutation);
+CheckAndMutate.Builder builder = CheckAndMutate.newBuilder(row)
+  .ifMatches(family, qualifier, op, 
comparator.getValue()).timeRange(timeRange);
+try {
+  if (mutation instanceof Put) {
+return checkAndMutate(builder.build((Put) mutation)).isSuccess();
+  } else if (mutation instanceof Delete) {
+return checkAndMutate(builder.build((Delete) mutation)).isSuccess();
+  } else {
+throw new DoNotRetryIOException(
+  "Unsupported mutate type: " + 
mutation.getClass().getSimpleName().toUpperCase());
+  }
+} catch (IllegalArgumentException e) {
+  throw new DoNotRetryIOException(e.getMessage());
+}
   }
 
   @Override
+  @Deprecated
   public boolean checkAndMutate(byte[] row, Filter filter, TimeRange 
timeRange, Mutation mutation)
 throws IOException {
-return doCheckAndRowMutate(row, null, null, null, null, filter, timeRange, 
null, mutation);
+CheckAndMutate.Builder builder = 
CheckAndMutate.newBuilder(row).ifMatches(filter)
+  .timeRange(timeRange);
+try {
+  if (mutation instanceof Put) {
+return checkAndMutate(builder.build((Put) mutation)).isSuccess();
+  } else if (mutation instanceof Delete) {
+return checkAndMutate(builder.build((Delete) mutation)).isSuccess();
+  } else {
+throw new DoNotRetryIOException("Unsupported mutate type: " +
+  mutation.getClass().getSimpleName().toUpperCase());
+  }
+} catch (IllegalArgumentException e) {

Review comment:
   Ditto.

##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
##
@@ -759,12 +810,150 @@ default boolean 
postCheckAndDelete(ObserverContext
* @param delete delete to commit if check succeeds
* @param result from the CheckAndDelete
* @return the possibly transformed returned value to return to client
+   *
+   * @deprecated since 3.0.0 and will be removed in 4.0.0. Use
+   *   {@link #postCheckAndMutate(ObserverContext, CheckAndMutate, 
CheckAndMutateResult)} instead.
*/
+  @Deprecated
   default boolean 
postCheckAndDelete(ObserverContext c, byte[] row,
 Filter filter, Delete delete, boolean result) throws IOException {
 return result;
   }
 
+  /**
+   * Called before checkAndMutate
+   * 
+   * Call CoprocessorEnvironment#bypass to skip default actions.
+   * If 'bypass' is set, we skip out on calling any subsequent chained 
coprocessors.
+   * 
+   * Note: Do not retain references to any Cells in actions beyond the life of 
this invocation.
+   * If need a Cell reference for later use, copy the cell and use that.
+   * @param c the environment provided by the region server
+   * @param checkAndMutate the CheckAndMutate object
+   * @param result the default value of the result
+   * @return the return value to return to client if bypassing default 
processing
+   * @throws IOException if an error occurred on the coprocessor
+   */
+  default CheckAndMutateResult 
preCheckAndMutate(ObserverContext c,
+CheckAndMutate checkAndMutate, CheckAndMutateResult result) throws 
IOException {
+if (checkAndMutate.getAction() instanceof Put) {

Review comment:
   Put/Delete is public so let's not do this...

##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -4335,32 +4391,14 @@ private boolean doCheckAndRowMutate(byte[] row, byte[] 
family, byte[] qualifier,
   checkRow(row, "doCheckAndRowMutate");
   RowLock rowLock = getRowLockInternal(get.getRow(), false, null);
   try {
-if (mutation != null && this.getCoprocessorHost() != null) {
-  // Call coprocessor.
-  Boolean processed = null;
-  if (mutation instanceof Put) {
-if (filter != null) {
-  processed = this.getCoprocessorHost()

Review comment:
   So the old coprocessor method

[GitHub] [hbase] Apache9 opened a new pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency

2020-07-23 Thread GitBox



Apache9 opened a new pull request #2128:
URL: https://github.com/apache/hbase/pull/2128


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-19256) [hbase-thirdparty] shade jetty

2020-07-23 Thread Duo Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163611#comment-17163611
 ] 

Duo Zhang commented on HBASE-19256:
---

Hadoop 3.3.0 is out. Let's move forward. [~weichiu] Hope you do not mind if I 
take this issue. It has been dead for a lont time.

Thanks.

> [hbase-thirdparty] shade jetty
> --
>
> Key: HBASE-19256
> URL: https://issues.apache.org/jira/browse/HBASE-19256
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, thirdparty
>Reporter: Mike Drob
>Assignee: Wei-Chiu Chuang
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] Apache-HBase commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2094:
URL: https://github.com/apache/hbase/pull/2094#issuecomment-663032975


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |  11m  6s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  hbaseanti  |   0m  0s |  Patch does not have any 
anti-patterns.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 23s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   4m 14s |  master passed  |
   | +1 :green_heart: |  checkstyle  |   2m 10s |  master passed  |
   | +1 :green_heart: |  spotbugs  |   3m 54s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 12s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 54s |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   0m 16s |  The patch passed checkstyle 
in hbase-hadoop-compat  |
   | +1 :green_heart: |  checkstyle  |   0m 33s |  The patch passed checkstyle 
in hbase-client  |
   | +1 :green_heart: |  checkstyle  |   1m 17s |  hbase-server: The patch 
generated 0 new + 410 unchanged - 22 fixed = 410 total (was 432)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  hadoopcheck  |  12m 50s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   | +1 :green_heart: |  spotbugs  |   5m 28s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 42s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  56m 28s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.9 Server=19.03.9 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2094 |
   | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti 
checkstyle |
   | uname | Linux 06564bdd9af8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Max. process+thread count | 84 (vs. ulimit of 12500) |
   | modules | C: hbase-hadoop-compat hbase-client hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) 
spotbugs=3.1.12 |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2124:
URL: https://github.com/apache/hbase/pull/2124#issuecomment-663044535


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 39s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  7s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ branch-2 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 30s |  branch-2 passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  branch-2 passed  |
   | +1 :green_heart: |  shadedjars  |   5m 12s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 49s |  branch-2 passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 20s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 18s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   5m 10s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 47s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 151m 24s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  unit  |   7m 28s |  hbase-shell in the patch passed.  |
   |  |   | 183m 54s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2124 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux cf99393fcc77 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | branch-2 / ce4e692699 |
   | Default Java | 1.8.0_232 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/testReport/
 |
   | Max. process+thread count | 3431 (vs. ulimit of 12500) |
   | modules | C: hbase-server hbase-shell U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2128:
URL: https://github.com/apache/hbase/pull/2128#issuecomment-663044354


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 29s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files 
found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any 
@author tags.  |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 30s |  Maven dependency ordering for branch  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 12s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace 
issues.  |
   | +1 :green_heart: |  xml  |   0m  6s |  The patch has no ill-formed XML 
file.  |
   | +1 :green_heart: |  hadoopcheck  |  14m 43s |  Patch does not cause any 
errors with Hadoop 3.1.2 3.2.1.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   1m  0s |  The patch does not generate 
ASF License warnings.  |
   |  |   |  27m  7s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-general-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2128 |
   | Optional Tests | dupname asflicense hadoopcheck xml |
   | uname | Linux 816d7ca9ebc0 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Max. process+thread count | 63 (vs. ulimit of 12500) |
   | modules | C: . hbase-client hbase-common hbase-examples hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2124:
URL: https://github.com/apache/hbase/pull/2124#issuecomment-663046009


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 39s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  5s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ branch-2 Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 15s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   4m 18s |  branch-2 passed  |
   | +1 :green_heart: |  compile  |   1m 32s |  branch-2 passed  |
   | +1 :green_heart: |  shadedjars  |   5m 54s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 42s |  hbase-server in branch-2 failed.  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 21s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 21s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 52s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 52s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   8m 10s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 50s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 141m 20s |  hbase-server in the patch passed.  
|
   | +1 :green_heart: |  unit  |   7m  8s |  hbase-shell in the patch passed.  |
   |  |   | 180m 22s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2124 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 61ffb573f6e8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | branch-2 / ce4e692699 |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/testReport/
 |
   | Max. process+thread count | 4167 (vs. ulimit of 12500) |
   | modules | C: hbase-server hbase-shell U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24738) [Shell] processlist command fails with ERROR: Unexpected end of file from server when SSL enabled

2020-07-23 Thread Elliot Miller (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163642#comment-17163642
 ] 

Elliot Miller commented on HBASE-24738:
---

Hey [~pankajkumar], thanks for doing this! Sorry for the delay.

My understanding is that so far, client configurations have been pretty simple 
and just need either a Zookeeper-based or master-based connection registry. The 
issue would seem to be that there is no notion of the protocol (ie. HTTP or 
HTTPS).

The implementation uncommented by your PR seems like it should work, although 
it uses configuration keys intended for the server. +It seems like more of a 
"workaround" solution than the ideal one, but maybe that is okay+.

The main alternatives I see:
 # We could have the client attempt an HTTP connection and try HTTPS if the 
first connection gets dropped.
 # We could allow users to specify SSL/TLS in the processlist command itself.

It would be great to settle on a pattern for any commands that need to connect 
to the info port of our region servers.

Worth a footnote is that we might be able to have Jetty upgrade HTTP to HTTPS 
automatically, but that would be server-side and perhaps a more long-term 
change.

It may be okay to go with your proposed changes, but I wanted to mention my 
thoughts first. What do you think?

> [Shell] processlist command fails with ERROR: Unexpected end of file from 
> server when SSL enabled
> -
>
> Key: HBASE-24738
> URL: https://issues.apache.org/jira/browse/HBASE-24738
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
>
> HBase Shell command "processlist" fails with ERROR: Unexpected end of file 
> from server when HBase SSL enabled.
>  
> Below code is commented since beginning, see HBASE-4368,
> [https://github.com/apache/hbase/blob/8076eafb187ce32d4a78aef482b6218d85a985ac/hbase-shell/src/main/ruby/hbase/taskmonitor.rb#L85]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] bitoffdev commented on pull request #2123: HBASE-24738 [Shell] processlist command fails with ERROR: Unexpected end of file from server when SSL enabled

2020-07-23 Thread GitBox



bitoffdev commented on pull request #2123:
URL: https://github.com/apache/hbase/pull/2123#issuecomment-663047251


   > @bitoffdev please review.
   
   - Just left a comment in Jira: 
https://issues.apache.org/jira/browse/HBASE-24738?focusedCommentId=17163642#comment-17163642
   - The changes should work, but I want to hear your thoughts first on what I 
just mentioned in Jira.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase-thirdparty] Apache9 opened a new pull request #22: HBASE-19256 [hbase-thirdparty] shade jetty

2020-07-23 Thread GitBox



Apache9 opened a new pull request #22:
URL: https://github.com/apache/hbase-thirdparty/pull/22


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase-thirdparty] Apache9 commented on pull request #22: HBASE-19256 [hbase-thirdparty] shade jetty

2020-07-23 Thread GitBox



Apache9 commented on pull request #22:
URL: https://github.com/apache/hbase-thirdparty/pull/22#issuecomment-663065833


   Still need to take a look at the jersey part. We need to exclude the general 
API jar, just like what I have done for jetty, to exclude servlet-api.
   
   > [INFO] Including 
org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.31 in the 
shaded jar.
   [INFO] Including org.glassfish.hk2.external:jakarta.inject:jar:2.6.1 in the 
shaded jar.
   [INFO] Including org.glassfish.jersey.core:jersey-common:jar:2.31 in the 
shaded jar.
   [INFO] Including org.glassfish.hk2:osgi-resource-locator:jar:1.0.3 in the 
shaded jar.
   [INFO] Including jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6 in the shaded jar.
   [INFO] Including org.glassfish.jersey.core:jersey-client:jar:2.31 in the 
shaded jar.
   [INFO] Including org.glassfish.jersey.core:jersey-server:jar:2.31 in the 
shaded jar.
   [INFO] Including org.glassfish.jersey.media:jersey-media-jaxb:jar:2.31 in 
the shaded jar.
   [INFO] Including jakarta.annotation:jakarta.annotation-api:jar:1.3.5 in the 
shaded jar.
   [INFO] Including jakarta.validation:jakarta.validation-api:jar:2.0.2 in the 
shaded jar.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] virajjasani commented on a change in pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency

2020-07-23 Thread GitBox



virajjasani commented on a change in pull request #2128:
URL: https://github.com/apache/hbase/pull/2128#discussion_r459532587



##
File path: hbase-examples/pom.xml
##
@@ -205,6 +205,7 @@
 
   com.google.protobuf
   protobuf-java
+  ${internal.protobuf.version}

Review comment:
   We are keeping only in hbase-examples, sounds good.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2094:
URL: https://github.com/apache/hbase/pull/2094#issuecomment-663094953


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 31s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 23s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 38s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 37s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m 39s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 20s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 15s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 23s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 38s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 38s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   5m 32s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   1m 18s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   0m 34s |  hbase-hadoop-compat in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m  3s |  hbase-client in the patch passed.  
|
   | +1 :green_heart: |  unit  | 139m 22s |  hbase-server in the patch passed.  
|
   |  |   | 168m 52s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2094 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 50c6da3d5a3d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 1.8.0_232 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/testReport/
 |
   | Max. process+thread count | 4723 (vs. ulimit of 12500) |
   | modules | C: hbase-hadoop-compat hbase-client hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2127:
URL: https://github.com/apache/hbase/pull/2127#issuecomment-663095071


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   0m 37s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 24s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   3m 43s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 21s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   5m 42s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 59s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 31s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 16s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 16s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   5m 32s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 57s |  the patch passed  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 16s |  hbase-common in the patch passed.  
|
   | +1 :green_heart: |  unit  | 141m 14s |  hbase-server in the patch passed.  
|
   |  |   | 169m 17s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2127 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 53a4726e0545 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 1.8.0_232 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/testReport/
 |
   | Max. process+thread count | 5124 (vs. ulimit of 12500) |
   | modules | C: hbase-common hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…

2020-07-23 Thread GitBox



wchevreuil commented on a change in pull request #2118:
URL: https://github.com/apache/hbase/pull/2118#discussion_r459570210



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java
##
@@ -513,8 +514,14 @@ public boolean replicate(ReplicateContext 
replicateContext) {
 
 int numSinks = replicationSinkMgr.getNumSinks();
 if (numSinks == 0) {
-  LOG.warn("{} No replication sinks found, returning without replicating. "
-+ "The source should retry with the same set of edits.", logPeerId());
+  if((System.currentTimeMillis() - lastSinkFetchTime) >= 
(maxRetriesMultiplier*1000)) {
+LOG.warn(
+  "No replication sinks found, returning without replicating. "
++ "The source should retry with the same set of edits. Not logging 
this again for "
++ "the next " + maxRetriesMultiplier + " seconds.");
+lastSinkFetchTime = System.currentTimeMillis();
+  }
+  sleepForRetries("No sinks available at peer", sleepMultiplier);

Review comment:
   peer id is already been logged inside _sleepForRetries_ method.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…

2020-07-23 Thread GitBox



wchevreuil commented on a change in pull request #2118:
URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java
##
@@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer 
sinkPeer) {
*/
   public synchronized void chooseSinks() {
 List slaveAddresses = endpoint.getRegionServers();
-Collections.shuffle(slaveAddresses, ThreadLocalRandom.current());
-int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
-sinks = slaveAddresses.subList(0, numSinks);
+if(slaveAddresses==null){

Review comment:
   > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding 
against custom endpoint implementations?
   
   Yeah, that's indeed the case. 
   
   >We should expose getRegionServers on a base-class or interface and 
explicitly say that we expect a non-null answer. Follow-on..
   
   If easy, this would be good to write a quick unit test to cover this method.
   
   Sounds good, actually. So we can leave `getRegionServers` at 
HBaseReplicationEndpoint, emphasize it should never return null and add UT.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…

2020-07-23 Thread GitBox



wchevreuil commented on a change in pull request #2118:
URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java
##
@@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer 
sinkPeer) {
*/
   public synchronized void chooseSinks() {
 List slaveAddresses = endpoint.getRegionServers();
-Collections.shuffle(slaveAddresses, ThreadLocalRandom.current());
-int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
-sinks = slaveAddresses.subList(0, numSinks);
+if(slaveAddresses==null){

Review comment:
   > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding 
against custom endpoint implementations?
   
   Yeah, that's indeed the case. 
   
   >We should expose getRegionServers on a base-class or interface and 
explicitly say that we expect a non-null answer. Follow-on..
   If easy, this would be good to write a quick unit test to cover this method.
   
   Sounds good, actually. So we can leave `getRegionServers` at 
HBaseReplicationEndpoint, emphasize it should never return null and add UT.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…

2020-07-23 Thread GitBox



wchevreuil commented on a change in pull request #2118:
URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java
##
@@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer 
sinkPeer) {
*/
   public synchronized void chooseSinks() {
 List slaveAddresses = endpoint.getRegionServers();
-Collections.shuffle(slaveAddresses, ThreadLocalRandom.current());
-int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
-sinks = slaveAddresses.subList(0, numSinks);
+if(slaveAddresses==null){

Review comment:
   > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding 
against custom endpoint implementations?
   
   Yeah, that's indeed the case. 
   
   >We should expose getRegionServers on a base-class or interface and 
explicitly say that we expect a non-null answer. Follow-on..
   If easy, this would be good to write a quick unit test to cover this method.
   
   Sounds good, actually. So we can leave `getRegionServers` at 
HBaseReplicationEndpoint, emphasize it should never return null and add UT.
   
   > So maybe we can safely log warning for if(slaveAddresses.size()==0) ?
   
   I guess we can leave it without the extra logging here, as this would 
already cause the extra logging we are suppressing in 
HBaseInterClusterReplicationEndpoint #518. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…

2020-07-23 Thread GitBox



wchevreuil commented on a change in pull request #2118:
URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java
##
@@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer 
sinkPeer) {
*/
   public synchronized void chooseSinks() {
 List slaveAddresses = endpoint.getRegionServers();
-Collections.shuffle(slaveAddresses, ThreadLocalRandom.current());
-int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
-sinks = slaveAddresses.subList(0, numSinks);
+if(slaveAddresses==null){

Review comment:
   > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding 
against custom endpoint implementations?
   
   Yeah, that's indeed the case. 
   
   >We should expose getRegionServers on a base-class or interface and 
explicitly say that we expect a non-null answer. Follow-on..
   If easy, this would be good to write a quick unit test to cover this method.
   
   Sounds good, actually. So we can leave `getRegionServers` at 
HBaseReplicationEndpoint, emphasize it should never return null and add UT.
   
   > So maybe we can safely log warning for if(slaveAddresses.size()==0) ?
   I guess we can leave it without the extra logging here, as this would 
already cause the extra logging we are suppressing in 
HBaseInterClusterReplicationEndpoint #518. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…

2020-07-23 Thread GitBox



wchevreuil commented on a change in pull request #2118:
URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java
##
@@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer 
sinkPeer) {
*/
   public synchronized void chooseSinks() {
 List slaveAddresses = endpoint.getRegionServers();
-Collections.shuffle(slaveAddresses, ThreadLocalRandom.current());
-int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
-sinks = slaveAddresses.subList(0, numSinks);
+if(slaveAddresses==null){

Review comment:
   > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding 
against custom endpoint implementations?
   
   Yeah, that's indeed the case. 
   
   >We should expose getRegionServers on a base-class or interface and 
explicitly say that we expect a non-null answer. Follow-on..
   If easy, this would be good to write a quick unit test to cover this method.
   
   Sounds good, actually. So we can leave `getRegionServers` at 
HBaseReplicationEndpoint, emphasize it should never return null and add UT.
   
   > So maybe we can safely log warning for if(slaveAddresses.size()==0) ?
   
   Yeah, can just leave that.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…

2020-07-23 Thread GitBox



wchevreuil commented on a change in pull request #2118:
URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083



##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java
##
@@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer 
sinkPeer) {
*/
   public synchronized void chooseSinks() {
 List slaveAddresses = endpoint.getRegionServers();
-Collections.shuffle(slaveAddresses, ThreadLocalRandom.current());
-int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
-sinks = slaveAddresses.subList(0, numSinks);
+if(slaveAddresses==null){

Review comment:
   > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding 
against custom endpoint implementations?
   
   Yeah, that's indeed the case. 
   
   >We should expose getRegionServers on a base-class or interface and 
explicitly say that we expect a non-null answer. Follow-on..
   If easy, this would be good to write a quick unit test to cover this method.
   
   So we can leave `getRegionServers` at HBaseReplicationEndpoint, emphasize it 
should never return null. I'm not sure a UT for this would be effective, 
though. Custom implementations of HBaseReplicationEndpoint returning null would 
still blow it at runtime.
   
   > So maybe we can safely log warning for if(slaveAddresses.size()==0) ?
   
   Yeah, can just leave that.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2125:
URL: https://github.com/apache/hbase/pull/2125#issuecomment-663115381


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 33s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   5m  6s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 16s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   6m 41s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 47s |  hbase-server in master failed.  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 40s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 16s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 16s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m 49s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 52s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 222m 20s |  hbase-server in the patch passed.  
|
   |  |   | 253m 28s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2125 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 4cfb3f3f1b1c 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/testReport/
 |
   | Max. process+thread count | 3201 (vs. ulimit of 12500) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] virajjasani closed pull request #2119: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.2

2020-07-23 Thread GitBox



virajjasani closed pull request #2119:
URL: https://github.com/apache/hbase/pull/2119


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (HBASE-24757) ReplicationSink should limit the batch size for batch mutations based on hbase.rpc.rows.warning.threshold

2020-07-23 Thread Michael Stack (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163762#comment-17163762
 ] 

Michael Stack commented on HBASE-24757:
---

Perhaps this belongs elsewhere but is there any point limiting by row count? 
I'd think size important and row count just an incidental? Does warning when 
too many rows have any benefit? I'd think a warn when a big batch and too few 
rows – i.e. incoming edits are really fat – would be more beneficial? Thanks.

> ReplicationSink should limit the batch size for batch mutations based on 
> hbase.rpc.rows.warning.threshold
> -
>
> Key: HBASE-24757
> URL: https://issues.apache.org/jira/browse/HBASE-24757
> Project: HBase
>  Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>
> At times there are quite a large no of WAL Edits to ship as part of 
> Replication and sometimes replication queues accumulate huge list of Edits to 
> process. ReplicationSink at the sink server usually goes through all Edits 
> and creates map of table -> list of rows grouped by clusterIds, and performs 
> batch mutation of all rows per table level. However, there is no limit to no 
> of Rows that are sent as part of batch mutate call. If no of rows > limit 
> threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn 
> "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is 
> turned on, RS will reject the whole batch without processing.
> We should let Replication Sink honour this threshold value and accordingly 
> keep the size lower per batch mutation call.
> Replication triggered batch mutations should always be consumed but keeping 
> limit of mutation low enough will let the system function at the same pace 
> and without triggering warnings. This will also restrict exploitation of heap 
> and cpu cycles at the destination.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24209) [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality

2020-07-23 Thread Wei-Chiu Chuang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163764#comment-17163764
 ] 

Wei-Chiu Chuang commented on HBASE-24209:
-

I'll resolve this one as a won't fix since Hadoop 3.3.0 is GA now. I'll get a 
new jira to add a 3.3.0 GA precommit check

> [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality
> -
>
> Key: HBASE-24209
> URL: https://issues.apache.org/jira/browse/HBASE-24209
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Nick Dimiduk
>Assignee: Wei-Chiu Chuang
>Priority: Major
>
> Since HBASE-23833 we're paying attention to our builds on Hadoop trunk, 
> currently 3.3.0-SNAPSHOT. Let's add this version to the version lists in 
> hadoopcheck so our CI will let us know when things break, at least 
> compile-time anyway.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [hbase] saintstack commented on a change in pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold

2020-07-23 Thread GitBox



saintstack commented on a change in pull request #2127:
URL: https://github.com/apache/hbase/pull/2127#discussion_r459587451



##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
##
@@ -1583,6 +1583,16 @@
 "hbase.regionserver.slowlog.systable.enabled";
   public static final boolean DEFAULT_SLOW_LOG_SYS_TABLE_ENABLED_KEY = false;
 
+  /**
+   * Number of rows in a batch operation above which a warning will be logged.
+   */
+  public static final String BATCH_ROWS_THRESHOLD_NAME = 
"hbase.rpc.rows.warning.threshold";
+
+  /**
+   * Default value of {@link #BATCH_ROWS_THRESHOLD_NAME}
+   */
+  public static final int BATCH_ROWS_THRESHOLD_DEFAULT = 5000;
+

Review comment:
   Why the move? HConstants is often considered an anti-pattern; better to 
have constants beside where they are used?

##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
##
@@ -93,15 +94,21 @@
   private SourceFSConfigurationProvider provider;
   private WALEntrySinkFilter walEntrySinkFilter;
 
+  /**
+   * Row size threshold for multi requests above which a warning is logged
+   */
+  private final int rowSizeWarnThreshold;
+
   /**
* Create a sink for replication
* @param conf conf object
-   * @param stopper boolean to tell this thread to stop
* @throws IOException thrown when HDFS goes bad or bad file name
*/
-  public ReplicationSink(Configuration conf, Stoppable stopper)
+  public ReplicationSink(Configuration conf)

Review comment:
   Is the Stoppable unused? Usually its a chain for a Service to pull on 
when it meets a condition it can't deal with... one that is so bad it wants to 
stop the process.

##
File path: 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
##
@@ -403,13 +398,24 @@ public void stopReplicationSinkServices() {
* Do the changes and handle the pool
* @param tableName table to insert into
* @param allRows list of actions
+   * @param batchRowSizeThreshold rowSize threshold for batch mutation
*/
-  private void batch(TableName tableName, Collection> allRows) 
throws IOException {
+  private void batch(TableName tableName, Collection> allRows, int 
batchRowSizeThreshold)
+  throws IOException {
 if (allRows.isEmpty()) {
   return;
 }
 AsyncTable table = getConnection().getTable(tableName);
-List> futures = 
allRows.stream().map(table::batchAll).collect(Collectors.toList());
+List> futures = new ArrayList<>();
+for (List rows : allRows) {
+  List> batchRows;
+  if (rows.size() > batchRowSizeThreshold) {
+batchRows = Lists.partition(rows, batchRowSizeThreshold);
+  } else {
+batchRows = Collections.singletonList(rows);
+  }
+  
futures.addAll(batchRows.stream().map(table::batchAll).collect(Collectors.toList()));
+}

Review comment:
   For the edits that are beyond the limit, they get handled in next batch?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2125:
URL: https://github.com/apache/hbase/pull/2125#issuecomment-663120995


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m 33s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m 18s |  master passed  |
   | +1 :green_heart: |  compile  |   1m  1s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   6m 19s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 40s |  master passed  |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   4m  0s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  0s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m  0s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m  7s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  the patch passed  |
   ||| _ Other Tests _ |
   | -1 :x: |  unit  | 237m  7s |  hbase-server in the patch failed.  |
   |  |   | 264m 41s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2125 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux 2814c65b9306 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 1.8.0_232 |
   | unit | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/testReport/
 |
   | Max. process+thread count | 2906 (vs. ulimit of 12500) |
   | modules | C: hbase-server U: hbase-server |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [hbase] Apache-HBase commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side

2020-07-23 Thread GitBox



Apache-HBase commented on pull request #2094:
URL: https://github.com/apache/hbase/pull/2094#issuecomment-663124264


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | +0 :ok: |  reexec  |   1m  6s |  Docker mode activated.  |
   | -0 :warning: |  yetus  |   0m  3s |  Unprocessed flag(s): 
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list 
--whitespace-tabs-ignore-list --quick-hadoopcheck  |
   ||| _ Prechecks _ |
   ||| _ master Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 20s |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |   4m 43s |  master passed  |
   | +1 :green_heart: |  compile  |   1m 59s |  master passed  |
   | +1 :green_heart: |  shadedjars  |   6m 24s |  branch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 27s |  hbase-client in master failed.  |
   | -0 :warning: |  javadoc  |   0m 18s |  hbase-hadoop-compat in master 
failed.  |
   | -0 :warning: |  javadoc  |   0m 39s |  hbase-server in master failed.  |
   ||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 13s |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   4m 27s |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 59s |  the patch passed  |
   | +1 :green_heart: |  javac  |   1m 59s |  the patch passed  |
   | +1 :green_heart: |  shadedjars  |   6m 20s |  patch has no errors when 
building our shaded downstream artifacts.  |
   | -0 :warning: |  javadoc  |   0m 17s |  hbase-hadoop-compat in the patch 
failed.  |
   | -0 :warning: |  javadoc  |   0m 25s |  hbase-client in the patch failed.  |
   | -0 :warning: |  javadoc  |   0m 41s |  hbase-server in the patch failed.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   0m 40s |  hbase-hadoop-compat in the patch 
passed.  |
   | +1 :green_heart: |  unit  |   1m 18s |  hbase-client in the patch passed.  
|
   | +1 :green_heart: |  unit  | 193m  2s |  hbase-server in the patch passed.  
|
   |  |   | 227m 33s |   |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.12 Server=19.03.12 base: 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hbase/pull/2094 |
   | Optional Tests | javac javadoc unit shadedjars compile |
   | uname | Linux f4b2ae5b9dfb 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/hbase-personality.sh |
   | git revision | master / 8191fbdd7d |
   | Default Java | 2020-01-14 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt
 |
   | javadoc | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
 |
   |  Test Results | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/testReport/
 |
   | Max. process+thread count | 3165 (vs. ulimit of 12500) |
   | modules | C: hbase-hadoop-compat hbase-client hbase-server U: . |
   | Console output | 
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/console |
   | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
   | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

1 2 3 >

1 - 100 of 224 matches

Mail list logo