[GitHub] [hbase] anoopsjohn commented on a change in pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll
anoopsjohn commented on a change in pull request #2021: URL: https://github.com/apache/hbase/pull/2021#discussion_r459256398 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java ## @@ -58,31 +58,31 @@ protected static final String WAL_ROLL_PERIOD_KEY = "hbase.regionserver.logroll.period"; - protected final ConcurrentMap walNeedsRoll = new ConcurrentHashMap<>(); + protected final ConcurrentMap wals = new ConcurrentHashMap<>(); Review comment: This will have all wal entries once the wal instance is created. Not just when it needs a roll. So ya these were wals which needed roll at some point. So wals name also fine IMO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
Apache9 commented on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662859269 OK, just saw this on jira > They were in a separate WAL directory. The assumption here is that all the data has been flushed and there is no data needing to be replayed from the WAL. So this is not a normal operation. Typically, you can just disable all the tables, restart the whole cluster on a new environment, and then enable all the tables. The assumption of our implementation is that, a region server should have a WAL directory for it, unless it is dead and has already been processed by SCP. And I prefer we just add a new operation in HBCK2, to scan for unknown servers and schedule SCP for them? Or maybe we already have one in place? @saintstack Can you recall sir? I'm not very familiar with all the operations in HBCK2. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2119: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.2
virajjasani commented on pull request #2119: URL: https://github.com/apache/hbase/pull/2119#issuecomment-662862203 Triggered one more run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2120: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.1
virajjasani commented on pull request #2120: URL: https://github.com/apache/hbase/pull/2120#issuecomment-662862754 I think we don't have active builds for branch-2.1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2122: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2122: URL: https://github.com/apache/hbase/pull/2122#issuecomment-662864609 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 41s | master passed | | +1 :green_heart: | compile | 1m 31s | master passed | | +1 :green_heart: | shadedjars | 6m 24s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | +1 :green_heart: | shadedjars | 6m 23s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 194m 57s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 3s | hbase-shell in the patch passed. | | | | 232m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2122 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9b5e2d90916e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/testReport/ | | Max. process+thread count | 3542 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163290#comment-17163290 ] Anoop Sam John commented on HBASE-23634: bq.One question, if we only partially write an HFile and we don't complete (because crash splitting the WAL say), does it get sidelined, cleaned up? Just wondering. Thanks. You mean when the next attempt of WAL split happens (by another RS). Ya this has to happen. I doubt whether we do this. Good point. > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] taklwu commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
taklwu commented on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662866824 yup, it should be only the case of ALL WALs (user table WAL and MasterRegion WALs) has been flushed but removed, it should have no data loss. You're right that the new set of region servers start on new environment that they are identified as 'Unknown server'. although it's not common on long running cluster, this is very common on a one-time cluster in a dynamic environment but StoreFiles and Tables has been persisted. HBCKSCP and `scheduleRecoveries` were the manual recovery, but our point were comparing to branch-1.4+ that didn't have this issues where `dead` and `unknown` servers were expired and regions can be assigned (and if I found the history correctly, that logic were removed from [processofflineServersWithOnlineRegions -> processOfflineRegions in HBASE-20708](https://github.com/apache/hbase/commit/3e33aecea2da1071d7336584ed29448c233fe3be#diff-322aa95013b832abcc3fa2a396a70748L1310-L1357). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
taklwu edited a comment on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662866824 yup, it should be only the case of ALL WALs (user table WAL and MasterRegion WALs) has been flushed but removed, it should have no data loss. You're right that the new set of region servers start on new environment that they are identified as 'Unknown server'. although it's not common on long running cluster, this is very common on a one-time cluster in a dynamic environment but StoreFiles and Tables has been persisted. HBCKSCP and `scheduleRecoveries` were the manual recovery, but our point were comparing to branch-1.4+ that didn't have this issues where `dead` and `unknown` servers were expired and regions can be assigned (and if I found the history correctly, that logic were removed from [processofflineServersWithOnlineRegions -> processOfflineRegions in HBASE-20708](https://github.com/apache/hbase/commit/3e33aecea2da1071d7336584ed29448c233fe3be#diff-322aa95013b832abcc3fa2a396a70748L1310-L1357). So, I added this flag as a optional feature in the last commit to avoid the confusion to operator that has `WAL` covered, what do you guys think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24759) Persisting configuration of default rsgroup
Sun Xin created HBASE-24759: --- Summary: Persisting configuration of default rsgroup Key: HBASE-24759 URL: https://issues.apache.org/jira/browse/HBASE-24759 Project: HBase Issue Type: New Feature Components: rsgroup Affects Versions: 3.0.0-alpha-1 Reporter: Sun Xin Assignee: Sun Xin Fix For: 3.0.0-alpha-1 In the current scenario, we didn't store the default rsgroup information. But after HBASE-24431 , we have added a config map, which need to be persisted to avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2077: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…
Apache-HBase commented on pull request #2077: URL: https://github.com/apache/hbase/pull/2077#issuecomment-662867634 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 45s | HBASE-24666 passed | | +1 :green_heart: | compile | 2m 39s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 6m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 42s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | compile | 2m 38s | the patch passed | | +1 :green_heart: | javac | 2m 38s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 47s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 13s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 209m 17s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 5m 27s | hbase-thrift in the patch passed. | | | | 250m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2077 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cbd25848ac76 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / fae9f0cd51 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/testReport/ | | Max. process+thread count | 3898 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sun Xin updated HBASE-24759: Issue Type: Bug (was: New Feature) > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] taklwu edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
taklwu edited a comment on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662866824 yup, it should be only the case of ALL WALs (user table WAL and MasterRegion WALs) has been flushed but removed, it should have no data loss. You're right that the new set of region servers start on new environment that they are identified as 'Unknown server'. although it's not common on long running cluster, this is very common on a one-time cluster in a dynamic environment but StoreFiles and Tables has been persisted. HBCKSCP and `scheduleRecoveries` were the manual recovery, but our point were comparing to branch-1.4+ that didn't have this issues where `dead` and `unknown` servers were expired and regions can be assigned (and if I found the history correctly, that logic were removed from [processofflineServersWithOnlineRegions -> processOfflineRegions in HBASE-20708](https://github.com/apache/hbase/commit/3e33aecea2da1071d7336584ed29448c233fe3be#diff-322aa95013b832abcc3fa2a396a70748L1310-L1357). Other than this missing `WAL` situation, somehow current unknown servers are showing up in the case of SCP or other cases. although we were from the `no WAL` scenario, should this change heal the cluster itself? as a second thought, I added this flag `hbase.catalogjanitor.repair.unknown.servers` as a optional feature in the last commit to avoid the confusion to operator that has `WAL` covered, what do you guys think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163315#comment-17163315 ] ramkrishna.s.vasudevan commented on HBASE-23634: We don't do that. If 'hbase.skip.errors' set to true we will atleast make the region open go ahead with out those files. If we don 't set it we will fail the region open. > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2077: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…
Apache-HBase commented on pull request #2077: URL: https://github.com/apache/hbase/pull/2077#issuecomment-662874012 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 10s | HBASE-24666 passed | | +1 :green_heart: | compile | 4m 6s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 7m 25s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 29s | hbase-client in HBASE-24666 failed. | | -0 :warning: | javadoc | 0m 50s | hbase-server in HBASE-24666 failed. | | -0 :warning: | javadoc | 1m 10s | hbase-thrift in HBASE-24666 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 27s | the patch passed | | +1 :green_heart: | compile | 4m 18s | the patch passed | | +1 :green_heart: | javac | 4m 18s | the patch passed | | +1 :green_heart: | shadedjars | 7m 19s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 36s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 57s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 1m 18s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 18s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 42s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 212m 50s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 5m 3s | hbase-thrift in the patch passed. | | | | 265m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2077 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0025876bdfbf 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / fae9f0cd51 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/testReport/ | | Max. process+thread count | 3670 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
Apache9 commented on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662876047 > You're right that the new set of region servers start on new environment that they are identified as 'Unknown server'. I think the old server will be unknown servers, not the new servers? I do not think there is a guarantee that you change the filesystem layout of HBase internal, and HBase cluster will still be functional. Even if sometimes it could, as you said, on 1.4, it does not mean that we will always keep this behavior in new versions. For your scenario, it is OK as you can confirm that there is no data loss as you manually flushed all the data. But what if another user just configs the wrong WAL directory? In this case, if we schedule SCPs automatically, there will be data loss. In general, in HBase, we only know that something strange happen, but we do not know how we come into this situation so it is not safe for us to just recover automatically. Only end user knows, so I still prefer we just provide tools for them to recover, not add some 'unsafe' options in HBase... Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2122: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2122: URL: https://github.com/apache/hbase/pull/2122#issuecomment-662878052 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed | | +1 :green_heart: | shadedjars | 6m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 229m 8s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 15s | hbase-shell in the patch passed. | | | | 265m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2122 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 707aab7f8377 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/testReport/ | | Max. process+thread count | 3358 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2122/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1716#comment-1716 ] Duo Zhang commented on HBASE-24759: --- Ah, this is a pain, hope we do not need to persist the region servers in default rs group? > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163342#comment-17163342 ] Sun Xin commented on HBASE-24759: - {quote}this is a pain, hope we do not need to persist the region servers in default rs group? {quote} If we persist the default rsgroup's information except servers and tables, for now, only persist config of default rs group. Is that OK? > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2124: URL: https://github.com/apache/hbase/pull/2124#issuecomment-662888285 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 40s | branch-2 passed | | +1 :green_heart: | compile | 1m 35s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 32s | the patch passed | | +1 :green_heart: | javac | 1m 32s | the patch passed | | +1 :green_heart: | shadedjars | 6m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 223m 33s | hbase-server in the patch passed. | | -1 :x: | unit | 7m 8s | hbase-shell in the patch failed. | | | | 261m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2124 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 00d68a21d13d 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ce4e692699 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-shell.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/testReport/ | | Max. process+thread count | 2534 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ddupg opened a new pull request #2126: HBASE-24759 Persisting configuration of default rsgroup
ddupg opened a new pull request #2126: URL: https://github.com/apache/hbase/pull/2126 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163346#comment-17163346 ] Pankaj Kumar commented on HBASE-24759: -- Any use case scenario to store additional info in default rsgroup? > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163346#comment-17163346 ] Pankaj Kumar edited comment on HBASE-24759 at 7/23/20, 8:45 AM: Any use case scenario to store additional info in default rsgroup? HBASE-22820 do not persist default group in rsgroup table. was (Author: pankajkumar): Any use case scenario to store additional info in default rsgroup? > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163350#comment-17163350 ] Duo Zhang commented on HBASE-24759: --- [~pankajkumar] HBASE-22820 is the time where the only thing in RSGroup is region servers, so we do not need to persist anything for the default rs group, as all the region servers do not belong to any rs groups will be in default rs group. But in HBASE-24431 we introduced a config map for rs group, so we need to persist at least the config map of the default rs group. > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163356#comment-17163356 ] Pankaj Kumar commented on HBASE-24759: -- Yeah [~zhangduo], I'm inline to HBASE-24431. Persisting additional info for user rsgroup make sense to me but do we allow users to set extra details for default rsgroup? > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163364#comment-17163364 ] Guanghao Zhang commented on HBASE-23634: {quote}We don't do that. If 'hbase.skip.errors' set to true we will atleast make the region open go ahead with out those files. If we don 't set it we will fail the region open. {quote} Yes. But I thought these should be cleanup directly when open region. {quote}Here if we do write the HFiles directly under thd CF, we have a chance that we will get to see a partially written HFile. {quote} What is the affect for other systems(offline compaction tool)? > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22561) modify HFilePrettyPrinter to accept non-hbase.rootdir directories
[ https://issues.apache.org/jira/browse/HBASE-22561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163371#comment-17163371 ] gurmukh singh commented on HBASE-22561: --- Thanks [~wchevreuil] , I am on 2.1.0. Will upgrade in the coming nothing. > modify HFilePrettyPrinter to accept non-hbase.rootdir directories > - > > Key: HBASE-22561 > URL: https://issues.apache.org/jira/browse/HBASE-22561 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.2.0, 2.3.0, 2.1.5 >Reporter: Artem Ervits >Assignee: Artem Ervits >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0, 2.0.6, 2.2.1, 2.1.6 > > Attachments: HBASE-22561.v06.patch, HBASE-22561.v07.patch, > HBASE-22561.v11.patch, HBASE-22561.v12.patch, non_root_dir.png, root_dir.png > > > Today {{hbase hfile}} utility accepts only _hbase.rootDir_ directory. It is > more useful to pass other locations with HFile as well. > {code:java} > hbase org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter hfileoutput > ... > ERROR, file (hfileoutput) is not in HBase's root directory > (hdfs://hadoop.example.com:9000/apps/hbase/) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2126: HBASE-24759 Persisting configuration of default rsgroup
Apache-HBase commented on pull request #2126: URL: https://github.com/apache/hbase/pull/2126#issuecomment-662907673 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | master passed | | +1 :green_heart: | checkstyle | 1m 29s | master passed | | +1 :green_heart: | spotbugs | 2m 40s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | checkstyle | 1m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 35m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2126 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux babc727a0286 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163383#comment-17163383 ] Sun Xin commented on HBASE-24759: - In HBASE-24431 , an idea was mentioned {quote}Use special RSGroup config to overright the cluster config. Then we can config different RSGroup to use different config in future. {quote} [~pankajkumar] If this idea is implemented, could region servers in default rs group use different config based on this issue? > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ramkrish86 commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
ramkrish86 commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-662916393 Seems the null check was causing some issues as the finally was not getting executed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163396#comment-17163396 ] Pankaj Kumar commented on HBASE-24759: -- Ok, got it.. Recently we met a blocking issue (in HBase-2.2.3) where all system tables were assigned to default rsgroup and due to some reason all RSs (only 1 RS is in default rsgroup) of default rsgroup went down. Hmaster stuck while persisting default rsgroup info to rsgroup table (since no RS in deafult rsgroup, so meta and other system region were not assigned). Even new RS join the default rsgroup, AM couldn't pick them as candidate since rsGroupMap will be updated after persisting default rsgroup to rsgroup table. HBASE-22820 addressed the problem. > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163398#comment-17163398 ] Francis Christopher Liu commented on HBASE-11288: - [~stack] and [~zhangduo] thank you for your suggestions of avoiding master branch and just rebasing on branch-2. I will go ahead and do that then. I will try to get as much information of course. I’ll do my best to get ITBLL to pass. [~zhangduo] Good to hear you are willing to change your mind with a reasonable result. As my intention has always been not only to prove/disprove it to myself but to you and [~stack] as well to help come to a consensus on a path forward. Looking at [~stack]’s recent comment the consensus seems to be that there was no consensus on a path forward. Although as [~stack] pointed out [~apurtell] did suggest we just do a reset so I missed the signs that we were doing that so my mistake. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2095: HBASE-24459 Move the locateMeta logic from AsyncMetaRegionTableLocato…
Apache-HBase commented on pull request #2095: URL: https://github.com/apache/hbase/pull/2095#issuecomment-662926329 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-11288.splittable-meta Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 50s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | checkstyle | 2m 4s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | spotbugs | 3m 45s | HBASE-11288.splittable-meta passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | checkstyle | 0m 24s | The patch passed checkstyle in hbase-common | | +1 :green_heart: | checkstyle | 0m 27s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 14s | hbase-server: The patch generated 0 new + 148 unchanged - 1 fixed = 148 total (was 149) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 20s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 42m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2095 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux c42110f3288d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-11288.splittable-meta / 1e358188ac | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
Apache-HBase commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-662927465 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +1 :green_heart: | checkstyle | 1m 13s | master passed | | +1 :green_heart: | spotbugs | 2m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 20s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 35m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2125 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 36fbf11808fd 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163415#comment-17163415 ] Duo Zhang commented on HBASE-11288: --- {quote} I’ll do my best to get ITBLL to pass. {quote} You do not need to get ITBLL to pass. I do not think the work can be done within a reasonable time. [~ndimiduk] has spent several month to archive this. Instead, set a deadline, 1 week or 2 weeks, then we come back to analyze the result, on how to explain the failures, and also the complexity of the code to archive the current state. Of course it will be good if you can pass the ITBLL. And an interesting thing is that, I offer that, I will change my mind if you can get a reasonable result, then what can make you change your mind? We both have a solution, I do not think it is fair to me that, only I offer how to change my mind, but you do not offer anything to change your mind. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24696) Include JVM information on Web UI under "Software Attributes"
[ https://issues.apache.org/jira/browse/HBASE-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163417#comment-17163417 ] Hudson commented on HBASE-24696: Results for branch branch-1 [build #1332 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332//General_Nightly_Build_Report/] (x) {color:red}-1 jdk7 checks{color} -- For more information [see jdk7 report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332//JDK7_Nightly_Build_Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1332//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 source release artifact{color} -- See build output for details. > Include JVM information on Web UI under "Software Attributes" > - > > Key: HBASE-24696 > URL: https://issues.apache.org/jira/browse/HBASE-24696 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Nick Dimiduk >Assignee: Mingliang Liu >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0 > > Attachments: Screen Shot 2020-07-17 at 10.55.56 PM.png > > > It's a small thing, but seems like an omission. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163420#comment-17163420 ] Anoop Sam John commented on HBASE-23634: The incomplete HFiles will be under the cf/recovered.edits dir then. When we try load those files to region/cf, we will have the validation and it will fail for such incomplete files. What will happen for such fail? This says we can not really avoid that validation step while loading the recovered.edits HFiles. bq.What is the affect for other systems(offline compaction tool)? It will see such files are corrupt. The files might not have a proper trailer. So every such external systems to handle such cases. If such files, just ignore. Then one issue is how the system can know whether it is a partially written file or a real corruption? > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24759) Persisting configuration of default rsgroup
[ https://issues.apache.org/jira/browse/HBASE-24759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163426#comment-17163426 ] Sun Xin commented on HBASE-24759: - I think the core problem you [~pankajkumar] had is that the system tables went unavailable. In HBASE-22738 we allow tables fallback to specific rs groups, but for system tables, maybe we should allow them fallback to any rsgroup to keep always available. Let me do this on a new jira. > Persisting configuration of default rsgroup > --- > > Key: HBASE-24759 > URL: https://issues.apache.org/jira/browse/HBASE-24759 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In the current scenario, we didn't store the default rsgroup information. But > after HBASE-24431 , we have added a config map, which need to be persisted to > avoid lossing config of default rsgroup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24752) NPE/500 accessing webui on master startup
[ https://issues.apache.org/jira/browse/HBASE-24752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenfeiyi666 reassigned HBASE-24752: --- Assignee: wenfeiyi666 > NPE/500 accessing webui on master startup > - > > Key: HBASE-24752 > URL: https://issues.apache.org/jira/browse/HBASE-24752 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Assignee: wenfeiyi666 >Priority: Minor > > I got a 500 error while accessing the master web ui (non-backup master) > during startup. > {noformat} > 2020-07-21 18:56:39,762 WARN org.eclipse.jetty.servlet.ServletHandler: > /master-status > java.lang.NullPointerException > at > org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:339) > at > org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:397) > at > org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:388) > at > org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:79) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > ... > {noformat} > {{MasterStatusTmplImpl.java:339}} appears to align with > {{MasterStatusTmpl.jamon:227}}: > {noformat} > <%if > master.getMasterCoprocessorHost().findCoprocessor("RSGroupAdminEndpoint") != > null && > serverManager.getOnlineServersList().size() > 0 %> > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24760) Allow system tables fallback to any rs groups
Sun Xin created HBASE-24760: --- Summary: Allow system tables fallback to any rs groups Key: HBASE-24760 URL: https://issues.apache.org/jira/browse/HBASE-24760 Project: HBase Issue Type: New Feature Components: rsgroup Affects Versions: 3.0.0-alpha-1 Reporter: Sun Xin Assignee: Sun Xin Fix For: 3.0.0-alpha-1 In HBASE-22738 we allow tables fallback to specific rs groups, If there is no online servers in the table's rsgroup. But for system tables, It is necessary to allow system tables fallback to any rsgroup in order to keey available at all times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ramkrish86 commented on a change in pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll
ramkrish86 commented on a change in pull request #2021: URL: https://github.com/apache/hbase/pull/2021#discussion_r459374807 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java ## @@ -58,31 +58,31 @@ protected static final String WAL_ROLL_PERIOD_KEY = "hbase.regionserver.logroll.period"; - protected final ConcurrentMap walNeedsRoll = new ConcurrentHashMap<>(); + protected final ConcurrentMap wals = new ConcurrentHashMap<>(); Review comment: Ok This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24760) Allow system tables fallback to any rs groups
[ https://issues.apache.org/jira/browse/HBASE-24760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sun Xin updated HBASE-24760: Description: In HBASE-22738 we allow tables fallback to specific rs groups, If there is no online servers in the table's rsgroup. But for system tables, if there is no specified fallback rsgroup or the servers in the fallback rsgroup all went down, It is necessary to allow system tables fallback to any rsgroup in order to keey available at all times. was: In HBASE-22738 we allow tables fallback to specific rs groups, If there is no online servers in the table's rsgroup. But for system tables, It is necessary to allow system tables fallback to any rsgroup in order to keey available at all times. > Allow system tables fallback to any rs groups > - > > Key: HBASE-24760 > URL: https://issues.apache.org/jira/browse/HBASE-24760 > Project: HBase > Issue Type: New Feature > Components: rsgroup >Affects Versions: 3.0.0-alpha-1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In HBASE-22738 we allow tables fallback to specific rs groups, If there is no > online servers in the table's rsgroup. > But for system tables, if there is no specified fallback rsgroup or the > servers in the fallback rsgroup all went down, It is necessary to allow > system tables fallback to any rsgroup in order to keey available at all times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163438#comment-17163438 ] Anoop Sam John commented on HBASE-23634: bq.Then one issue is how the system can know whether it is a partially written file or a real corruption? This is applicable when we do split wals to HFiles and during that there was some wal file split failed in between and reattempted. End of the day there might be some HFiles which are duplicate. Some HFiles may be incomplete. Now while reading back the file (verification while loading to cf) we dont know whether this is a failed attempt's partial HFile or a real corrupted file once it was written to FS.So on wal file split failure, a cleanup before next attempt is imp. Now one problem is there might be N WAL files and all splits will create the HFiles for a region:cf under same dir region/cf/recovered.edits. Now if we wanted this cleanup, these files should have been generated with some way to identify them as result of which wal file's split. Say if HFile was placed under region/cf/recovered.edits/ dir, we could have cleaned it up before doing the next attempt. Thoughts? > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2126: HBASE-24759 Persisting configuration of default rsgroup
Apache-HBase commented on pull request #2126: URL: https://github.com/apache/hbase/pull/2126#issuecomment-662955563 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 1m 28s | master passed | | +1 :green_heart: | shadedjars | 5m 49s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 21s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 29s | the patch passed | | +1 :green_heart: | javac | 1m 29s | the patch passed | | +1 :green_heart: | shadedjars | 5m 45s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 131m 50s | hbase-server in the patch passed. | | | | 161m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2126 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a28da9d3a1e3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/testReport/ | | Max. process+thread count | 4299 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll
ramkrish86 commented on a change in pull request #2021: URL: https://github.com/apache/hbase/pull/2021#discussion_r459384595 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java ## @@ -148,53 +147,49 @@ private void abort(String reason, Throwable cause) { @Override public void run() { while (running) { - boolean periodic = false; long now = System.currentTimeMillis(); checkLowReplication(now); - periodic = (now - this.lastRollTime) > this.rollPeriod; - if (periodic) { -// Time for periodic roll, fall through -LOG.debug("WAL roll period {} ms elapsed", this.rollPeriod); - } else { -synchronized (this) { - if (walNeedsRoll.values().stream().anyMatch(Boolean::booleanValue)) { -// WAL roll requested, fall through -LOG.debug("WAL roll requested"); - } else { -try { - wait(this.threadWakeFrequency); -} catch (InterruptedException e) { - // restore the interrupt state - Thread.currentThread().interrupt(); -} -// goto the beginning to check whether again whether we should fall through to roll -// several WALs, and also check whether we should quit. -continue; + synchronized (this) { +if (wals.values().stream().noneMatch(rc -> rc.needsRoll(now))) { + try { +wait(this.threadWakeFrequency); + } catch (InterruptedException e) { +// restore the interrupt state +Thread.currentThread().interrupt(); } + // goto the beginning to check whether again whether we should fall through to roll + // several WALs, and also check whether we should quit. + continue; } } try { -this.lastRollTime = System.currentTimeMillis(); -for (Iterator> iter = walNeedsRoll.entrySet().iterator(); iter - .hasNext();) { - Entry entry = iter.next(); +for (Iterator> iter = wals.entrySet().iterator(); + iter.hasNext();) { + Entry entry = iter.next(); WAL wal = entry.getKey(); - // reset the flag in front to avoid missing roll request before we return from rollWriter. - walNeedsRoll.put(wal, Boolean.FALSE); - Map> regionsToFlush = null; + RollController controller = entry.getValue(); + if (controller.isRollRequested()) { +// WAL roll requested, fall through +LOG.debug("WAL {} roll requested", wal); + } else if (controller.needsPeriodicRoll(now)){ +// Time for periodic roll, fall through +LOG.debug("WAL {} roll period {} ms elapsed", wal, this.rollPeriod); + } else { +continue; Review comment: This is they key. We ensure we only roll if one of the condition is met (if either size reach caused a log roll ) or the time elapsed. That is also tracked per wal. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
anoopsjohn commented on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329 >Typically, you can just disable all the tables, restart the whole cluster on a new environment, and then enable all the tables. Agree to Duo. That might be a better path for such use case where the cluster is deleted (after persisting all data) and later create pointing to existing FS. >In general, in HBase, we only know that something strange happen, but we do not know how we come into this situation so it is not safe for us to just recover automatically Ya I am also agreeing to this point. In 2.x we store the table's disable/enable state in META. So we will have that info available in the newly recreated cluster also. (The zk data from old cluster also wont be there in new cluster I believe) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
anoopsjohn edited a comment on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329 >Typically, you can just disable all the tables, restart the whole cluster on a new environment, and then enable all the tables.< Agree to Duo. That might be a better path for such use case where the cluster is deleted (after persisting all data) and later create pointing to existing FS. >In general, in HBase, we only know that something strange happen, but we do not know how we come into this situation so it is not safe for us to just recover automatically Ya I am also agreeing to this point. In 2.x we store the table's disable/enable state in META. So we will have that info available in the newly recreated cluster also. (The zk data from old cluster also wont be there in new cluster I believe) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
anoopsjohn edited a comment on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329 >Typically, you can just disable all the tables, restart the whole cluster on a new environment, and then enable all the tables. Agree to Duo. That might be a better path for such use case where the cluster is deleted (after persisting all data) and later create pointing to existing FS. >In general, in HBase, we only know that something strange happen, but we do not know how we come into this situation so it is not safe for us to just recover automatically Ya I am also agreeing to this point. In 2.x we store the table's disable/enable state in META. So we will have that info available in the newly recreated cluster also. (The zk data from old cluster also wont be there in new cluster I believe) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn edited a comment on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
anoopsjohn edited a comment on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-662958329 >Typically, you can just disable all the tables, restart the whole cluster on a new environment, and then enable all the tables. Agree to Duo. That might be a better path for such use case where the cluster is deleted (after persisting all data) and later create pointing to existing FS. >In general, in HBase, we only know that something strange happen, but we do not know how we come into this situation so it is not safe for us to just recover automatically Ya I am also agreeing to this point. In 2.x we store the table's disable/enable state in META. So we will have that info available in the newly recreated cluster also. (The zk data from old cluster also wont be there in new cluster I believe) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24665) all wal of RegionGroupingProvider together roll
[ https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163464#comment-17163464 ] Anoop Sam John commented on HBASE-24665: Will wait till tomorrow if we have any more comments. If none, will commit the PR. > all wal of RegionGroupingProvider together roll > --- > > Key: HBASE-24665 > URL: https://issues.apache.org/jira/browse/HBASE-24665 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.0, master, 2.1.10, 1.4.14, 2.2.6 >Reporter: wenfeiyi666 >Assignee: wenfeiyi666 >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.1.10, 1.4.14, 2.2.7 > > > when use multiwal, any a wal request roll, all wal will be together roll. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24665) MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[ https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-24665: --- Summary: MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll (was: all wal of RegionGroupingProvider together roll) > MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll > -- > > Key: HBASE-24665 > URL: https://issues.apache.org/jira/browse/HBASE-24665 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.0, master, 2.1.10, 1.4.14, 2.2.6 >Reporter: wenfeiyi666 >Assignee: wenfeiyi666 >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.1.10, 1.4.14, 2.2.7 > > > when use multiwal, any a wal request roll, all wal will be together roll. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2126: HBASE-24759 Persisting configuration of default rsgroup
Apache-HBase commented on pull request #2126: URL: https://github.com/apache/hbase/pull/2126#issuecomment-662962818 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 34s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed | | +1 :green_heart: | shadedjars | 5m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 22s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 153m 18s | hbase-server in the patch passed. | | | | 180m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2126 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 08eb90886ca5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/testReport/ | | Max. process+thread count | 3997 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2126/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2119: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.2
Apache-HBase commented on pull request #2119: URL: https://github.com/apache/hbase/pull/2119#issuecomment-662966432 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 33s | branch-2.2 passed | | +1 :green_heart: | compile | 2m 3s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 2m 25s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 26s | branch-2.2 passed | | +0 :ok: | spotbugs | 1m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 6m 3s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 8s | the patch passed | | +1 :green_heart: | compile | 2m 4s | the patch passed | | +1 :green_heart: | javac | 2m 4s | the patch passed | | +1 :green_heart: | checkstyle | 2m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 25s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 27m 22s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 1m 27s | the patch passed | | +1 :green_heart: | findbugs | 5m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 173m 50s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 4m 4s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 6m 10s | hbase-rest in the patch passed. | | +1 :green_heart: | asflicense | 1m 16s | The patch does not generate ASF License warnings. | | | | 264m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2119/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2119 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8d7614015d03 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-2119/out/precommit/personality/provided.sh | | git revision | branch-2.2 / 2c08876d70 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2119/2/testReport/ | | Max. process+thread count | 5344 (vs. ulimit of 1) | | modules | C: hbase-server hbase-thrift hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2119/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2124: URL: https://github.com/apache/hbase/pull/2124#issuecomment-662967670 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 27s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 6s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 32s | the patch passed | | +1 :green_heart: | checkstyle | 1m 27s | the patch passed | | -0 :warning: | rubocop | 0m 7s | The patch generated 62 new + 294 unchanged - 17 fixed = 356 total (was 311) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 36s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | | | | 37m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2124 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 3c2c145f9b15 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ce4e692699 | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2095: HBASE-24459 Move the locateMeta logic from AsyncMetaRegionTableLocato…
Apache-HBase commented on pull request #2095: URL: https://github.com/apache/hbase/pull/2095#issuecomment-662975082 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-11288.splittable-meta Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 37s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | compile | 1m 42s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | shadedjars | 5m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | HBASE-11288.splittable-meta passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 1m 44s | the patch passed | | +1 :green_heart: | javac | 1m 44s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 25s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 3s | hbase-client in the patch passed. | | -1 :x: | unit | 139m 17s | hbase-server in the patch failed. | | | | 170m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2095 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6b1036ef1a82 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-11288.splittable-meta / 1e358188ac | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/testReport/ | | Max. process+thread count | 4066 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
Apache-HBase commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-662978376 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | master passed | | +1 :green_heart: | compile | 0m 52s | master passed | | +1 :green_heart: | shadedjars | 5m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 5m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 145m 6s | hbase-server in the patch failed. | | | | 168m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b302ab71068d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/testReport/ | | Max. process+thread count | 4245 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2095: HBASE-24459 Move the locateMeta logic from AsyncMetaRegionTableLocato…
Apache-HBase commented on pull request #2095: URL: https://github.com/apache/hbase/pull/2095#issuecomment-662979595 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 52s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-11288.splittable-meta Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 29s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | compile | 2m 19s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | shadedjars | 7m 44s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 32s | hbase-client in HBASE-11288.splittable-meta failed. | | -0 :warning: | javadoc | 0m 20s | hbase-common in HBASE-11288.splittable-meta failed. | | -0 :warning: | javadoc | 0m 48s | hbase-server in HBASE-11288.splittable-meta failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 23s | the patch passed | | +1 :green_heart: | compile | 2m 27s | the patch passed | | +1 :green_heart: | javac | 2m 27s | the patch passed | | +1 :green_heart: | shadedjars | 7m 34s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 29s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 49s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 49s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 20s | hbase-client in the patch passed. | | -1 :x: | unit | 139m 20s | hbase-server in the patch failed. | | | | 181m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2095 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1a2c5b15603e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-11288.splittable-meta / 1e358188ac | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/testReport/ | | Max. process+thread count | 4406 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2095/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
Apache-HBase commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-662980878 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 52s | master passed | | +1 :green_heart: | compile | 1m 17s | master passed | | +1 :green_heart: | shadedjars | 6m 57s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 50s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 52s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 7m 9s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 49s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 143m 15s | hbase-server in the patch failed. | | | | 174m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6fe797610be1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/testReport/ | | Max. process+thread count | 4868 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[ https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163542#comment-17163542 ] Anoop Sam John commented on HBASE-24749: bq.On events such as flush and compaction, we write markers to the WAL w/ notes listing files that participated in the event. On recovery, we read these events completing compactions if all participants present and it looked like we crashed after compaction completed but before we got to slot the new files into place and remove the old. This is not only for META (or ROOT) table you suggesting stack? (But generically) Ya during flush and compaction we have start markers with files involved in that op and on completion another marker. During region recovery, we can use these markers to identify uncommitted files right? Need to see whether there is chance that we wall gets rolled and miss the start markers. > Direct insert HFiles and Persist in-memory HFile tracking > - > > Key: HBASE-24749 > URL: https://issues.apache.org/jira/browse/HBASE-24749 > Project: HBase > Issue Type: Umbrella > Components: Compaction, HFile >Affects Versions: 3.0.0-alpha-1 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: design, discussion, objectstore, storeFile, storeengine > Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct > insert HFiles and Persist in-memory HFile tracking.pdf > > > We propose a new feature (a new store engine) to remove the {{.tmp}} > directory used in the commit stage for common HFile operations such as flush > and compaction to improve the write throughput and latency on object stores. > Specifically for S3 filesystems, this will also mitigate read-after-write > inconsistencies caused by immediate HFiles validation after moving the > HFile(s) to data directory. > Please see attached for this proposal and the initial result captured with > 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, > and workload C RUN result. > The goal of this JIRA is to discuss with the community if the proposed > improvement on the object stores use case makes senses and if we miss > anything should be included. > Improvement Highlights > 1. Lower write latency, especially the p99+ > 2. Higher write throughput on flush and compaction > 3. Lower MTTR on region (re)open or assignment > 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file > system implementation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[ https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163552#comment-17163552 ] ramkrishna.s.vasudevan commented on HBASE-24749: I think in flushes this is possible to recover from a start marker (assuming we will abort on a flush failure). But for compactions I think the markers in the WAL may get rolled over, right? > Direct insert HFiles and Persist in-memory HFile tracking > - > > Key: HBASE-24749 > URL: https://issues.apache.org/jira/browse/HBASE-24749 > Project: HBase > Issue Type: Umbrella > Components: Compaction, HFile >Affects Versions: 3.0.0-alpha-1 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: design, discussion, objectstore, storeFile, storeengine > Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct > insert HFiles and Persist in-memory HFile tracking.pdf > > > We propose a new feature (a new store engine) to remove the {{.tmp}} > directory used in the commit stage for common HFile operations such as flush > and compaction to improve the write throughput and latency on object stores. > Specifically for S3 filesystems, this will also mitigate read-after-write > inconsistencies caused by immediate HFiles validation after moving the > HFile(s) to data directory. > Please see attached for this proposal and the initial result captured with > 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, > and workload C RUN result. > The goal of this JIRA is to discuss with the community if the proposed > improvement on the object stores use case makes senses and if we miss > anything should be included. > Improvement Highlights > 1. Lower write latency, especially the p99+ > 2. Higher write throughput on flush and compaction > 3. Lower MTTR on region (re)open or assignment > 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file > system implementation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24632) Enable procedure-based log splitting as default in hbase3
[ https://issues.apache.org/jira/browse/HBASE-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163551#comment-17163551 ] Anoop Sam John commented on HBASE-24632: So for every WAL file, it will add a sub procedure in ProcedureExecutor and that in turn add entries for RemoteProcExecutor correct? So at this stage also these Sub proc executing threads will yield once the split proc is submitted to RemoteProcExecutor ? And there in that executor we have 128 threads by default. Will that wait once it issue split request to RS for split? Or will that also yield? If former, then we will get big backlog in a bigger cluster with many RS down and every having many WALs to split > Enable procedure-based log splitting as default in hbase3 > - > > Key: HBASE-24632 > URL: https://issues.apache.org/jira/browse/HBASE-24632 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Means changing this value in HConstants to false: >public static final boolean DEFAULT_HBASE_SPLIT_COORDINATED_BY_ZK = true; > Should probably also deprecate the current zk distributed split too so we can > clear out those classes to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
Apache-HBase commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-662996848 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | checkstyle | 1m 6s | master passed | | +1 :green_heart: | spotbugs | 1m 59s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | checkstyle | 1m 3s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 15s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 32m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2125 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 9b05678ad07a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[ https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163558#comment-17163558 ] Duo Zhang commented on HBASE-24749: --- IIRC there is a issue for storing the compacted files in HFile's metadata, to solve the problem that the wal file contains the compaction marker may be deleted before wal splitting. > Direct insert HFiles and Persist in-memory HFile tracking > - > > Key: HBASE-24749 > URL: https://issues.apache.org/jira/browse/HBASE-24749 > Project: HBase > Issue Type: Umbrella > Components: Compaction, HFile >Affects Versions: 3.0.0-alpha-1 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: design, discussion, objectstore, storeFile, storeengine > Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct > insert HFiles and Persist in-memory HFile tracking.pdf > > > We propose a new feature (a new store engine) to remove the {{.tmp}} > directory used in the commit stage for common HFile operations such as flush > and compaction to improve the write throughput and latency on object stores. > Specifically for S3 filesystems, this will also mitigate read-after-write > inconsistencies caused by immediate HFiles validation after moving the > HFile(s) to data directory. > Please see attached for this proposal and the initial result captured with > 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, > and workload C RUN result. > The goal of this JIRA is to discuss with the community if the proposed > improvement on the object stores use case makes senses and if we miss > anything should be included. > Improvement Highlights > 1. Lower write latency, especially the p99+ > 2. Higher write throughput on flush and compaction > 3. Lower MTTR on region (re)open or assignment > 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file > system implementation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23285) walArchiveDir does not exist when separate the old WALs into different regionserver directories
[ https://issues.apache.org/jira/browse/HBASE-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xuqinya updated HBASE-23285: Summary: walArchiveDir does not exist when separate the old WALs into different regionserver directories (was: Sometimes walArchiveDir does not exist when separate the old WALs into different regionserver directories) > walArchiveDir does not exist when separate the old WALs into different > regionserver directories > --- > > Key: HBASE-23285 > URL: https://issues.apache.org/jira/browse/HBASE-23285 > Project: HBase > Issue Type: Bug >Affects Versions: master >Reporter: xuqinya >Assignee: xuqinya >Priority: Minor > > [HBASE-14247|https://issues.apache.org/jira/browse/HBASE-14247] Add a new > config hbase.separate.oldlogdir.by.regionserver, the old wal dir will be > separated by regionservers. > But,if there are no files, the regionserver directory will be cleared. > CommonFSUtils.renameAndSetModifyTime(this.fs, p, newPath) will throw > exception. > {code:java} > java.io.IOException: Unable to rename > hdfs://127.0.0.1:59066/user/xuqinya/test-data/0836eb3c-cf89-ffdd-5ca1-54049cb50c15/WALs/169.254.31.146,59229,1573636627328/169.254.31.146%2C59229%2C1573636627328.1573636642613 > to > hdfs://127.0.0.1:59066/user/xuqinya/test-data/0836eb3c-cf89-ffdd-5ca1-54049cb50c15/oldWALs/169.254.31.146,59229,1573636627328/169.254.31.146%2C59229%2C1573636627328.1573636642613 > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:735) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.cleanOldLogs(AbstractFSWAL.java:706) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:849) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:540) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani opened a new pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold
virajjasani opened a new pull request #2127: URL: https://github.com/apache/hbase/pull/2127 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-24757) ReplicationSink should limit the batch size for batch mutations based on hbase.rpc.rows.warning.threshold
[ https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24757 started by Viraj Jasani. > ReplicationSink should limit the batch size for batch mutations based on > hbase.rpc.rows.warning.threshold > - > > Key: HBASE-24757 > URL: https://issues.apache.org/jira/browse/HBASE-24757 > Project: HBase > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > At times there are quite a large no of WAL Edits to ship as part of > Replication and sometimes replication queues accumulate huge list of Edits to > process. ReplicationSink at the sink server usually goes through all Edits > and creates map of table -> list of rows grouped by clusterIds, and performs > batch mutation of all rows per table level. However, there is no limit to no > of Rows that are sent as part of batch mutate call. If no of rows > limit > threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn > "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is > turned on, RS will reject the whole batch without processing. > We should let Replication Sink honour this threshold value and accordingly > keep the size lower per batch mutation call. > Replication triggered batch mutations should always be consumed but keeping > limit of mutation low enough will let the system function at the same pace > and without triggering warnings. This will also restrict exploitation of heap > and cpu cycles at the destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[ https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163565#comment-17163565 ] Anoop Sam John commented on HBASE-24749: Ya that will help for deleting the compacted away files while opening the result file. But the other case where we write compaction result file directly under region/cf and we had the start compaction marker in wal. The compaction did not finish and RS crashed. By that time the wal file which had the start compaction marker got rolled and deleted also. Now while replay of wal, we dont have start compaction marker for this wal file. So we think this is an old valid file but that is wrong. This is a partial file. This is possible. On that approach of solving the issue with WAL marker, this is a possible issue (?) > Direct insert HFiles and Persist in-memory HFile tracking > - > > Key: HBASE-24749 > URL: https://issues.apache.org/jira/browse/HBASE-24749 > Project: HBase > Issue Type: Umbrella > Components: Compaction, HFile >Affects Versions: 3.0.0-alpha-1 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: design, discussion, objectstore, storeFile, storeengine > Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct > insert HFiles and Persist in-memory HFile tracking.pdf > > > We propose a new feature (a new store engine) to remove the {{.tmp}} > directory used in the commit stage for common HFile operations such as flush > and compaction to improve the write throughput and latency on object stores. > Specifically for S3 filesystems, this will also mitigate read-after-write > inconsistencies caused by immediate HFiles validation after moving the > HFile(s) to data directory. > Please see attached for this proposal and the initial result captured with > 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, > and workload C RUN result. > The goal of this JIRA is to discuss with the community if the proposed > improvement on the object stores use case makes senses and if we miss > anything should be included. > Improvement Highlights > 1. Lower write latency, especially the p99+ > 2. Higher write throughput on flush and compaction > 3. Lower MTTR on region (re)open or assignment > 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file > system implementation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24761) Support hadoop 3.3.0
Duo Zhang created HBASE-24761: - Summary: Support hadoop 3.3.0 Key: HBASE-24761 URL: https://issues.apache.org/jira/browse/HBASE-24761 Project: HBase Issue Type: Umbrella Components: hadoop3 Reporter: Duo Zhang A good thing is that, finally hadoop upgraded protobuf and also shaded the protobuf it uses. Let's try to support it, and also we can purge our own dependencies on protobuf 2.5.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] brfrn169 commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side
brfrn169 commented on pull request #2094: URL: https://github.com/apache/hbase/pull/2094#issuecomment-663010093 @joshelser Thank you very much for reviewing this! I will wait for any other opinions/objections until tomorrow. If nothing, will commit this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24762) Purge protobuf java 2.5.0 dependency
Duo Zhang created HBASE-24762: - Summary: Purge protobuf java 2.5.0 dependency Key: HBASE-24762 URL: https://issues.apache.org/jira/browse/HBASE-24762 Project: HBase Issue Type: Sub-task Components: dependencies, Protobufs Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0-alpha-1 On master branch, we have removed the hbase-protocol module so in general, we do not need to depend on protobuf 2.5.0 directl. Especially for hadoop 3.3.0, hadoop will not depend on 2.5.0 any more, we should make sure hbase do not introduce protobuf 2.5.0 too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24762) Purge protobuf java 2.5.0 dependency
[ https://issues.apache.org/jira/browse/HBASE-24762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24762: -- Parent Issue: HBASE-22953 (was: HBASE-24761) > Purge protobuf java 2.5.0 dependency > > > Key: HBASE-24762 > URL: https://issues.apache.org/jira/browse/HBASE-24762 > Project: HBase > Issue Type: Sub-task > Components: dependencies, Protobufs >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > On master branch, we have removed the hbase-protocol module so in general, we > do not need to depend on protobuf 2.5.0 directl. Especially for hadoop 3.3.0, > hadoop will not depend on 2.5.0 any more, we should make sure hbase do not > introduce protobuf 2.5.0 too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24761) Support hadoop 3.3.0
[ https://issues.apache.org/jira/browse/HBASE-24761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24761. --- Resolution: Duplicate > Support hadoop 3.3.0 > > > Key: HBASE-24761 > URL: https://issues.apache.org/jira/browse/HBASE-24761 > Project: HBase > Issue Type: Umbrella > Components: hadoop3 >Reporter: Duo Zhang >Priority: Major > > A good thing is that, finally hadoop upgraded protobuf and also shaded the > protobuf it uses. > Let's try to support it, and also we can purge our own dependencies on > protobuf 2.5.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold
Apache-HBase commented on pull request #2127: URL: https://github.com/apache/hbase/pull/2127#issuecomment-663023627 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 42s | master passed | | +1 :green_heart: | checkstyle | 1m 34s | master passed | | +1 :green_heart: | spotbugs | 2m 42s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | -0 :warning: | checkstyle | 1m 4s | hbase-server: The patch generated 1 new + 80 unchanged - 0 fixed = 81 total (was 80) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 38s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 39m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2127 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 698adefbb8b0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side
Apache9 commented on a change in pull request #2094: URL: https://github.com/apache/hbase/pull/2094#discussion_r459471919 ## File path: hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java ## @@ -86,6 +86,12 @@ */ void updateCheckAndPut(long t); + /** + * Update checkAndMutate histogram + * @param t time it took + */ + void updateCheckAndMutate(long t); Review comment: After this change, do we still have checkAndPut and checkAndDelete? ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java ## @@ -4272,43 +4275,96 @@ protected Durability getEffectiveDurability(Durability d) { } @Override + @Deprecated public boolean checkAndMutate(byte[] row, byte[] family, byte[] qualifier, CompareOperator op, ByteArrayComparable comparator, TimeRange timeRange, Mutation mutation) throws IOException { -return doCheckAndRowMutate(row, family, qualifier, op, comparator, null, timeRange, null, - mutation); +CheckAndMutate.Builder builder = CheckAndMutate.newBuilder(row) + .ifMatches(family, qualifier, op, comparator.getValue()).timeRange(timeRange); +try { + if (mutation instanceof Put) { +return checkAndMutate(builder.build((Put) mutation)).isSuccess(); + } else if (mutation instanceof Delete) { +return checkAndMutate(builder.build((Delete) mutation)).isSuccess(); + } else { +throw new DoNotRetryIOException( + "Unsupported mutate type: " + mutation.getClass().getSimpleName().toUpperCase()); + } +} catch (IllegalArgumentException e) { + throw new DoNotRetryIOException(e.getMessage()); +} } @Override + @Deprecated public boolean checkAndMutate(byte[] row, Filter filter, TimeRange timeRange, Mutation mutation) throws IOException { -return doCheckAndRowMutate(row, null, null, null, null, filter, timeRange, null, mutation); +CheckAndMutate.Builder builder = CheckAndMutate.newBuilder(row).ifMatches(filter) + .timeRange(timeRange); +try { + if (mutation instanceof Put) { +return checkAndMutate(builder.build((Put) mutation)).isSuccess(); + } else if (mutation instanceof Delete) { +return checkAndMutate(builder.build((Delete) mutation)).isSuccess(); + } else { +throw new DoNotRetryIOException("Unsupported mutate type: " + + mutation.getClass().getSimpleName().toUpperCase()); + } +} catch (IllegalArgumentException e) { Review comment: Ditto. ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java ## @@ -759,12 +810,150 @@ default boolean postCheckAndDelete(ObserverContext * @param delete delete to commit if check succeeds * @param result from the CheckAndDelete * @return the possibly transformed returned value to return to client + * + * @deprecated since 3.0.0 and will be removed in 4.0.0. Use + * {@link #postCheckAndMutate(ObserverContext, CheckAndMutate, CheckAndMutateResult)} instead. */ + @Deprecated default boolean postCheckAndDelete(ObserverContext c, byte[] row, Filter filter, Delete delete, boolean result) throws IOException { return result; } + /** + * Called before checkAndMutate + * + * Call CoprocessorEnvironment#bypass to skip default actions. + * If 'bypass' is set, we skip out on calling any subsequent chained coprocessors. + * + * Note: Do not retain references to any Cells in actions beyond the life of this invocation. + * If need a Cell reference for later use, copy the cell and use that. + * @param c the environment provided by the region server + * @param checkAndMutate the CheckAndMutate object + * @param result the default value of the result + * @return the return value to return to client if bypassing default processing + * @throws IOException if an error occurred on the coprocessor + */ + default CheckAndMutateResult preCheckAndMutate(ObserverContext c, +CheckAndMutate checkAndMutate, CheckAndMutateResult result) throws IOException { +if (checkAndMutate.getAction() instanceof Put) { Review comment: Put/Delete is public so let's not do this... ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java ## @@ -4335,32 +4391,14 @@ private boolean doCheckAndRowMutate(byte[] row, byte[] family, byte[] qualifier, checkRow(row, "doCheckAndRowMutate"); RowLock rowLock = getRowLockInternal(get.getRow(), false, null); try { -if (mutation != null && this.getCoprocessorHost() != null) { - // Call coprocessor. - Boolean processed = null; - if (mutation instanceof Put) { -if (filter != null) { - processed = this.getCoprocessorHost() Review comment: So the old coprocessor method
[GitHub] [hbase] Apache9 opened a new pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency
Apache9 opened a new pull request #2128: URL: https://github.com/apache/hbase/pull/2128 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-19256) [hbase-thirdparty] shade jetty
[ https://issues.apache.org/jira/browse/HBASE-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163611#comment-17163611 ] Duo Zhang commented on HBASE-19256: --- Hadoop 3.3.0 is out. Let's move forward. [~weichiu] Hope you do not mind if I take this issue. It has been dead for a lont time. Thanks. > [hbase-thirdparty] shade jetty > -- > > Key: HBASE-19256 > URL: https://issues.apache.org/jira/browse/HBASE-19256 > Project: HBase > Issue Type: Task > Components: dependencies, thirdparty >Reporter: Mike Drob >Assignee: Wei-Chiu Chuang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side
Apache-HBase commented on pull request #2094: URL: https://github.com/apache/hbase/pull/2094#issuecomment-663032975 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | checkstyle | 2m 10s | master passed | | +1 :green_heart: | spotbugs | 3m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 54s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | The patch passed checkstyle in hbase-hadoop-compat | | +1 :green_heart: | checkstyle | 0m 33s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 17s | hbase-server: The patch generated 0 new + 410 unchanged - 22 fixed = 410 total (was 432) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 5m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. | | | | 56m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2094 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 06564bdd9af8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-hadoop-compat hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2124: URL: https://github.com/apache/hbase/pull/2124#issuecomment-663044535 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 30s | branch-2 passed | | +1 :green_heart: | compile | 1m 18s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 5m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 151m 24s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 28s | hbase-shell in the patch passed. | | | | 183m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2124 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cf99393fcc77 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ce4e692699 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/testReport/ | | Max. process+thread count | 3431 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency
Apache-HBase commented on pull request #2128: URL: https://github.com/apache/hbase/pull/2128#issuecomment-663044354 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 6s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 14m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 0s | The patch does not generate ASF License warnings. | | | | 27m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2128 | | Optional Tests | dupname asflicense hadoopcheck xml | | uname | Linux 816d7ca9ebc0 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 63 (vs. ulimit of 12500) | | modules | C: . hbase-client hbase-common hbase-examples hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2124: URL: https://github.com/apache/hbase/pull/2124#issuecomment-663046009 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 18s | branch-2 passed | | +1 :green_heart: | compile | 1m 32s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 54s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 21s | the patch passed | | +1 :green_heart: | compile | 1m 52s | the patch passed | | +1 :green_heart: | javac | 1m 52s | the patch passed | | +1 :green_heart: | shadedjars | 8m 10s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 141m 20s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 8s | hbase-shell in the patch passed. | | | | 180m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2124 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 61ffb573f6e8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ce4e692699 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/testReport/ | | Max. process+thread count | 4167 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2124/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24738) [Shell] processlist command fails with ERROR: Unexpected end of file from server when SSL enabled
[ https://issues.apache.org/jira/browse/HBASE-24738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163642#comment-17163642 ] Elliot Miller commented on HBASE-24738: --- Hey [~pankajkumar], thanks for doing this! Sorry for the delay. My understanding is that so far, client configurations have been pretty simple and just need either a Zookeeper-based or master-based connection registry. The issue would seem to be that there is no notion of the protocol (ie. HTTP or HTTPS). The implementation uncommented by your PR seems like it should work, although it uses configuration keys intended for the server. +It seems like more of a "workaround" solution than the ideal one, but maybe that is okay+. The main alternatives I see: # We could have the client attempt an HTTP connection and try HTTPS if the first connection gets dropped. # We could allow users to specify SSL/TLS in the processlist command itself. It would be great to settle on a pattern for any commands that need to connect to the info port of our region servers. Worth a footnote is that we might be able to have Jetty upgrade HTTP to HTTPS automatically, but that would be server-side and perhaps a more long-term change. It may be okay to go with your proposed changes, but I wanted to mention my thoughts first. What do you think? > [Shell] processlist command fails with ERROR: Unexpected end of file from > server when SSL enabled > - > > Key: HBASE-24738 > URL: https://issues.apache.org/jira/browse/HBASE-24738 > Project: HBase > Issue Type: Bug > Components: shell >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Major > > HBase Shell command "processlist" fails with ERROR: Unexpected end of file > from server when HBase SSL enabled. > > Below code is commented since beginning, see HBASE-4368, > [https://github.com/apache/hbase/blob/8076eafb187ce32d4a78aef482b6218d85a985ac/hbase-shell/src/main/ruby/hbase/taskmonitor.rb#L85] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bitoffdev commented on pull request #2123: HBASE-24738 [Shell] processlist command fails with ERROR: Unexpected end of file from server when SSL enabled
bitoffdev commented on pull request #2123: URL: https://github.com/apache/hbase/pull/2123#issuecomment-663047251 > @bitoffdev please review. - Just left a comment in Jira: https://issues.apache.org/jira/browse/HBASE-24738?focusedCommentId=17163642#comment-17163642 - The changes should work, but I want to hear your thoughts first on what I just mentioned in Jira. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache9 opened a new pull request #22: HBASE-19256 [hbase-thirdparty] shade jetty
Apache9 opened a new pull request #22: URL: https://github.com/apache/hbase-thirdparty/pull/22 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache9 commented on pull request #22: HBASE-19256 [hbase-thirdparty] shade jetty
Apache9 commented on pull request #22: URL: https://github.com/apache/hbase-thirdparty/pull/22#issuecomment-663065833 Still need to take a look at the jersey part. We need to exclude the general API jar, just like what I have done for jetty, to exclude servlet-api. > [INFO] Including org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.31 in the shaded jar. [INFO] Including org.glassfish.hk2.external:jakarta.inject:jar:2.6.1 in the shaded jar. [INFO] Including org.glassfish.jersey.core:jersey-common:jar:2.31 in the shaded jar. [INFO] Including org.glassfish.hk2:osgi-resource-locator:jar:1.0.3 in the shaded jar. [INFO] Including jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6 in the shaded jar. [INFO] Including org.glassfish.jersey.core:jersey-client:jar:2.31 in the shaded jar. [INFO] Including org.glassfish.jersey.core:jersey-server:jar:2.31 in the shaded jar. [INFO] Including org.glassfish.jersey.media:jersey-media-jaxb:jar:2.31 in the shaded jar. [INFO] Including jakarta.annotation:jakarta.annotation-api:jar:1.3.5 in the shaded jar. [INFO] Including jakarta.validation:jakarta.validation-api:jar:2.0.2 in the shaded jar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency
virajjasani commented on a change in pull request #2128: URL: https://github.com/apache/hbase/pull/2128#discussion_r459532587 ## File path: hbase-examples/pom.xml ## @@ -205,6 +205,7 @@ com.google.protobuf protobuf-java + ${internal.protobuf.version} Review comment: We are keeping only in hbase-examples, sounds good. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side
Apache-HBase commented on pull request #2094: URL: https://github.com/apache/hbase/pull/2094#issuecomment-663094953 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 38s | master passed | | +1 :green_heart: | compile | 1m 37s | master passed | | +1 :green_heart: | shadedjars | 5m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 20s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | +1 :green_heart: | compile | 1m 38s | the patch passed | | +1 :green_heart: | javac | 1m 38s | the patch passed | | +1 :green_heart: | shadedjars | 5m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 34s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 1m 3s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 139m 22s | hbase-server in the patch passed. | | | | 168m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2094 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 50c6da3d5a3d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/testReport/ | | Max. process+thread count | 4723 (vs. ulimit of 12500) | | modules | C: hbase-hadoop-compat hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold
Apache-HBase commented on pull request #2127: URL: https://github.com/apache/hbase/pull/2127#issuecomment-663095071 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 5m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | shadedjars | 5m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 16s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 141m 14s | hbase-server in the patch passed. | | | | 169m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2127 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 53a4726e0545 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/testReport/ | | Max. process+thread count | 5124 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on a change in pull request #2118: URL: https://github.com/apache/hbase/pull/2118#discussion_r459570210 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java ## @@ -513,8 +514,14 @@ public boolean replicate(ReplicateContext replicateContext) { int numSinks = replicationSinkMgr.getNumSinks(); if (numSinks == 0) { - LOG.warn("{} No replication sinks found, returning without replicating. " -+ "The source should retry with the same set of edits.", logPeerId()); + if((System.currentTimeMillis() - lastSinkFetchTime) >= (maxRetriesMultiplier*1000)) { +LOG.warn( + "No replication sinks found, returning without replicating. " ++ "The source should retry with the same set of edits. Not logging this again for " ++ "the next " + maxRetriesMultiplier + " seconds."); +lastSinkFetchTime = System.currentTimeMillis(); + } + sleepForRetries("No sinks available at peer", sleepMultiplier); Review comment: peer id is already been logged inside _sleepForRetries_ method. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on a change in pull request #2118: URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java ## @@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer sinkPeer) { */ public synchronized void chooseSinks() { List slaveAddresses = endpoint.getRegionServers(); -Collections.shuffle(slaveAddresses, ThreadLocalRandom.current()); -int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); -sinks = slaveAddresses.subList(0, numSinks); +if(slaveAddresses==null){ Review comment: > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding against custom endpoint implementations? Yeah, that's indeed the case. >We should expose getRegionServers on a base-class or interface and explicitly say that we expect a non-null answer. Follow-on.. If easy, this would be good to write a quick unit test to cover this method. Sounds good, actually. So we can leave `getRegionServers` at HBaseReplicationEndpoint, emphasize it should never return null and add UT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on a change in pull request #2118: URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java ## @@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer sinkPeer) { */ public synchronized void chooseSinks() { List slaveAddresses = endpoint.getRegionServers(); -Collections.shuffle(slaveAddresses, ThreadLocalRandom.current()); -int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); -sinks = slaveAddresses.subList(0, numSinks); +if(slaveAddresses==null){ Review comment: > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding against custom endpoint implementations? Yeah, that's indeed the case. >We should expose getRegionServers on a base-class or interface and explicitly say that we expect a non-null answer. Follow-on.. If easy, this would be good to write a quick unit test to cover this method. Sounds good, actually. So we can leave `getRegionServers` at HBaseReplicationEndpoint, emphasize it should never return null and add UT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on a change in pull request #2118: URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java ## @@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer sinkPeer) { */ public synchronized void chooseSinks() { List slaveAddresses = endpoint.getRegionServers(); -Collections.shuffle(slaveAddresses, ThreadLocalRandom.current()); -int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); -sinks = slaveAddresses.subList(0, numSinks); +if(slaveAddresses==null){ Review comment: > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding against custom endpoint implementations? Yeah, that's indeed the case. >We should expose getRegionServers on a base-class or interface and explicitly say that we expect a non-null answer. Follow-on.. If easy, this would be good to write a quick unit test to cover this method. Sounds good, actually. So we can leave `getRegionServers` at HBaseReplicationEndpoint, emphasize it should never return null and add UT. > So maybe we can safely log warning for if(slaveAddresses.size()==0) ? I guess we can leave it without the extra logging here, as this would already cause the extra logging we are suppressing in HBaseInterClusterReplicationEndpoint #518. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on a change in pull request #2118: URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java ## @@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer sinkPeer) { */ public synchronized void chooseSinks() { List slaveAddresses = endpoint.getRegionServers(); -Collections.shuffle(slaveAddresses, ThreadLocalRandom.current()); -int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); -sinks = slaveAddresses.subList(0, numSinks); +if(slaveAddresses==null){ Review comment: > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding against custom endpoint implementations? Yeah, that's indeed the case. >We should expose getRegionServers on a base-class or interface and explicitly say that we expect a non-null answer. Follow-on.. If easy, this would be good to write a quick unit test to cover this method. Sounds good, actually. So we can leave `getRegionServers` at HBaseReplicationEndpoint, emphasize it should never return null and add UT. > So maybe we can safely log warning for if(slaveAddresses.size()==0) ? I guess we can leave it without the extra logging here, as this would already cause the extra logging we are suppressing in HBaseInterClusterReplicationEndpoint #518. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on a change in pull request #2118: URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java ## @@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer sinkPeer) { */ public synchronized void chooseSinks() { List slaveAddresses = endpoint.getRegionServers(); -Collections.shuffle(slaveAddresses, ThreadLocalRandom.current()); -int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); -sinks = slaveAddresses.subList(0, numSinks); +if(slaveAddresses==null){ Review comment: > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding against custom endpoint implementations? Yeah, that's indeed the case. >We should expose getRegionServers on a base-class or interface and explicitly say that we expect a non-null answer. Follow-on.. If easy, this would be good to write a quick unit test to cover this method. Sounds good, actually. So we can leave `getRegionServers` at HBaseReplicationEndpoint, emphasize it should never return null and add UT. > So maybe we can safely log warning for if(slaveAddresses.size()==0) ? Yeah, can just leave that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on a change in pull request #2118: URL: https://github.com/apache/hbase/pull/2118#discussion_r459572083 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java ## @@ -150,9 +151,14 @@ public synchronized void reportSinkSuccess(SinkPeer sinkPeer) { */ public synchronized void chooseSinks() { List slaveAddresses = endpoint.getRegionServers(); -Collections.shuffle(slaveAddresses, ThreadLocalRandom.current()); -int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); -sinks = slaveAddresses.subList(0, numSinks); +if(slaveAddresses==null){ Review comment: > Doesn't look like HBaseReplicationEndpoint ever returns null. Guarding against custom endpoint implementations? Yeah, that's indeed the case. >We should expose getRegionServers on a base-class or interface and explicitly say that we expect a non-null answer. Follow-on.. If easy, this would be good to write a quick unit test to cover this method. So we can leave `getRegionServers` at HBaseReplicationEndpoint, emphasize it should never return null. I'm not sure a UT for this would be effective, though. Custom implementations of HBaseReplicationEndpoint returning null would still blow it at runtime. > So maybe we can safely log warning for if(slaveAddresses.size()==0) ? Yeah, can just leave that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
Apache-HBase commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-663115381 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 6s | master passed | | +1 :green_heart: | compile | 1m 16s | master passed | | +1 :green_heart: | shadedjars | 6m 41s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 47s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 40s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | shadedjars | 6m 49s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 52s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 222m 20s | hbase-server in the patch passed. | | | | 253m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4cfb3f3f1b1c 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/testReport/ | | Max. process+thread count | 3201 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani closed pull request #2119: Backport "HBASE-24696 Include JVM information on Web UI under "Software Attributes"" to branch-2.2
virajjasani closed pull request #2119: URL: https://github.com/apache/hbase/pull/2119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24757) ReplicationSink should limit the batch size for batch mutations based on hbase.rpc.rows.warning.threshold
[ https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163762#comment-17163762 ] Michael Stack commented on HBASE-24757: --- Perhaps this belongs elsewhere but is there any point limiting by row count? I'd think size important and row count just an incidental? Does warning when too many rows have any benefit? I'd think a warn when a big batch and too few rows – i.e. incoming edits are really fat – would be more beneficial? Thanks. > ReplicationSink should limit the batch size for batch mutations based on > hbase.rpc.rows.warning.threshold > - > > Key: HBASE-24757 > URL: https://issues.apache.org/jira/browse/HBASE-24757 > Project: HBase > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > At times there are quite a large no of WAL Edits to ship as part of > Replication and sometimes replication queues accumulate huge list of Edits to > process. ReplicationSink at the sink server usually goes through all Edits > and creates map of table -> list of rows grouped by clusterIds, and performs > batch mutation of all rows per table level. However, there is no limit to no > of Rows that are sent as part of batch mutate call. If no of rows > limit > threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn > "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is > turned on, RS will reject the whole batch without processing. > We should let Replication Sink honour this threshold value and accordingly > keep the size lower per batch mutation call. > Replication triggered batch mutations should always be consumed but keeping > limit of mutation low enough will let the system function at the same pace > and without triggering warnings. This will also restrict exploitation of heap > and cpu cycles at the destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24209) [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality
[ https://issues.apache.org/jira/browse/HBASE-24209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163764#comment-17163764 ] Wei-Chiu Chuang commented on HBASE-24209: - I'll resolve this one as a won't fix since Hadoop 3.3.0 is GA now. I'll get a new jira to add a 3.3.0 GA precommit check > [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality > - > > Key: HBASE-24209 > URL: https://issues.apache.org/jira/browse/HBASE-24209 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Wei-Chiu Chuang >Priority: Major > > Since HBASE-23833 we're paying attention to our builds on Hadoop trunk, > currently 3.3.0-SNAPSHOT. Let's add this version to the version lists in > hadoopcheck so our CI will let us know when things break, at least > compile-time anyway. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #2127: HBASE-24757 : ReplicationSink should limit row size in batch mutation based on hbase.rpc.rows.warning.threshold
saintstack commented on a change in pull request #2127: URL: https://github.com/apache/hbase/pull/2127#discussion_r459587451 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -1583,6 +1583,16 @@ "hbase.regionserver.slowlog.systable.enabled"; public static final boolean DEFAULT_SLOW_LOG_SYS_TABLE_ENABLED_KEY = false; + /** + * Number of rows in a batch operation above which a warning will be logged. + */ + public static final String BATCH_ROWS_THRESHOLD_NAME = "hbase.rpc.rows.warning.threshold"; + + /** + * Default value of {@link #BATCH_ROWS_THRESHOLD_NAME} + */ + public static final int BATCH_ROWS_THRESHOLD_DEFAULT = 5000; + Review comment: Why the move? HConstants is often considered an anti-pattern; better to have constants beside where they are used? ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java ## @@ -93,15 +94,21 @@ private SourceFSConfigurationProvider provider; private WALEntrySinkFilter walEntrySinkFilter; + /** + * Row size threshold for multi requests above which a warning is logged + */ + private final int rowSizeWarnThreshold; + /** * Create a sink for replication * @param conf conf object - * @param stopper boolean to tell this thread to stop * @throws IOException thrown when HDFS goes bad or bad file name */ - public ReplicationSink(Configuration conf, Stoppable stopper) + public ReplicationSink(Configuration conf) Review comment: Is the Stoppable unused? Usually its a chain for a Service to pull on when it meets a condition it can't deal with... one that is so bad it wants to stop the process. ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java ## @@ -403,13 +398,24 @@ public void stopReplicationSinkServices() { * Do the changes and handle the pool * @param tableName table to insert into * @param allRows list of actions + * @param batchRowSizeThreshold rowSize threshold for batch mutation */ - private void batch(TableName tableName, Collection> allRows) throws IOException { + private void batch(TableName tableName, Collection> allRows, int batchRowSizeThreshold) + throws IOException { if (allRows.isEmpty()) { return; } AsyncTable table = getConnection().getTable(tableName); -List> futures = allRows.stream().map(table::batchAll).collect(Collectors.toList()); +List> futures = new ArrayList<>(); +for (List rows : allRows) { + List> batchRows; + if (rows.size() > batchRowSizeThreshold) { +batchRows = Lists.partition(rows, batchRowSizeThreshold); + } else { +batchRows = Collections.singletonList(rows); + } + futures.addAll(batchRows.stream().map(table::batchAll).collect(Collectors.toList())); +} Review comment: For the edits that are beyond the limit, they get handled in next batch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
Apache-HBase commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-663120995 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 6m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 6m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 237m 7s | hbase-server in the patch failed. | | | | 264m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2814c65b9306 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/testReport/ | | Max. process+thread count | 2906 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2125/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2094: HBASE-24680 Refactor the checkAndMutate code on the server side
Apache-HBase commented on pull request #2094: URL: https://github.com/apache/hbase/pull/2094#issuecomment-663124264 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 43s | master passed | | +1 :green_heart: | compile | 1m 59s | master passed | | +1 :green_heart: | shadedjars | 6m 24s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-hadoop-compat in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | +1 :green_heart: | compile | 1m 59s | the patch passed | | +1 :green_heart: | javac | 1m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 20s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-hadoop-compat in the patch failed. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 40s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 1m 18s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 193m 2s | hbase-server in the patch passed. | | | | 227m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2094 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f4b2ae5b9dfb 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/testReport/ | | Max. process+thread count | 3165 (vs. ulimit of 12500) | | modules | C: hbase-hadoop-compat hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2094/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org