[GitHub] [hbase] bharathv commented on a change in pull request #2555: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
bharathv commented on a change in pull request #2555:
URL: https://github.com/apache/hbase/pull/2555#discussion_r509864934
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
##
@@ -101,7 +107,7 @@
* Basic constructor that simply initializes values to reasonable defaults.
*/
public WALPrettyPrinter() {
-this(false, false, -1, new HashSet(), null, null, false, false,
System.out);
+this(false, false, -1, new HashSet(), null, null, null, false,
false, System.out);
Review comment:
I think its time to use builder pattern for this, its getting annoyingly
long (perhaps in another patch).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25029) Resolve the TODO of AssignmentManager's loadMeta() method.
[ https://issues.apache.org/jira/browse/HBASE-25029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218764#comment-17218764 ] Sanjeet Nishad commented on HBASE-25029: [~GeorryHuang] Will share a patch by this weekend. > Resolve the TODO of AssignmentManager's loadMeta() method. > -- > > Key: HBASE-25029 > URL: https://issues.apache.org/jira/browse/HBASE-25029 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 2.2.3 >Reporter: Sanjeet Nishad >Assignee: Sanjeet Nishad >Priority: Minor > > All the assignments are blocked until meta is loaded. Better to use a > threadpool for visiting the meta regions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25029) Resolve the TODO of AssignmentManager's loadMeta() method.
[ https://issues.apache.org/jira/browse/HBASE-25029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218762#comment-17218762 ] Zhuoyue Huang commented on HBASE-25029: --- [~sanjeetnishad] still working on it? > Resolve the TODO of AssignmentManager's loadMeta() method. > -- > > Key: HBASE-25029 > URL: https://issues.apache.org/jira/browse/HBASE-25029 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 2.2.3 >Reporter: Sanjeet Nishad >Assignee: Sanjeet Nishad >Priority: Minor > > All the assignments are blocked until meta is loaded. Better to use a > threadpool for visiting the meta regions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25090) CompactionConfiguration logs unrealistic store file sizes
[
https://issues.apache.org/jira/browse/HBASE-25090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218756#comment-17218756
]
Zhuoyue Huang commented on HBASE-25090:
---
Let me handle this issue
> CompactionConfiguration logs unrealistic store file sizes
> -
>
> Key: HBASE-25090
> URL: https://issues.apache.org/jira/browse/HBASE-25090
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 2.3.2
>Reporter: Nick Dimiduk
>Assignee: Zhuoyue Huang
>Priority: Minor
>
> I noticed this in the debug logging of a unit test that makes use of a
> MiniCluster. I'm pretty sure {{8.00 EB}} -- exabytes?? -- is not an accurate
> reporting of the size of these HFiles.
> {noformat}
> 2020-09-22 16:20:57,996 INFO
> [StoreOpener-5e0b426a952230abb65623fc54869833-1]
> compactions.CompactionConfiguration(177): size [128 MB, 8.00 EB, 8.00 EB);
> files [3, 10); ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25090) CompactionConfiguration logs unrealistic store file sizes
[
https://issues.apache.org/jira/browse/HBASE-25090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhuoyue Huang reassigned HBASE-25090:
-
Assignee: Zhuoyue Huang
> CompactionConfiguration logs unrealistic store file sizes
> -
>
> Key: HBASE-25090
> URL: https://issues.apache.org/jira/browse/HBASE-25090
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 2.3.2
>Reporter: Nick Dimiduk
>Assignee: Zhuoyue Huang
>Priority: Minor
>
> I noticed this in the debug logging of a unit test that makes use of a
> MiniCluster. I'm pretty sure {{8.00 EB}} -- exabytes?? -- is not an accurate
> reporting of the size of these HFiles.
> {noformat}
> 2020-09-22 16:20:57,996 INFO
> [StoreOpener-5e0b426a952230abb65623fc54869833-1]
> compactions.CompactionConfiguration(177): size [128 MB, 8.00 EB, 8.00 EB);
> files [3, 10); ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-714233013 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +1 :green_heart: | compile | 1m 31s | master passed | | +1 :green_heart: | shadedjars | 9m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 8s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 46s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | +1 :green_heart: | shadedjars | 8m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 39s | hbase-common in the patch passed. | | -1 :x: | unit | 153m 17s | hbase-server in the patch failed. | | | | 190m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f474d78007bf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/testReport/ | | Max. process+thread count | 3733 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2575: URL: https://github.com/apache/hbase/pull/2575#issuecomment-714232829 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 16s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 58s | branch-2 passed | | +1 :green_heart: | compile | 1m 15s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 7m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 203m 36s | hbase-server in the patch passed. | | | | 235m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2575 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 806f6aea1b0d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9d61afae6e | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/testReport/ | | Max. process+thread count | 2873 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2577: HBASE-25186 Addendum remove extra sleep to fix TestMasterRegionWALCle…
Apache-HBase commented on pull request #2577: URL: https://github.com/apache/hbase/pull/2577#issuecomment-714231373 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | branch-2 passed | | +1 :green_heart: | compile | 1m 1s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 7m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 49s | hbase-server in the patch passed. | | | | 178m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2577 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 231ca6e7a5d9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9d61afae6e | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/testReport/ | | Max. process+thread count | 3374 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-714230243 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | shadedjars | 7m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 7m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 205m 31s | hbase-server in the patch failed. | | | | 234m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2574 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b55810642ec3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/testReport/ | | Max. process+thread count | 3398 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2575: URL: https://github.com/apache/hbase/pull/2575#issuecomment-714229883 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | branch-2 passed | | +1 :green_heart: | compile | 0m 58s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 201m 43s | hbase-server in the patch passed. | | | | 228m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2575 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d9dd7b401588 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9d61afae6e | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/testReport/ | | Max. process+thread count | 2797 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-714223685 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | compile | 1m 30s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 1m 32s | the patch passed | | +1 :green_heart: | javac | 1m 32s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 44s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 134m 25s | hbase-server in the patch passed. | | | | 166m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 031a5fcf1450 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/testReport/ | | Max. process+thread count | 4193 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2577: HBASE-25186 Addendum remove extra sleep to fix TestMasterRegionWALCle…
Apache-HBase commented on pull request #2577: URL: https://github.com/apache/hbase/pull/2577#issuecomment-714223315 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | branch-2 passed | | +1 :green_heart: | compile | 1m 7s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 129m 17s | hbase-server in the patch passed. | | | | 157m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2577 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c84afbbc8011 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9d61afae6e | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/testReport/ | | Max. process+thread count | 4725 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-714212342 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 25s | master passed | | +1 :green_heart: | compile | 1m 24s | master passed | | +1 :green_heart: | shadedjars | 9m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 38s | the patch passed | | +1 :green_heart: | compile | 1m 22s | the patch passed | | +1 :green_heart: | javac | 1m 22s | the patch passed | | +1 :green_heart: | shadedjars | 8m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 46s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 146m 30s | hbase-server in the patch failed. | | | | 183m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2574 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b514233cfa79 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/testReport/ | | Max. process+thread count | 3967 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
bharathv commented on a change in pull request #2574:
URL: https://github.com/apache/hbase/pull/2574#discussion_r509864069
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -8748,6 +8863,11 @@ public void startRegionOperation(Operation op) throws
IOException {
throw new
NotServingRegionException(getRegionInfo().getRegionNameAsString() + " is
closing");
}
lock(lock.readLock());
+// Update regionLockHolders ONLY for any startRegionOperation call that is
invoked from an RPC handler
+Thread thisThread = Thread.currentThread();
+if (isInterruptableOp) {
+ regionLockHolders.put(thisThread.hashCode(), thisThread);
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
bharathv commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509861954
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,78 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
+ @GuardedBy("configUpdateLock") private boolean splitEnabled;
+ @GuardedBy("configUpdateLock") private boolean mergeEnabled;
+ @GuardedBy("configUpdateLock") private int minRegionCount;
+ @GuardedBy("configUpdateLock") private Period mergeMinRegionAge;
+ @GuardedBy("configUpdateLock") private int mergeMinRegionSizeMb;
public SimpleRegionNormalizer() {
-splitEnabled = DEFAULT_SPLIT_ENABLED;
-mergeEnabled = DEFAULT_MERGE_ENABLED;
-minRegionCount = DEFAULT_MIN_REGION_COUNT;
-mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
-mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
+try {
+ splitEnabled = DEFAULT_SPLIT_ENABLED;
+ mergeEnabled = DEFAULT_MERGE_ENABLED;
+ minRegionCount = DEFAULT_MIN_REGION_COUNT;
+ mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
+ mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+} finally {
+ writeLock.unlock();
+}
}
@Override
public Configuration getConf() {
-return conf;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return conf;
+} finally {
+ readLock.unlock();
+}
}
@Override
public void setConf(final Configuration conf) {
if (conf == null) {
return;
}
-this.conf = conf;
-splitEnabled = conf.getBoolean(SPLIT_ENABLED_KEY, DEFAULT_SPLIT_ENABLED);
-mergeEnabled = conf.getBoolean(MERGE_ENABLED_KEY, DEFAULT_MERGE_ENABLED);
-minRegionCount = parseMinRegionCount(conf);
-mergeMinRegionAge = parseMergeMinRegionAge(conf);
-mergeMinRegionSizeMb = parseMergeMinRegionSizeMb(conf);
+
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
Review comment:
Stumbled upon this change. I think there is a much simpler way to
achieve this without locks and fewer lines of code (and cleaner). If we can
factor all of the configs into a single object (with appropriate getters and
setters if needed), something like,
```
static class NormalizerConfig {
Configuration conf;
boolean splitEnabled;
private boolean mergeEnabled;
private Period mergeMinRegionAge;
private int mergeMinRegionSizeMb;
...
static parseFromConfig(Conf conf);
}
private NormalizerConfig normalizerConf;
public void setConf(final Configuration conf) {
normalizerConf = parseFromConfig(conf);
}
public boolean isSplitEnabled() {
return normalizerConf.isSplitEnabled();
}
```
Reference assignment is atomic. So even if multiple threads call
setConf(conf), each thread calls its own parseFromConfig() in it's own context,
constructs the whole object and the reference assignment works cleanly. On the
reader side depending on what reference is being used that point, the value is
returned (ex: isSplitEnabled() above)..
The advantage of using these locks is the memory ordering that they enforce
in methods like isSplitEnabled(). We essentially block until the reference is
updated but I don't think that is a requirement here because we don't guarantee
the callers of these methods (like computePlansForTable()) that they will work
on the latest config while the config update is in progress (we can't guarantee
that level of ordering anyway). Point here being the above approach gets rid of
most code and is still not racy. WDYT.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2578: HBASE-24768 Clear cached service kerberos ticket in case of SASL fail…
Apache-HBase commented on pull request #2578: URL: https://github.com/apache/hbase/pull/2578#issuecomment-714197769 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 10m 2s | branch-1 passed | | +1 :green_heart: | compile | 0m 21s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | compile | 0m 28s | branch-1 passed with JDK v1.7.0_272 | | +1 :green_heart: | checkstyle | 0m 58s | branch-1 passed | | +1 :green_heart: | shadedjars | 4m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 27s | branch-1 passed with JDK v1.7.0_272 | | +0 :ok: | spotbugs | 2m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 11s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 40s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | javac | 0m 27s | the patch passed | | -1 :x: | checkstyle | 0m 47s | hbase-client: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 6m 7s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | findbugs | 1m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 56s | hbase-client in the patch passed. | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 45m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2578/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2578 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0a3fdc61440c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2578/out/precommit/personality/provided.sh | | git revision | branch-1 / 8e387e9 | | Default Java | 1.7.0_272 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2578/1/artifact/out/diff-checkstyle-hbase-client.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2578/1/testReport/ | | Max. process+thread count | 138 (vs. ulimit of 1) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2578/1/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25186) TestMasterRegionOnTwoFileSystems is failing after HBASE-25065
[ https://issues.apache.org/jira/browse/HBASE-25186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25186. --- Resolution: Fixed > TestMasterRegionOnTwoFileSystems is failing after HBASE-25065 > - > > Key: HBASE-25186 > URL: https://issues.apache.org/jira/browse/HBASE-25186 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: ramkrishna.s.vasudevan >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.4.0 > > > After HBASE-25065, we are having a test case failure with > TestMasterRegionOnTwoFileSystems. > The reason is that we manually trigger a WAL roll on the master region. As > part of the WAL roll we expect the Master region's WAL will also be moved > from region oldWAL dir to the global oldWAL directory. This happens after > afterRoll() method in AbstractWALRoller. > Since now the WAL archival is asynchronous the afterRoll() method does not > find any WAL file to be moved in the local region oldWAL dir. So the movement > to global oldWAL dir does not happen. > The test case checks for the file in the oldWAL dir and since it is not found > the test timesout. WE need a way to fix this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #2577: HBASE-25186 Addendum remove extra sleep to fix TestMasterRegionWALCle…
Apache9 merged pull request #2577: URL: https://github.com/apache/hbase/pull/2577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-20998) Move hbase-thrift to hbase-connectors repository
[ https://issues.apache.org/jira/browse/HBASE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuqi reassigned HBASE-20998: Assignee: yuqi > Move hbase-thrift to hbase-connectors repository > > > Key: HBASE-20998 > URL: https://issues.apache.org/jira/browse/HBASE-20998 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors, Thrift >Affects Versions: 2.1.0 >Reporter: Michael Stack >Assignee: yuqi >Priority: Major > > Move hbase-thrift to our new hbase-connectors repository (see parent for > context). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2577: HBASE-25186 Addendum remove extra sleep to fix TestMasterRegionWALCle…
Apache-HBase commented on pull request #2577: URL: https://github.com/apache/hbase/pull/2577#issuecomment-714187021 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 15s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 8s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 44s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 37m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2577 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux bcfc1232e282 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9d61afae6e | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2577/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-714186037 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 24s | master passed | | +1 :green_heart: | checkstyle | 1m 31s | master passed | | +1 :green_heart: | spotbugs | 2m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | checkstyle | 0m 24s | hbase-common: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) | | +1 :green_heart: | checkstyle | 1m 5s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 41m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 181d3a9ab10d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Joseph295 commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Joseph295 commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509847080
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -5301,7 +5301,11 @@ private long replayRecoveredEditsForPaths(long
minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ if (isSkipErrorEnabled() &&
fileName.endsWith(HConstants.HREGION_EDITS_CORRUPTED_SUFFIX)) {
Review comment:
OK, let me add a UT for that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1531: HBASE-24200 Upgrade to Yetus 0.12.0
Apache-HBase commented on pull request #1531: URL: https://github.com/apache/hbase/pull/1531#issuecomment-714185514 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | hadolint | 0m 1s | There were no new hadolint issues. | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 18m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 27m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1531/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1531 | | Optional Tests | dupname asflicense shellcheck shelldocs hadolint hadoopcheck xml | | uname | Linux 151735863b28 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Max. process+thread count | 63 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1531/9/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 hadolint=1.17.5-0-g443423c | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sguggilam opened a new pull request #2578: HBASE-24768 Clear cached service kerberos ticket in case of SASL fail…
sguggilam opened a new pull request #2578: URL: https://github.com/apache/hbase/pull/2578 …ures thrown from server side This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
Apache9 commented on a change in pull request #2574:
URL: https://github.com/apache/hbase/pull/2574#discussion_r509840637
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -1174,7 +1178,7 @@ public HStore call() throws IOException {
LOG.info("Setting FlushNonSloppyStoresFirstPolicy for the region=" +
this);
}
} catch (InterruptedException e) {
-throw (InterruptedIOException)new
InterruptedIOException().initCause(e);
+throwOnInterrupt(e);
Review comment:
Better make it return an exception and make here 'throw
throwOnInterrupt(e)' so the compiler will know that this is an exit point.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -6588,8 +6677,10 @@ protected RowLock getRowLockInternal(byte[] row, boolean
readLock, final RowLock
success = true;
return result;
} catch (InterruptedException ie) {
- LOG.warn("Thread interrupted waiting for lock on row: {}, in region {}",
rowKey,
-getRegionInfo().getRegionNameAsString());
+ if (LOG.isDebugEnabled()) {
+LOG.debug("Thread interrupted waiting for lock on row: {}, in region
{}", rowKey,
Review comment:
Here the interrupt may come from the checkInterrupt call, not acquire
lock?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -8748,6 +8863,11 @@ public void startRegionOperation(Operation op) throws
IOException {
throw new
NotServingRegionException(getRegionInfo().getRegionNameAsString() + " is
closing");
}
lock(lock.readLock());
+// Update regionLockHolders ONLY for any startRegionOperation call that is
invoked from an RPC handler
+Thread thisThread = Thread.currentThread();
+if (isInterruptableOp) {
+ regionLockHolders.put(thisThread.hashCode(), thisThread);
Review comment:
Maybe we could just use Collections.newSetFromMap(new
ConcurrentHashMap<>()) here?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -4550,6 +4619,10 @@ private void doMiniBatchMutate(BatchOperation
batchOp) throws IOException {
/** Keep track of the locks we hold so we can release them in finally
clause */
List acquiredRowLocks =
Lists.newArrayListWithCapacity(batchOp.size());
try {
+ // Check for thread interrupt status in case we have been signaled from
+ // #interruptRegionOperation.
+ checkInterrupt();
Review comment:
Pity we need to repeat this call many times in the same method...
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
##
@@ -7364,4 +7367,154 @@ protected HStoreForTesting(final HRegion region,
return super.doCompaction(cr, filesToCompact, user, compactionStartTime,
newFiles);
}
}
+
+ @Test(timeout=2)
Review comment:
Usually we do not add this to a single test as we have a test timeout
for the whole test file.
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
##
@@ -7364,4 +7367,154 @@ protected HStoreForTesting(final HRegion region,
return super.doCompaction(cr, filesToCompact, user, compactionStartTime,
newFiles);
}
}
+
+ @Test(timeout=2)
+ public void testCloseNoInterrupt() throws Exception {
+byte[] cf1 = Bytes.toBytes("CF1");
+byte[][] families = { cf1 };
+
+Configuration conf = HBaseConfiguration.create();
Review comment:
Use the config from the HBTU? Maybe new
Configuration(TEST_UTIL.getConfiguration())
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionInterrupt.java
##
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertTrue;
+
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import
[GitHub] [hbase] Joseph295 commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Joseph295 commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r509842639
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -5324,22 +5328,25 @@ private long replayRecoveredEditsForPaths(long
minSeqIdForTheRegion, FileSystem
}
private void handleException(FileSystem fs, Path edits, IOException e)
throws IOException {
-boolean skipErrors =
conf.getBoolean(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS,
- conf.getBoolean("hbase.skip.errors",
HConstants.DEFAULT_HREGION_EDITS_REPLAY_SKIP_ERRORS));
-if (conf.get("hbase.skip.errors") != null) {
- LOG.warn("The property 'hbase.skip.errors' has been deprecated. Please
use "
- + HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + " instead.");
-}
-if (skipErrors) {
+if (isSkipErrorEnabled()) {
Path p = WALSplitUtil.moveAsideBadEditsFile(fs, edits);
- LOG.error(HConstants.HREGION_EDITS_REPLAY_SKIP_ERRORS + "=true so
continuing. Renamed "
- + edits + " as " + p,
-e);
+ LOG.info(
Review comment:
This is a behavior as expected?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509841246
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java
##
@@ -435,6 +475,14 @@ private void locateInMeta(TableName tableName,
LocateRequest req) {
.setReadType(ReadType.PREAD);
if (useMetaReplicas) {
scan.setConsistency(Consistency.TIMELINE);
+
+ // If it is meta replica LoadBalance mode, pick up a meta replica to
scan. It will try to
+ // pick up meta replica region to scan, in certain error cases, it may
pick primary meta
+ // region to scan.
+ if (isMetaReplicaLBMode()) {
+scan.setReplicaId(this.metaReplicaChooser.chooseReplicaToGo(tableName,
Review comment:
Mark here. Just realized that there is a logical bug issue. When
chooseReplicaToGo() returns primary region to go, in this case, it should unset
TIMELINE. Otherwise, it will fall back to HighlyAvailable mode, which is not
what we want here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509841481
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java
##
@@ -577,6 +627,15 @@ private void removeLocationFromCache(HRegionLocation loc) {
if (!canUpdateOnError(loc, oldLoc)) {
return;
}
+ // Tell metaReplicaChooser that the location is stale. It will create a
stale entry
+ // with timestamp internally. Next time the client looks up the same
location,
+ // it will pick a different meta replica region. For the current
implementation,
Review comment:
Will do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509840556
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
##
@@ -130,8 +133,12 @@ public AsyncConnectionImpl(Configuration conf,
ConnectionRegistry registry, Stri
SocketAddress localAddress, User user) {
this.conf = conf;
this.user = user;
-if (user.isLoginFromKeytab()) {
- spawnRenewalChore(user.getUGI());
+
+if (user.isLoginFromKeytab() || isMetaReplicaLBMode()) {
Review comment:
My bad, Guava cache does not work in this case, we need a tablecache
like implementation which can look up locations. Will consider chore service
on-demand.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 opened a new pull request #2577: HBASE-25186 Addendum remove extra sleep to fix TestMasterRegionWALCle…
Apache9 opened a new pull request #2577: URL: https://github.com/apache/hbase/pull/2577 …aner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-25186) TestMasterRegionOnTwoFileSystems is failing after HBASE-25065
[ https://issues.apache.org/jira/browse/HBASE-25186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-25186: --- TestMasterRegionWALCleaner is still failing. Just a typo, let me open an addendum PR. > TestMasterRegionOnTwoFileSystems is failing after HBASE-25065 > - > > Key: HBASE-25186 > URL: https://issues.apache.org/jira/browse/HBASE-25186 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: ramkrishna.s.vasudevan >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.4.0 > > > After HBASE-25065, we are having a test case failure with > TestMasterRegionOnTwoFileSystems. > The reason is that we manually trigger a WAL roll on the master region. As > part of the WAL roll we expect the Master region's WAL will also be moved > from region oldWAL dir to the global oldWAL directory. This happens after > afterRoll() method in AbstractWALRoller. > Since now the WAL archival is asynchronous the afterRoll() method does not > find any WAL file to be moved in the local region oldWAL dir. So the movement > to global oldWAL dir does not happen. > The test case checks for the file in the oldWAL dir and since it is not found > the test timesout. WE need a way to fix this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2576: HBASE-25212 [branch-1] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2576: URL: https://github.com/apache/hbase/pull/2576#issuecomment-714175334 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 57s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 51s | branch-1 passed | | +1 :green_heart: | compile | 0m 41s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | compile | 0m 48s | branch-1 passed with JDK v1.7.0_272 | | +1 :green_heart: | checkstyle | 2m 6s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 43s | branch-1 passed with JDK v1.7.0_272 | | +0 :ok: | spotbugs | 3m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 11s | branch-1 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 1m 14s | root in the patch failed. | | -1 :x: | compile | 0m 25s | hbase-server in the patch failed with JDK v1.8.0_262. | | -1 :x: | javac | 0m 25s | hbase-server in the patch failed with JDK v1.8.0_262. | | -1 :x: | compile | 0m 33s | hbase-server in the patch failed with JDK v1.7.0_272. | | -1 :x: | javac | 0m 33s | hbase-server in the patch failed with JDK v1.7.0_272. | | -1 :x: | checkstyle | 1m 55s | hbase-server: The patch generated 7 new + 787 unchanged - 2 fixed = 794 total (was 789) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | shadedjars | 1m 57s | patch has 36 errors when building our shaded downstream artifacts. | | -1 :x: | hadoopcheck | 1m 7s | The patch causes 36 errors with Hadoop v2.8.5. | | -1 :x: | hadoopcheck | 2m 50s | The patch causes 36 errors with Hadoop v2.9.2. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 42s | the patch passed with JDK v1.7.0_272 | | -1 :x: | findbugs | 0m 33s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 33s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 46m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2576 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux d5e07e14bae6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2576/out/precommit/personality/provided.sh | | git revision | branch-1 / 8e387e9 | | Default Java | 1.7.0_272 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/1/artifact/out/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/1/artifact/out/patch-compile-hbase-server-jdk1.8.0_262.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/1/artifact/out/patch-compile-hbase-server-jdk1.8.0_262.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/1/artifact/out/patch-compile-hbase-server-jdk1.7.0_272.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/1/artifact/out/patch-compile-hbase-server-jdk1.7.0_272.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/1/artifact/out/diff-checkstyle-hbase-server.txt | | shadedjars |
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-714173519 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 59s | master passed | | +1 :green_heart: | compile | 1m 46s | master passed | | +1 :green_heart: | shadedjars | 8m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 10s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 41s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 9m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 38s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 150m 2s | hbase-server in the patch passed. | | | | 189m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 52d323e54bac 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/testReport/ | | Max. process+thread count | 4902 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-714172243 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | checkstyle | 1m 18s | master passed | | +1 :green_heart: | spotbugs | 2m 7s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | -0 :warning: | checkstyle | 1m 18s | hbase-server: The patch generated 9 new + 407 unchanged - 1 fixed = 416 total (was 408) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 56s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | -1 :x: | spotbugs | 2m 18s | hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 43m 18s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Switch statement found in org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(Region$Operation) where one case falls through to the next case At HRegion.java:where one case falls through to the next case At HRegion.java:[lines 8845-8852] | | | Switch statement found in org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(Region$Operation) where default case is missing At HRegion.java:where default case is missing At HRegion.java:[lines 8842-8852] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2574 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux dab23d3c88f8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2575: URL: https://github.com/apache/hbase/pull/2575#issuecomment-714170697 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 20s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 58s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 15s | the patch passed | | -0 :warning: | checkstyle | 1m 17s | hbase-server: The patch generated 12 new + 477 unchanged - 1 fixed = 489 total (was 478) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 30s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | -1 :x: | spotbugs | 2m 11s | hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 33m 14s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Switch statement found in org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(Region$Operation) where one case falls through to the next case At HRegion.java:where one case falls through to the next case At HRegion.java:[lines 8816-8823] | | | Switch statement found in org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(Region$Operation) where default case is missing At HRegion.java:where default case is missing At HRegion.java:[lines 8813-8823] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2575 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux c4108232e46b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9d61afae6e | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25212) Optionally abort requests in progress after deciding a region should close
[
https://issues.apache.org/jira/browse/HBASE-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25212:
Status: Patch Available (was: Open)
> Optionally abort requests in progress after deciding a region should close
> --
>
> Key: HBASE-25212
> URL: https://issues.apache.org/jira/browse/HBASE-25212
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> After deciding a region should be closed, the regionserver will set the
> internal region state to closing and wait for all pending requests to
> complete, via a rendezvous on the region lock. In closing state the region
> will not accept any new requests but requests in progress will be allowed to
> complete before the close action takes place. In our production we see
> outlier wait times on this lock in excess of several minutes.
> During close when there are requests in flight the regionserver is subject to
> any conceivable reason for delay, like full scans over large regions,
> expensive filtering hierarchies, bugs, or store level performance problems
> like slow HDFS. The regionserver should interrupt requests in progress to
> facilitate smaller/shorter close times on an opt-in basis.
> Optionally, via configuration parameter -- which would be a system wide
> default set in hbase-site.xml in common practice but could be overridden in
> table schema for per table settings -- interrupt requests in progress holding
> the region lock rather than wait for completion of all operations in flight.
> Send back NotServingRegionException("region is closing") to the clients of
> the interrupted operations, like we do after the write lock is acquired. The
> client will transparently relocate the region data and resubmit the aborted
> requests per normal retry policy. This can be less disruptive than waiting
> for very long times for a region to close in extreme outlier cases (e.g. 50
> minutes). In such extreme cases it is better to abort the regionserver if the
> close lock cannot be acquired in a reasonable amount of time, because the
> region cannot be made available again until it has closed.
> After waiting for all requests to complete then we flush the region's
> memstore and finish the close. The flush portion of the close process is out
> of scope of this proposal. Under normal conditions the flush portion of the
> close completes quickly. It is specifically waits on the close lock that has
> been an occasional issue in our production that causes difficulty achieving
> 99.99% availability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-714159949 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 6m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | +1 :green_heart: | shadedjars | 6m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 134m 20s | hbase-server in the patch passed. | | | | 166m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e10efab39cb0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/testReport/ | | Max. process+thread count | 4176 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell opened a new pull request #2576: HBASE-25212 [branch-1] Optionally abort requests in progress after deciding a region should close
apurtell opened a new pull request #2576: URL: https://github.com/apache/hbase/pull/2576 If hbase.regionserver.close.wait.abort is set to true, interrupt RPC handler threads holding the region close lock. Until requests in progress can be aborted, wait on the region close lock for a configurable interval (specified by hbase.regionserver.close.wait.time, in ms). If we have failed to acquire the close lock after this interval elapses, if allowed (also specified by hbase.regionserver.close.wait.abort), abort the regionserver. Define a subset of region operations as interruptable. Track threads holding the close lock transiting those operations. Set the thread interrupt status of tracked threads when trying to close the region. Use the thread interrupt status where safe to break out of request processing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell opened a new pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
apurtell opened a new pull request #2575: URL: https://github.com/apache/hbase/pull/2575 If hbase.regionserver.close.wait.abort is set to true, interrupt RPC handler threads holding the region close lock. Until requests in progress can be aborted, wait on the region close lock for a configurable interval (specified by hbase.regionserver.close.wait.time, in ms). If we have failed to acquire the close lock after this interval elapses, if allowed (also specified by hbase.regionserver.close.wait.abort), abort the regionserver. Define a subset of region operations as interruptable. Track threads holding the close lock transiting those operations. Set the thread interrupt status of tracked threads when trying to close the region. Use the thread interrupt status where safe to break out of request processing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2565: HBASE-25003 Backport HBASE-24350 and HBASE-24779 to branch-2.2
infraio commented on pull request #2565: URL: https://github.com/apache/hbase/pull/2565#issuecomment-714119345 Seems these patch not go into branch-2.3, too? I thought these should go into 2.3 firstly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell opened a new pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
apurtell opened a new pull request #2574: URL: https://github.com/apache/hbase/pull/2574 If hbase.regionserver.close.wait.abort is set to true, interrupt RPC handler threads holding the region close lock. Until requests in progress can be aborted, wait on the region close lock for a configurable interval (specified by hbase.regionserver.close.wait.time, in ms). If we have failed to acquire the close lock after this interval elapses, if allowed (also specified by hbase.regionserver.close.wait.abort), abort the regionserver. Define a subset of region operations as interruptable. Track threads holding the close lock transiting those operations. Set the thread interrupt status of tracked threads when trying to close the region. Use the thread interrupt status where safe to break out of request processing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25212) Optionally abort requests in progress after deciding a region should close
[
https://issues.apache.org/jira/browse/HBASE-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25212:
Description:
After deciding a region should be closed, the regionserver will set the
internal region state to closing and wait for all pending requests to complete,
via a rendezvous on the region lock. In closing state the region will not
accept any new requests but requests in progress will be allowed to complete
before the close action takes place. In our production we see outlier wait
times on this lock in excess of several minutes.
During close when there are requests in flight the regionserver is subject to
any conceivable reason for delay, like full scans over large regions, expensive
filtering hierarchies, bugs, or store level performance problems like slow
HDFS. The regionserver should interrupt requests in progress to facilitate
smaller/shorter close times on an opt-in basis.
Optionally, via configuration parameter -- which would be a system wide default
set in hbase-site.xml in common practice but could be overridden in table
schema for per table settings -- interrupt requests in progress holding the
region lock rather than wait for completion of all operations in flight. Send
back NotServingRegionException("region is closing") to the clients of the
interrupted operations, like we do after the write lock is acquired. The client
will transparently relocate the region data and resubmit the aborted requests
per normal retry policy. This can be less disruptive than waiting for very long
times for a region to close in extreme outlier cases (e.g. 50 minutes). In such
extreme cases it is better to abort the regionserver if the close lock cannot
be acquired in a reasonable amount of time, because the region cannot be made
available again until it has closed.
After waiting for all requests to complete then we flush the region's memstore
and finish the close. The flush portion of the close process is out of scope of
this proposal. Under normal conditions the flush portion of the close completes
quickly. It is specifically waits on the close lock that has been an occasional
issue in our production that causes difficulty achieving 99.99% availability.
was:
After deciding a region should be closed, the regionserver will set the
internal region state to closing and wait for all pending requests to complete,
via a rendezvous on the region lock. In closing state the region will not
accept any new requests but requests in progress will be allowed to complete
before the close action takes place. In our production we see outlier wait
times on this lock in excess of several minutes.
During close when there are requests in flight the regionserver is subject to
any conceivable reason for delay, like full scans over large regions, expensive
filtering hierarchies, bugs, or store level performance problems like slow
HDFS. The regionserver should interrupt requests in progress to facilitate
smaller/shorter close times on an opt-in basis.
Optionally, via configuration parameter -- which would be a system wide default
set in hbase-site.xml in common practice but could be overridden in table
schema for per table settings -- interrupt requests in progress holding the
region lock rather than wait for completion of all operations in flight. Send
back NotServingRegionException("region is closing") to the clients of the
interrupted operations, like we do after the write lock is acquired. The client
will transparently relocate the region data and resubmit the aborted requests
per normal retry policy. This can be less disruptive than waiting for very long
times for a region to close in extreme outlier cases (e.g. 50 minutes). In such
extreme cases it is better to abort the regionserver if the close lock cannot
be acquired in a reasonable amount of time.
After waiting for all requests to complete then we flush the region's memstore
and finish the close. The flush portion of the close process is out of scope of
this proposal. Under normal conditions the flush portion of the close completes
quickly. It is specifically waits on the close lock that has been an occasional
issue in our production that causes difficulty achieving 99.99% availability.
> Optionally abort requests in progress after deciding a region should close
> --
>
> Key: HBASE-25212
> URL: https://issues.apache.org/jira/browse/HBASE-25212
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> After deciding a region should be closed, the regionserver will set the
> internal region
[jira] [Updated] (HBASE-25212) Optionally abort requests in progress after deciding a region should close
[
https://issues.apache.org/jira/browse/HBASE-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25212:
Description:
After deciding a region should be closed, the regionserver will set the
internal region state to closing and wait for all pending requests to complete,
via a rendezvous on the region lock. In closing state the region will not
accept any new requests but requests in progress will be allowed to complete
before the close action takes place. In our production we see outlier wait
times on this lock in excess of several minutes.
During close when there are requests in flight the regionserver is subject to
any conceivable reason for delay, like full scans over large regions, expensive
filtering hierarchies, bugs, or store level performance problems like slow
HDFS. The regionserver should interrupt requests in progress to facilitate
smaller/shorter close times on an opt-in basis.
Optionally, via configuration parameter -- which would be a system wide default
set in hbase-site.xml in common practice but could be overridden in table
schema for per table settings -- interrupt requests in progress holding the
region lock rather than wait for completion of all operations in flight. Send
back NotServingRegionException("region is closing") to the clients of the
interrupted operations, like we do after the write lock is acquired. The client
will transparently relocate the region data and resubmit the aborted requests
per normal retry policy. This can be less disruptive than waiting for very long
times for a region to close in extreme outlier cases (e.g. 50 minutes). In such
extreme cases it is better to abort the regionserver if the close lock cannot
be acquired in a reasonable amount of time.
After waiting for all requests to complete then we flush the region's memstore
and finish the close. The flush portion of the close process is out of scope of
this proposal. Under normal conditions the flush portion of the close completes
quickly. It is specifically waits on the close lock that has been an occasional
issue in our production that causes difficulty achieving 99.99% availability.
was:
After deciding a region should be closed, the regionserver will set the
internal region state to closing and wait for all pending requests to complete,
via a rendezvous on the region lock. In closing state the region will not
accept any new requests but requests in progress will be allowed to complete
before the close action takes place. In our production we see outlier wait
times on this lock in excess of several minutes.
During close when there are requests in flight the regionserver is subject to
any conceivable reason for delay, like full scans over large regions, expensive
filtering hierarchies, bugs, or store level performance problems like slow
HDFS. The regionserver should interrupt requests in progress to facilitate
smaller/shorter close times on an opt-in basis.
Optionally, via configuration parameter -- which would be a system wide default
set in hbase-site.xml in common practice but could be overridden in table
schema for per table settings -- interrupt requests in progress holding the
region lock rather than wait for completion of all operations in flight. Send
back NotServingRegionException("region is closing") to the clients of the
interrupted operations, like we do after the write lock is acquired. The client
will transparently relocate the region data and resubmit the aborted requests
per normal retry policy. This can be less disruptive than waiting for very long
times for a region to close in extreme outlier cases (e.g. 50 minutes).
After waiting for all requests to complete then we flush the region's memstore
and finish the close. The flush portion of the close process is out of scope of
this proposal. Under normal conditions the flush portion of the close completes
quickly. It is specifically waits on the close lock that has been an occasional
issue in our production that causes difficulty achieving 99.99% availability.
> Optionally abort requests in progress after deciding a region should close
> --
>
> Key: HBASE-25212
> URL: https://issues.apache.org/jira/browse/HBASE-25212
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> After deciding a region should be closed, the regionserver will set the
> internal region state to closing and wait for all pending requests to
> complete, via a rendezvous on the region lock. In closing state the region
> will not accept any new requests but requests in progress will be
[jira] [Created] (HBASE-25212) Optionally abort requests in progress after deciding a region should close
Andrew Kyle Purtell created HBASE-25212:
---
Summary: Optionally abort requests in progress after deciding a
region should close
Key: HBASE-25212
URL: https://issues.apache.org/jira/browse/HBASE-25212
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
After deciding a region should be closed, the regionserver will set the
internal region state to closing and wait for all pending requests to complete,
via a rendezvous on the region lock. In closing state the region will not
accept any new requests but requests in progress will be allowed to complete
before the close action takes place. In our production we see outlier wait
times on this lock in excess of several minutes.
During close when there are requests in flight the regionserver is subject to
any conceivable reason for delay, like full scans over large regions, expensive
filtering hierarchies, bugs, or store level performance problems like slow
HDFS. The regionserver should interrupt requests in progress to facilitate
smaller/shorter close times on an opt-in basis.
Optionally, via configuration parameter -- which would be a system wide default
set in hbase-site.xml in common practice but could be overridden in table
schema for per table settings -- interrupt requests in progress holding the
region lock rather than wait for completion of all operations in flight. Send
back NotServingRegionException("region is closing") to the clients of the
interrupted operations, like we do after the write lock is acquired. The client
will transparently relocate the region data and resubmit the aborted requests
per normal retry policy. This can be less disruptive than waiting for very long
times for a region to close in extreme outlier cases (e.g. 50 minutes).
After waiting for all requests to complete then we flush the region's memstore
and finish the close. The flush portion of the close process is out of scope of
this proposal. Under normal conditions the flush portion of the close completes
quickly. It is specifically waits on the close lock that has been an occasional
issue in our production that causes difficulty achieving 99.99% availability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache9 commented on pull request #2570: URL: https://github.com/apache/hbase/pull/2570#issuecomment-714062524 > Do you think "fallback to primary" logic needs to be passed down from rpc retrying caller? Then it needs to be aware of this feature and needs to maintain some state. Was trying to avoid it. Yeah, this part can be optimized later. If you want to make a decision based on whether the previous returned location is stale, then you need to pass something down from the rpc retrying caller to the selector, as only in rpc retrying caller we can get the exception we want. Either by adding a parameter as I proposed above to not go to secondary replicas, or as your current solution, adding a stale cache in the selector and you need to pass down the exception and the related location. My concern here is that, the current 'simple' algorithm may not work well for some cases as my mentioned above. I suggest that we remove this part of code from the patch here(just do random selecting, without other consideration) to get this patch in quickly, and then open another issue to discuss the suitable algorithm, or maybe multiple algorithms. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
Apache-HBase commented on pull request #2490: URL: https://github.com/apache/hbase/pull/2490#issuecomment-714048462 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 31s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 6m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 6m 27s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 30s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 146m 57s | hbase-server in the patch passed. | | | | 177m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2490 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8016496bc4bf 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/testReport/ | | Max. process+thread count | 4341 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2571: HBASE-25207 Revisit the implementation and usage of RegionStates.include
Apache9 commented on pull request #2571: URL: https://github.com/apache/hbase/pull/2571#issuecomment-714044695 @brfrn169 Mind taking a look? Let's get this in first and then work on HBASE-25206. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
Apache-HBase commented on pull request #2490: URL: https://github.com/apache/hbase/pull/2490#issuecomment-714040832 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 6m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 1m 34s | the patch passed | | +1 :green_heart: | javac | 1m 34s | the patch passed | | +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 34s | hbase-common in the patch passed. | | -1 :x: | unit | 138m 11s | hbase-server in the patch failed. | | | | 171m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2490 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5f729e279f26 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/testReport/ | | Max. process+thread count | 4762 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1531: HBASE-24200 Upgrade to Yetus 0.12.0
Apache-HBase commented on pull request #1531: URL: https://github.com/apache/hbase/pull/1531#issuecomment-714034437 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 15s | master passed | | +1 :green_heart: | compile | 3m 22s | master passed | | +1 :green_heart: | shadedjars | 7m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 3m 9s | the patch passed | | +1 :green_heart: | javac | 3m 9s | the patch passed | | +1 :green_heart: | shadedjars | 7m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 252m 32s | root in the patch passed. | | | | 296m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1531/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1531 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6f9e2e3557da 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1531/8/testReport/ | | Max. process+thread count | 4932 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1531/8/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2555: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Apache-HBase commented on pull request #2555: URL: https://github.com/apache/hbase/pull/2555#issuecomment-714017351 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 52s | branch-1 passed | | +1 :green_heart: | compile | 0m 39s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | compile | 0m 44s | branch-1 passed with JDK v1.7.0_272 | | +1 :green_heart: | checkstyle | 1m 43s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 40s | branch-1 passed with JDK v1.7.0_272 | | +0 :ok: | spotbugs | 3m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 56s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 55s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | javac | 0m 44s | the patch passed | | -1 :x: | checkstyle | 1m 33s | hbase-server: The patch generated 1 new + 21 unchanged - 1 fixed = 22 total (was 22) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 2m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 36s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 41s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | findbugs | 2m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 101m 47s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 149m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2555/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2555 | | JIRA Issue | HBASE-25193 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux f02b2fbcf92a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-agent/workspace/Base-PreCommit-GitHub-PR_PR-2555@2/out/precommit/personality/provided.sh | | git revision | branch-1 / 8e387e9 | | Default Java | 1.7.0_272 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2555/3/artifact/out/diff-checkstyle-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2555/3/testReport/ | | Max. process+thread count | 3464 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2555/3/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713984113 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 25s | master passed | | +1 :green_heart: | checkstyle | 1m 30s | master passed | | +1 :green_heart: | spotbugs | 2m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 25s | hbase-common: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) | | +1 :green_heart: | checkstyle | 1m 6s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 41m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 12c6a4e2a6b5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Apache-HBase commented on pull request #2556: URL: https://github.com/apache/hbase/pull/2556#issuecomment-713923552 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 56s | master passed | | +1 :green_heart: | compile | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 7m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 8m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 199m 19s | hbase-server in the patch passed. | | | | 232m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2556 | | JIRA Issue | HBASE-25193 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9c3b482d276d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/testReport/ | | Max. process+thread count | 3086 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509778357
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
+ @GuardedBy("configUpdateLock") private boolean splitEnabled;
+ @GuardedBy("configUpdateLock") private boolean mergeEnabled;
+ @GuardedBy("configUpdateLock") private int minRegionCount;
+ @GuardedBy("configUpdateLock") private Period mergeMinRegionAge;
+ @GuardedBy("configUpdateLock") private int mergeMinRegionSizeMb;
public SimpleRegionNormalizer() {
-splitEnabled = DEFAULT_SPLIT_ENABLED;
-mergeEnabled = DEFAULT_MERGE_ENABLED;
-minRegionCount = DEFAULT_MIN_REGION_COUNT;
-mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
-mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
+try {
+ splitEnabled = DEFAULT_SPLIT_ENABLED;
+ mergeEnabled = DEFAULT_MERGE_ENABLED;
+ minRegionCount = DEFAULT_MIN_REGION_COUNT;
+ mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
+ mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+} finally {
+ writeLock.unlock();
+}
}
@Override
public Configuration getConf() {
-return conf;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return conf;
+} finally {
+ readLock.unlock();
+}
}
@Override
public void setConf(final Configuration conf) {
if (conf == null) {
return;
}
-this.conf = conf;
-splitEnabled = conf.getBoolean(SPLIT_ENABLED_KEY, DEFAULT_SPLIT_ENABLED);
-mergeEnabled = conf.getBoolean(MERGE_ENABLED_KEY, DEFAULT_MERGE_ENABLED);
-minRegionCount = parseMinRegionCount(conf);
-mergeMinRegionAge = parseMergeMinRegionAge(conf);
-mergeMinRegionSizeMb = parseMergeMinRegionSizeMb(conf);
+
+final Lock readLock = configUpdateLock.readLock();
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock(); // "a writer can acquire the read lock, but not
vice-versa."
+readLock.lock();
Review comment:
Or I just did it wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509774334
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
+ @GuardedBy("configUpdateLock") private boolean splitEnabled;
+ @GuardedBy("configUpdateLock") private boolean mergeEnabled;
+ @GuardedBy("configUpdateLock") private int minRegionCount;
+ @GuardedBy("configUpdateLock") private Period mergeMinRegionAge;
+ @GuardedBy("configUpdateLock") private int mergeMinRegionSizeMb;
public SimpleRegionNormalizer() {
-splitEnabled = DEFAULT_SPLIT_ENABLED;
-mergeEnabled = DEFAULT_MERGE_ENABLED;
-minRegionCount = DEFAULT_MIN_REGION_COUNT;
-mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
-mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
+try {
+ splitEnabled = DEFAULT_SPLIT_ENABLED;
+ mergeEnabled = DEFAULT_MERGE_ENABLED;
+ minRegionCount = DEFAULT_MIN_REGION_COUNT;
+ mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
+ mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+} finally {
+ writeLock.unlock();
+}
}
@Override
public Configuration getConf() {
-return conf;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return conf;
+} finally {
+ readLock.unlock();
+}
}
@Override
public void setConf(final Configuration conf) {
if (conf == null) {
return;
}
-this.conf = conf;
-splitEnabled = conf.getBoolean(SPLIT_ENABLED_KEY, DEFAULT_SPLIT_ENABLED);
-mergeEnabled = conf.getBoolean(MERGE_ENABLED_KEY, DEFAULT_MERGE_ENABLED);
-minRegionCount = parseMinRegionCount(conf);
-mergeMinRegionAge = parseMergeMinRegionAge(conf);
-mergeMinRegionSizeMb = parseMergeMinRegionSizeMb(conf);
+
+final Lock readLock = configUpdateLock.readLock();
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock(); // "a writer can acquire the read lock, but not
vice-versa."
+readLock.lock();
Review comment:
But test output suggests otherwise :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509773168
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
+ @GuardedBy("configUpdateLock") private boolean splitEnabled;
+ @GuardedBy("configUpdateLock") private boolean mergeEnabled;
+ @GuardedBy("configUpdateLock") private int minRegionCount;
+ @GuardedBy("configUpdateLock") private Period mergeMinRegionAge;
+ @GuardedBy("configUpdateLock") private int mergeMinRegionSizeMb;
public SimpleRegionNormalizer() {
-splitEnabled = DEFAULT_SPLIT_ENABLED;
-mergeEnabled = DEFAULT_MERGE_ENABLED;
-minRegionCount = DEFAULT_MIN_REGION_COUNT;
-mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
-mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
+try {
+ splitEnabled = DEFAULT_SPLIT_ENABLED;
+ mergeEnabled = DEFAULT_MERGE_ENABLED;
+ minRegionCount = DEFAULT_MIN_REGION_COUNT;
+ mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
+ mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+} finally {
+ writeLock.unlock();
+}
}
@Override
public Configuration getConf() {
-return conf;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return conf;
+} finally {
+ readLock.unlock();
+}
}
@Override
public void setConf(final Configuration conf) {
if (conf == null) {
return;
}
-this.conf = conf;
-splitEnabled = conf.getBoolean(SPLIT_ENABLED_KEY, DEFAULT_SPLIT_ENABLED);
-mergeEnabled = conf.getBoolean(MERGE_ENABLED_KEY, DEFAULT_MERGE_ENABLED);
-minRegionCount = parseMinRegionCount(conf);
-mergeMinRegionAge = parseMergeMinRegionAge(conf);
-mergeMinRegionSizeMb = parseMergeMinRegionSizeMb(conf);
+
+final Lock readLock = configUpdateLock.readLock();
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock(); // "a writer can acquire the read lock, but not
vice-versa."
+readLock.lock();
Review comment:
To me, the next sentence, _This is what read-write locks allow: a
resource can be accessed by multiple readers or a single writer at a time, but
not both._ To me, that means that by taking a writeLock, the thread holding it
has exclusive access. writeLock should be enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
ndimiduk commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509749819
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java
##
@@ -0,0 +1,248 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM;
+import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM;
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+
+/**
+ * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica
load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up meta,
+ * it first check if the row is the stale location cache, if yes, this means
the the location from
+ * meta replica is stale, it will go to the primary meta to look up
update-to-date location;
+ * otherwise, it will randomly pick up a meta replica region for meta lookup.
When clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a chore.
+ */
+
+/**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+class StaleLocationCacheEntry {
+ // meta replica id where
+ private int metaReplicaId;
+
+ // timestamp in milliseconds
+ private long timestamp;
+
+ private byte[] endKey;
+
+ StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+this.metaReplicaId = metaReplicaId;
+this.endKey = endKey;
+timestamp = System.currentTimeMillis();
+ }
+
+ public byte[] getEndKey() {
+return this.endKey;
+ }
+
+ public int getMetaReplicaId() {
+return this.metaReplicaId;
+ }
+ public long getTimestamp() {
+return this.timestamp;
+ }
+
+ @Override
+ public String toString() {
+return new ToStringBuilder(this)
+ .append("endKey", endKey)
+ .append("metaReplicaId", metaReplicaId)
+ .append("timestamp", timestamp)
+ .toString();
+ }
+}
+
+/**
+ * A simple implementation of MetaReplicaLoadBalanceReplicaChooser.
+ *
+ * It follows a simple algorithm to choose a meta replica to go:
+ *
+ * 1. If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a meta replica region to do lookup.
+ * 2. If the location from meta replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * MetaReplicaLoadBalanceReplicaSimpleChooser.
+ * 3. When client tries to do meta lookup, it checks StaleLocationCache first
for rows it tries to
+ * lookup, if entry exists, it will go with primary meta region to do
lookup; otherwise, it
+ * will follow step 1.
+ * 4. A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ */
+class MetaReplicaLoadBalanceReplicaSimpleChooser implements
MetaReplicaLoadBalanceReplicaChooser {
+ private static final Logger LOG =
+LoggerFactory.getLogger(MetaReplicaLoadBalanceReplicaSimpleChooser.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL = 1500; // 1.5 seconds
+
+ private final class
[GitHub] [hbase] Apache-HBase commented on pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
Apache-HBase commented on pull request #2490: URL: https://github.com/apache/hbase/pull/2490#issuecomment-713906972 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | checkstyle | 1m 35s | master passed | | +1 :green_heart: | spotbugs | 2m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | checkstyle | 1m 34s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 45m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2490 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 9055bedec6e2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2490/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Apache-HBase commented on pull request #2556: URL: https://github.com/apache/hbase/pull/2556#issuecomment-713900751 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 150m 26s | hbase-server in the patch passed. | | | | 176m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2556 | | JIRA Issue | HBASE-25193 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bba5031066aa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/testReport/ | | Max. process+thread count | 3948 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] HorizonNet commented on a change in pull request #2558: HBASE-25174 Remove deprecated fields in HConstants
HorizonNet commented on a change in pull request #2558: URL: https://github.com/apache/hbase/pull/2558#discussion_r509734109 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -661,10 +660,8 @@ * Timestamp to use when we want to refer to the oldest cell. * Special! Used in fake Cells only. Should never be the timestamp on an actual Cell returned to * a client. - * @deprecated Should not be public since hbase-1.3.0. For internal use only. Move internal to - * Scanners flagged as special timestamp value never to be returned as timestamp on a Cell. */ - @Deprecated + @InterfaceAudience.Private public static final long OLDEST_TIMESTAMP = Long.MIN_VALUE; Review comment: Makes sense. I looked through the usages and there doesn't seem to be a common place. Do we have a place for kind of "private" constants? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] HorizonNet commented on a change in pull request #2562: HBASE-25199 Remove deprecated HStore#getStoreHomedir methods
HorizonNet commented on a change in pull request #2562:
URL: https://github.com/apache/hbase/pull/2562#discussion_r509731682
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -511,32 +511,6 @@ public OptionalLong getMaxMemStoreTS() {
return StoreUtils.getMaxMemStoreTSInList(this.getStorefiles());
}
- /**
- * @param tabledir {@link Path} to where the table is being stored
- * @param hri {@link RegionInfo} for the region.
- * @param family {@link ColumnFamilyDescriptor} describing the column family
- * @return Path to family/Store home directory.
- * @deprecated Since 05/05/2013, HBase-7808, hbase-1.0.0
- */
- @Deprecated
- public static Path getStoreHomedir(final Path tabledir,
- final RegionInfo hri, final byte[] family) {
-return getStoreHomedir(tabledir, hri.getEncodedName(), family);
- }
-
- /**
- * @param tabledir {@link Path} to where the table is being stored
- * @param encodedName Encoded region name.
- * @param family {@link ColumnFamilyDescriptor} describing the column family
- * @return Path to family/Store home directory.
- * @deprecated Since 05/05/2013, HBase-7808, hbase-1.0.0
- */
- @Deprecated
- public static Path getStoreHomedir(final Path tabledir,
Review comment:
Good point. I went through the original ticket again where the
deprecation happened. Seems like `HRegionFileSystem` is the intended place. Let
me try that one.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
ndimiduk commented on pull request #2490: URL: https://github.com/apache/hbase/pull/2490#issuecomment-713876958 Rebased. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
ndimiduk commented on a change in pull request #2490:
URL: https://github.com/apache/hbase/pull/2490#discussion_r509692111
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -315,35 +316,60 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
* towards target average or target region count.
*/
private List computeMergeNormalizationPlans(final
NormalizeContext ctx) {
-if (ctx.getTableRegions().size() < minRegionCount) {
+if (isEmpty(ctx.getTableRegions()) || ctx.getTableRegions().size() <
minRegionCount) {
LOG.debug("Table {} has {} regions, required min number of regions for
normalizer to run"
+ " is {}, not computing merge plans.", ctx.getTableName(),
ctx.getTableRegions().size(),
minRegionCount);
return Collections.emptyList();
}
-final double avgRegionSizeMb = ctx.getAverageRegionSizeMb();
+final long avgRegionSizeMb = (long) ctx.getAverageRegionSizeMb();
+if (avgRegionSizeMb < mergeMinRegionSizeMb) {
+ return Collections.emptyList();
+}
LOG.debug("Computing normalization plan for table {}. average region size:
{}, number of"
+ " regions: {}.", ctx.getTableName(), avgRegionSizeMb,
ctx.getTableRegions().size());
-final List plans = new ArrayList<>();
-for (int candidateIdx = 0; candidateIdx < ctx.getTableRegions().size() -
1; candidateIdx++) {
- final RegionInfo current = ctx.getTableRegions().get(candidateIdx);
- final RegionInfo next = ctx.getTableRegions().get(candidateIdx + 1);
- if (skipForMerge(ctx.getRegionStates(), current)
-|| skipForMerge(ctx.getRegionStates(), next)) {
-continue;
+// this nested loop walks the table's region chain once, looking for
contiguous sequences of
+// regions that meet the criteria for merge. The outer loop tracks the
starting point of the
+// next sequence, the inner loop looks for the end of that sequence. A
single sequence becomes
+// an instance of MergeNormalizationPlan.
+
+final List plans = new LinkedList<>();
+final List rangeMembers = new LinkedList<>();
+long sumRangeMembersSizeMb;
+int current = 0;
+for (int rangeStart = 0;
+ rangeStart < ctx.getTableRegions().size() - 1 && current <
ctx.getTableRegions().size();) {
+ // walk the region chain looking for contiguous sequences of regions
that can be merged.
+ rangeMembers.clear();
+ sumRangeMembersSizeMb = 0;
+ for (current = rangeStart; current < ctx.getTableRegions().size();
current++) {
+final RegionInfo regionInfo = ctx.getTableRegions().get(current);
+final long regionSizeMb = getRegionSizeMB(regionInfo);
+if (skipForMerge(ctx.getRegionStates(), regionInfo)) {
+ // this region cannot participate in a range. resume the outer loop.
+ rangeStart = Math.max(current, rangeStart + 1);
Review comment:
This line of questions convinces me you really read the code. Thank you
for the thoughtful review!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1531: HBASE-24200 Upgrade to Yetus 0.12.0
ndimiduk commented on pull request #1531: URL: https://github.com/apache/hbase/pull/1531#issuecomment-713873295 Removed change to `audience-annotations.version` in pom.xml This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] gvprathyusha6 commented on a change in pull request #2542: HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
gvprathyusha6 commented on a change in pull request #2542:
URL: https://github.com/apache/hbase/pull/2542#discussion_r509689995
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -466,13 +466,31 @@
*/
protected String useThisHostnameInstead;
+ /**
+ * @deprecated
+ * Use {@link
HRegionServer#UNSAFE_RS_HOSTNAME_DISABLE_MASTER_REVERSEDNS_KEY} instead.
+ */
+ @Deprecated
+ @InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.CONFIG)
+ final static String DEPRECATED_RS_HOSTNAME_DISABLE_MASTER_REVERSEDNS_KEY =
Review comment:
Have seen this kind of convention at couple of places in this repo -
https://github.com/apache/hbase/blob/01974385642ea5672542b1d4a22fcb54e8e2ba02/hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBuffAllocator.java#L81
https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/CompactionThroughputControllerFactory.java#L43
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
huaxiangsun commented on a change in pull request #2490:
URL: https://github.com/apache/hbase/pull/2490#discussion_r509649990
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -315,35 +316,60 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
* towards target average or target region count.
*/
private List computeMergeNormalizationPlans(final
NormalizeContext ctx) {
-if (ctx.getTableRegions().size() < minRegionCount) {
+if (isEmpty(ctx.getTableRegions()) || ctx.getTableRegions().size() <
minRegionCount) {
LOG.debug("Table {} has {} regions, required min number of regions for
normalizer to run"
+ " is {}, not computing merge plans.", ctx.getTableName(),
ctx.getTableRegions().size(),
minRegionCount);
return Collections.emptyList();
}
-final double avgRegionSizeMb = ctx.getAverageRegionSizeMb();
+final long avgRegionSizeMb = (long) ctx.getAverageRegionSizeMb();
+if (avgRegionSizeMb < mergeMinRegionSizeMb) {
+ return Collections.emptyList();
+}
LOG.debug("Computing normalization plan for table {}. average region size:
{}, number of"
+ " regions: {}.", ctx.getTableName(), avgRegionSizeMb,
ctx.getTableRegions().size());
-final List plans = new ArrayList<>();
-for (int candidateIdx = 0; candidateIdx < ctx.getTableRegions().size() -
1; candidateIdx++) {
- final RegionInfo current = ctx.getTableRegions().get(candidateIdx);
- final RegionInfo next = ctx.getTableRegions().get(candidateIdx + 1);
- if (skipForMerge(ctx.getRegionStates(), current)
-|| skipForMerge(ctx.getRegionStates(), next)) {
-continue;
+// this nested loop walks the table's region chain once, looking for
contiguous sequences of
+// regions that meet the criteria for merge. The outer loop tracks the
starting point of the
+// next sequence, the inner loop looks for the end of that sequence. A
single sequence becomes
+// an instance of MergeNormalizationPlan.
+
+final List plans = new LinkedList<>();
+final List rangeMembers = new LinkedList<>();
+long sumRangeMembersSizeMb;
+int current = 0;
+for (int rangeStart = 0;
+ rangeStart < ctx.getTableRegions().size() - 1 && current <
ctx.getTableRegions().size();) {
+ // walk the region chain looking for contiguous sequences of regions
that can be merged.
+ rangeMembers.clear();
+ sumRangeMembersSizeMb = 0;
+ for (current = rangeStart; current < ctx.getTableRegions().size();
current++) {
+final RegionInfo regionInfo = ctx.getTableRegions().get(current);
+final long regionSizeMb = getRegionSizeMB(regionInfo);
+if (skipForMerge(ctx.getRegionStates(), regionInfo)) {
+ // this region cannot participate in a range. resume the outer loop.
+ rangeStart = Math.max(current, rangeStart + 1);
Review comment:
Oh, nvm, it is taken care of, after the break, there is a check to see
if there are enough members which deserves a plan.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
huaxiangsun commented on a change in pull request #2490:
URL: https://github.com/apache/hbase/pull/2490#discussion_r509648091
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -315,35 +316,60 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
* towards target average or target region count.
*/
private List computeMergeNormalizationPlans(final
NormalizeContext ctx) {
-if (ctx.getTableRegions().size() < minRegionCount) {
+if (isEmpty(ctx.getTableRegions()) || ctx.getTableRegions().size() <
minRegionCount) {
LOG.debug("Table {} has {} regions, required min number of regions for
normalizer to run"
+ " is {}, not computing merge plans.", ctx.getTableName(),
ctx.getTableRegions().size(),
minRegionCount);
return Collections.emptyList();
}
-final double avgRegionSizeMb = ctx.getAverageRegionSizeMb();
+final long avgRegionSizeMb = (long) ctx.getAverageRegionSizeMb();
+if (avgRegionSizeMb < mergeMinRegionSizeMb) {
+ return Collections.emptyList();
+}
LOG.debug("Computing normalization plan for table {}. average region size:
{}, number of"
+ " regions: {}.", ctx.getTableName(), avgRegionSizeMb,
ctx.getTableRegions().size());
-final List plans = new ArrayList<>();
-for (int candidateIdx = 0; candidateIdx < ctx.getTableRegions().size() -
1; candidateIdx++) {
- final RegionInfo current = ctx.getTableRegions().get(candidateIdx);
- final RegionInfo next = ctx.getTableRegions().get(candidateIdx + 1);
- if (skipForMerge(ctx.getRegionStates(), current)
-|| skipForMerge(ctx.getRegionStates(), next)) {
-continue;
+// this nested loop walks the table's region chain once, looking for
contiguous sequences of
+// regions that meet the criteria for merge. The outer loop tracks the
starting point of the
+// next sequence, the inner loop looks for the end of that sequence. A
single sequence becomes
+// an instance of MergeNormalizationPlan.
+
+final List plans = new LinkedList<>();
+final List rangeMembers = new LinkedList<>();
+long sumRangeMembersSizeMb;
+int current = 0;
+for (int rangeStart = 0;
+ rangeStart < ctx.getTableRegions().size() - 1 && current <
ctx.getTableRegions().size();) {
+ // walk the region chain looking for contiguous sequences of regions
that can be merged.
+ rangeMembers.clear();
+ sumRangeMembersSizeMb = 0;
+ for (current = rangeStart; current < ctx.getTableRegions().size();
current++) {
+final RegionInfo regionInfo = ctx.getTableRegions().get(current);
+final long regionSizeMb = getRegionSizeMB(regionInfo);
+if (skipForMerge(ctx.getRegionStates(), regionInfo)) {
+ // this region cannot participate in a range. resume the outer loop.
+ rangeStart = Math.max(current, rangeStart + 1);
Review comment:
Meant for the case that region 1, 2, 3, 4, 5
Let say 1 and 2 are in rangeMembers, but 3 needs to be skipped. In this
case, 1,2 needs to be added to plans.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
huaxiangsun commented on a change in pull request #2490:
URL: https://github.com/apache/hbase/pull/2490#discussion_r509644400
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -315,35 +316,60 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
* towards target average or target region count.
*/
private List computeMergeNormalizationPlans(final
NormalizeContext ctx) {
-if (ctx.getTableRegions().size() < minRegionCount) {
+if (isEmpty(ctx.getTableRegions()) || ctx.getTableRegions().size() <
minRegionCount) {
LOG.debug("Table {} has {} regions, required min number of regions for
normalizer to run"
+ " is {}, not computing merge plans.", ctx.getTableName(),
ctx.getTableRegions().size(),
minRegionCount);
return Collections.emptyList();
}
-final double avgRegionSizeMb = ctx.getAverageRegionSizeMb();
+final long avgRegionSizeMb = (long) ctx.getAverageRegionSizeMb();
+if (avgRegionSizeMb < mergeMinRegionSizeMb) {
+ return Collections.emptyList();
+}
LOG.debug("Computing normalization plan for table {}. average region size:
{}, number of"
+ " regions: {}.", ctx.getTableName(), avgRegionSizeMb,
ctx.getTableRegions().size());
-final List plans = new ArrayList<>();
-for (int candidateIdx = 0; candidateIdx < ctx.getTableRegions().size() -
1; candidateIdx++) {
- final RegionInfo current = ctx.getTableRegions().get(candidateIdx);
- final RegionInfo next = ctx.getTableRegions().get(candidateIdx + 1);
- if (skipForMerge(ctx.getRegionStates(), current)
-|| skipForMerge(ctx.getRegionStates(), next)) {
-continue;
+// this nested loop walks the table's region chain once, looking for
contiguous sequences of
+// regions that meet the criteria for merge. The outer loop tracks the
starting point of the
+// next sequence, the inner loop looks for the end of that sequence. A
single sequence becomes
+// an instance of MergeNormalizationPlan.
+
+final List plans = new LinkedList<>();
+final List rangeMembers = new LinkedList<>();
+long sumRangeMembersSizeMb;
+int current = 0;
+for (int rangeStart = 0;
+ rangeStart < ctx.getTableRegions().size() - 1 && current <
ctx.getTableRegions().size();) {
+ // walk the region chain looking for contiguous sequences of regions
that can be merged.
+ rangeMembers.clear();
+ sumRangeMembersSizeMb = 0;
+ for (current = rangeStart; current < ctx.getTableRegions().size();
current++) {
+final RegionInfo regionInfo = ctx.getTableRegions().get(current);
+final long regionSizeMb = getRegionSizeMB(regionInfo);
+if (skipForMerge(ctx.getRegionStates(), regionInfo)) {
+ // this region cannot participate in a range. resume the outer loop.
+ rangeStart = Math.max(current, rangeStart + 1);
Review comment:
In this case, shall we check if there is already something in
rangeMembers and add it to the plan?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Apache-HBase commented on pull request #2556: URL: https://github.com/apache/hbase/pull/2556#issuecomment-713840444 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 4s | master passed | | +1 :green_heart: | checkstyle | 1m 18s | master passed | | +1 :green_heart: | spotbugs | 2m 54s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 21s | the patch passed | | +1 :green_heart: | checkstyle | 1m 25s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 26m 30s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 59m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2556 | | JIRA Issue | HBASE-25193 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 342948e7c316 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Max. process+thread count | 85 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on pull request #2570: URL: https://github.com/apache/hbase/pull/2570#issuecomment-713833194 Forgot to mention that patch against master, the only nit is testing cases as there is no meta replication support in master, will check the tests and see how much test can be ported. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1531: HBASE-24200 Upgrade to Yetus 0.12.0
Apache-HBase commented on pull request #1531: URL: https://github.com/apache/hbase/pull/1531#issuecomment-713832360 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 19s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | hadolint | 0m 1s | There were no new hadolint issues. | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 19m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 28m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1531/8/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1531 | | Optional Tests | dupname asflicense shellcheck shelldocs hadolint hadoopcheck xml | | uname | Linux c6891950f987 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Max. process+thread count | 63 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1531/8/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 hadolint=1.17.5-0-g443423c | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713831199 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | checkstyle | 1m 26s | master passed | | +1 :green_heart: | spotbugs | 2m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | -0 :warning: | checkstyle | 0m 11s | hbase-hadoop-compat: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -0 :warning: | checkstyle | 1m 15s | hbase-server: The patch generated 6 new + 283 unchanged - 0 fixed = 289 total (was 283) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 19m 22s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 45m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 5d7fb63a5259 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-general-check/output/diff-checkstyle-hbase-hadoop-compat.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713830268 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 33s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 7m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 7m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 39s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 10m 49s | hbase-server in the patch failed. | | | | 43m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b238a0ded1c5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/testReport/ | | Max. process+thread count | 600 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713829628 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 1m 29s | master passed | | +1 :green_heart: | shadedjars | 7m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 54s | the patch passed | | +1 :green_heart: | compile | 1m 22s | the patch passed | | +1 :green_heart: | javac | 1m 22s | the patch passed | | +1 :green_heart: | shadedjars | 7m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 37s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 11m 2s | hbase-server in the patch failed. | | | | 42m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6006c9b10aeb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/testReport/ | | Max. process+thread count | 464 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/14/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on pull request #2570: URL: https://github.com/apache/hbase/pull/2570#issuecomment-713829326 > On a high level design, we used to have a `hbase.meta.replicas.use` config to control whether to make use meta replicas at client side. Do we want to deprecate this config and let our new configs rule here? Just asking. > For me, original thought is that `hbase.meta.replicas.use` is an existing config, so it still needs to be honored. A new "mode" config is added to support LoadBalance(new) and HighlyAvailable(the existing mode). If we want to deprecate this config, then, it makes sense. > For me, I think the scope of `hbase.meta.replicas.use` is too wide, as it will impact all the client operations on meta(not sure if we have hacked all the places but at least it is designed to). After reviewing the PR, I think our approach here is only for the locator related logic right? This is also what I expect. In general, I think there are 3 ways we access meta table at client side: > > 1. Locator related logic. This is the most critical path at client side and also makes the most pressure on meta table. > 2. Admin related logic. We have delegated most of the operaations through master but there are still some places we will access meta directly. But admin operation is expected to be low frequent so I do not think we need to deal with it here. > 3. Users access meta table directly by their own. This is controlled by user written code so I do not think we need to deal with it either, users should take care of it by their own. Agreed. > > I think only 1 is what we really care here, so I suggest that, we just narrow the scope of the newly introduced configs to be locator only(maybe by adding 'locator' keyword in the config name), and consider it first before `hbase.meta.replicas.use` in the locator related logic. So users do not need to set `hbase.meta.replicas.use` to true to enable this feature, just select the LB mode. If the new config is disabled(set to None maybe), then we honor the old `hbase.meta.replicas.use` config. Ok. > > In general, I think the abstraction and trick here are both good. Setting the replica id directly on Query is a straight forward way to archive our goal here, and the chooser or selector is also a good abstraction. The only concern is how to implement the 'fallback to primary' logic as we need to pass down from the rpc retrying caller of the actual exception type, anyway, this can be done later. Do you think "fallback to primary" logic needs to be passed down from rpc retrying caller? Then it needs to be aware of this feature and needs to maintain some state. Was trying to avoid it. Yeah, this part can be optimized later. > > And I suggest we just make this PR against master branch, it is only client side change and whether we have implement the meta region replication should not impact the code. Why I suggest this is that, the code for master and branch-2 will be different, as on branch-2, the sync client has its own logic to do region locating, which is not well constructed I believe(we expose a bunch of locating methods in ClusterConnection interface and use it everywhere). So if we want to include this feature in 2.4.0, we'd better make this PR against master, and also backport it to branch-2 ASAP. Yeah, the client side change is quite independent. It will work with today's meta replica, only nit is that there will be more stale data. I will put a new patch against master. All implementation will be renamed as Catalog instead of meta, as it can be applied to similar location lookup service. The config will remain same as here it applies to meta only. Thanks for the feedbacks, Duo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713829224 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 32s | master passed | | +1 :green_heart: | spotbugs | 2m 43s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | checkstyle | 0m 23s | hbase-common: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) | | +1 :green_heart: | checkstyle | 1m 8s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 15s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 42m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 1d9f47f0c98f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713827334 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 1m 31s | master passed | | +1 :green_heart: | shadedjars | 6m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 44s | hbase-common in the patch passed. | | -1 :x: | unit | 7m 34s | hbase-server in the patch failed. | | | | 38m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 17243e6fb99f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/testReport/ | | Max. process+thread count | 729 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713826510 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | compile | 1m 23s | master passed | | +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 32s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 6m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 32s | hbase-common in the patch passed. | | -1 :x: | unit | 7m 18s | hbase-server in the patch failed. | | | | 36m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 20d074584494 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/testReport/ | | Max. process+thread count | 768 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509607895
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
##
@@ -130,8 +133,12 @@ public AsyncConnectionImpl(Configuration conf,
ConnectionRegistry registry, Stri
SocketAddress localAddress, User user) {
this.conf = conf;
this.user = user;
-if (user.isLoginFromKeytab()) {
- spawnRenewalChore(user.getUGI());
+
+if (user.isLoginFromKeytab() || isMetaReplicaLBMode()) {
Review comment:
Yeah, trying to convert it to guava cache, which seems light weight.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509607454
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java
##
@@ -0,0 +1,248 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM;
+import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM;
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+
+/**
+ * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica
load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up meta,
+ * it first check if the row is the stale location cache, if yes, this means
the the location from
+ * meta replica is stale, it will go to the primary meta to look up
update-to-date location;
+ * otherwise, it will randomly pick up a meta replica region for meta lookup.
When clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a chore.
+ */
+
+/**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+class StaleLocationCacheEntry {
+ // meta replica id where
+ private int metaReplicaId;
+
+ // timestamp in milliseconds
+ private long timestamp;
+
+ private byte[] endKey;
+
+ StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+this.metaReplicaId = metaReplicaId;
+this.endKey = endKey;
+timestamp = System.currentTimeMillis();
+ }
+
+ public byte[] getEndKey() {
+return this.endKey;
+ }
+
+ public int getMetaReplicaId() {
+return this.metaReplicaId;
+ }
+ public long getTimestamp() {
+return this.timestamp;
+ }
+
+ @Override
+ public String toString() {
+return new ToStringBuilder(this)
+ .append("endKey", endKey)
+ .append("metaReplicaId", metaReplicaId)
+ .append("timestamp", timestamp)
+ .toString();
+ }
+}
+
+/**
+ * A simple implementation of MetaReplicaLoadBalanceReplicaChooser.
+ *
+ * It follows a simple algorithm to choose a meta replica to go:
+ *
+ * 1. If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a meta replica region to do lookup.
+ * 2. If the location from meta replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * MetaReplicaLoadBalanceReplicaSimpleChooser.
+ * 3. When client tries to do meta lookup, it checks StaleLocationCache first
for rows it tries to
+ * lookup, if entry exists, it will go with primary meta region to do
lookup; otherwise, it
+ * will follow step 1.
+ * 4. A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ */
+class MetaReplicaLoadBalanceReplicaSimpleChooser implements
MetaReplicaLoadBalanceReplicaChooser {
Review comment:
For a normal case, > 99% of region locations from meta replica will be
up to date. In extreme cases such as region server crash, it will depends on
how fast replication catches up. Will try to add more tests to find out.
[GitHub] [hbase] busbey commented on a change in pull request #1531: HBASE-24200 Upgrade to Yetus 0.12.0
busbey commented on a change in pull request #1531: URL: https://github.com/apache/hbase/pull/1531#discussion_r509605959 ## File path: pom.xml ## @@ -1593,7 +1593,7 @@ comes in via hbase-thirdparty hbase-shaded-netty--> 3.10.5.Final -0.5.0 +0.12.0 Review comment: IIRC this was being left alone out of an abundance of caution since it is exposed to our downstream client users. Since test infra changes need to go to all branches could we break it into a follow on so we can better target specific branches and release note it separately? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509605617
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java
##
@@ -0,0 +1,248 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM;
+import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM;
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+
+/**
+ * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica
load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up meta,
+ * it first check if the row is the stale location cache, if yes, this means
the the location from
+ * meta replica is stale, it will go to the primary meta to look up
update-to-date location;
+ * otherwise, it will randomly pick up a meta replica region for meta lookup.
When clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a chore.
+ */
+
+/**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+class StaleLocationCacheEntry {
+ // meta replica id where
+ private int metaReplicaId;
+
+ // timestamp in milliseconds
+ private long timestamp;
+
+ private byte[] endKey;
+
+ StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+this.metaReplicaId = metaReplicaId;
+this.endKey = endKey;
+timestamp = System.currentTimeMillis();
+ }
+
+ public byte[] getEndKey() {
+return this.endKey;
+ }
+
+ public int getMetaReplicaId() {
+return this.metaReplicaId;
+ }
+ public long getTimestamp() {
+return this.timestamp;
+ }
+
+ @Override
+ public String toString() {
+return new ToStringBuilder(this)
+ .append("endKey", endKey)
+ .append("metaReplicaId", metaReplicaId)
+ .append("timestamp", timestamp)
+ .toString();
+ }
+}
+
+/**
+ * A simple implementation of MetaReplicaLoadBalanceReplicaChooser.
+ *
+ * It follows a simple algorithm to choose a meta replica to go:
+ *
+ * 1. If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a meta replica region to do lookup.
+ * 2. If the location from meta replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * MetaReplicaLoadBalanceReplicaSimpleChooser.
+ * 3. When client tries to do meta lookup, it checks StaleLocationCache first
for rows it tries to
+ * lookup, if entry exists, it will go with primary meta region to do
lookup; otherwise, it
+ * will follow step 1.
+ * 4. A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ */
+class MetaReplicaLoadBalanceReplicaSimpleChooser implements
MetaReplicaLoadBalanceReplicaChooser {
Review comment:
Our original plan is to come up with a simple algorithm first.
1. Select a meta replica region first for lookup, if the data is stale, then
go to primary.
Later, maybe we can experiment some smarter way about how to select the next
meta replica in case of
[GitHub] [hbase] virajjasani commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
virajjasani commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509599484
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
+ @GuardedBy("configUpdateLock") private boolean splitEnabled;
+ @GuardedBy("configUpdateLock") private boolean mergeEnabled;
+ @GuardedBy("configUpdateLock") private int minRegionCount;
+ @GuardedBy("configUpdateLock") private Period mergeMinRegionAge;
+ @GuardedBy("configUpdateLock") private int mergeMinRegionSizeMb;
public SimpleRegionNormalizer() {
-splitEnabled = DEFAULT_SPLIT_ENABLED;
-mergeEnabled = DEFAULT_MERGE_ENABLED;
-minRegionCount = DEFAULT_MIN_REGION_COUNT;
-mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
-mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
+try {
+ splitEnabled = DEFAULT_SPLIT_ENABLED;
+ mergeEnabled = DEFAULT_MERGE_ENABLED;
+ minRegionCount = DEFAULT_MIN_REGION_COUNT;
+ mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
+ mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+} finally {
+ writeLock.unlock();
+}
}
@Override
public Configuration getConf() {
-return conf;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return conf;
+} finally {
+ readLock.unlock();
+}
}
@Override
public void setConf(final Configuration conf) {
if (conf == null) {
return;
}
-this.conf = conf;
-splitEnabled = conf.getBoolean(SPLIT_ENABLED_KEY, DEFAULT_SPLIT_ENABLED);
-mergeEnabled = conf.getBoolean(MERGE_ENABLED_KEY, DEFAULT_MERGE_ENABLED);
-minRegionCount = parseMinRegionCount(conf);
-mergeMinRegionAge = parseMergeMinRegionAge(conf);
-mergeMinRegionSizeMb = parseMergeMinRegionSizeMb(conf);
+
+final Lock readLock = configUpdateLock.readLock();
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock(); // "a writer can acquire the read lock, but not
vice-versa."
+readLock.lock();
Review comment:
> As long as each thread is guaranteed an up-to-date view of the data
and no other thread modifies the data while the readers are viewing it, there
will be no problems.
Do we not need `readLock()` + `writeLock()` to achieve this? How can we
achieve writers not updating while readers are reading conf?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-23894) [JDK11] build with -PerrorProne silently fails
[
https://issues.apache.org/jira/browse/HBASE-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-23894.
--
Resolution: Won't Fix
I don't think we can solve this. ErrorProne's maven configurations are
incompatible between JDK8 and JDK11, and given that we gate it by a profile,
and maven's lack of expression when it comes to profile combinations, I don't
see how to implement this.
I think we're stuck with running ErrorProne on a single JVM version target for
now.
> [JDK11] build with -PerrorProne silently fails
> --
>
> Key: HBASE-23894
> URL: https://issues.apache.org/jira/browse/HBASE-23894
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Attachments: patch-compile-root.txt
>
>
> Build seems to simply stop at {{hbase-protocol-shaded}} without explanation.
> Dropping {{-PerrorProne}} allows the build to proceed, at least locally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-23894) [JDK11] build with -PerrorProne silently fails
[
https://issues.apache.org/jira/browse/HBASE-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reassigned HBASE-23894:
Assignee: (was: Nick Dimiduk)
> [JDK11] build with -PerrorProne silently fails
> --
>
> Key: HBASE-23894
> URL: https://issues.apache.org/jira/browse/HBASE-23894
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: Nick Dimiduk
>Priority: Major
> Attachments: patch-compile-root.txt
>
>
> Build seems to simply stop at {{hbase-protocol-shaded}} without explanation.
> Dropping {{-PerrorProne}} allows the build to proceed, at least locally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r509598756
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with meta replica support.
+ * HighAvailable- Client sends requests to the primary meta region
first, within a
+ * configured amount of time, if there is no response
coming back,
+ * client sends requests to all replica regions and takes
the first
+ * response.
+ *
+ * LoadBalance - Client sends requests to meta replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stable locations to the primary meta region. In
this mode, scan
+ * requests are load balanced across all replica regions.
+ */
+enum MetaReplicaMode {
+ None,
+ HighAvailable,
Review comment:
Yeah, will do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1547: HBASE-23894 Support ErrorProne for both JDK8 and JDK11
ndimiduk commented on pull request #1547: URL: https://github.com/apache/hbase/pull/1547#issuecomment-713814829 I don't think this is feasible; given how we mask its use behind a profile, we're going to have to run ErrorProne on only one JVM configuration. Closing old/stale PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk closed pull request #1547: HBASE-23894 Support ErrorProne for both JDK8 and JDK11
ndimiduk closed pull request #1547: URL: https://github.com/apache/hbase/pull/1547 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #2490: HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
ndimiduk commented on pull request #2490: URL: https://github.com/apache/hbase/pull/2490#issuecomment-713813965 Bump. The last build-bot run was all greens, I'm not sure why the GitHub status didn't update. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25193) Add support for row prefix and type in the WAL Pretty Printer
[ https://issues.apache.org/jira/browse/HBASE-25193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated HBASE-25193: Description: Currently, the WAL Pretty Printer has an option to filter the keys with an exact match of row. However, it is super useful sometimes to have a row key prefix instead of an exact match. The prefix can act as a full match filter as well due to the nature of the prefix. Secondly, we are not having the cell type in the WAL Pretty Printer in any of the branches. Lastly, the option rowkey only options prints additional stuff as well. was: Currently, the WAL Pretty Printer has an option to filter the keys with an exact match of row. However, it is super useful sometimes to have a row key prefix instead of an exact match. The prefix can act as a full match filter as well due to the nature of the prefix. Additionally, we are not having the cell type in the WAL Pretty Printer in any of the branches. > Add support for row prefix and type in the WAL Pretty Printer > - > > Key: HBASE-25193 > URL: https://issues.apache.org/jira/browse/HBASE-25193 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Minor > > Currently, the WAL Pretty Printer has an option to filter the keys with an > exact match of row. However, it is super useful sometimes to have a row key > prefix instead of an exact match. > The prefix can act as a full match filter as well due to the nature of the > prefix. > Secondly, we are not having the cell type in the WAL Pretty Printer in any of > the branches. > Lastly, the option rowkey only options prints additional stuff as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25193) Add support for row prefix and type in the WAL Pretty Printer and some minor fixes
[ https://issues.apache.org/jira/browse/HBASE-25193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Pal updated HBASE-25193: Summary: Add support for row prefix and type in the WAL Pretty Printer and some minor fixes (was: Add support for row prefix and type in the WAL Pretty Printer) > Add support for row prefix and type in the WAL Pretty Printer and some minor > fixes > -- > > Key: HBASE-25193 > URL: https://issues.apache.org/jira/browse/HBASE-25193 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Minor > > Currently, the WAL Pretty Printer has an option to filter the keys with an > exact match of row. However, it is super useful sometimes to have a row key > prefix instead of an exact match. > The prefix can act as a full match filter as well due to the nature of the > prefix. > Secondly, we are not having the cell type in the WAL Pretty Printer in any of > the branches. > Lastly, the option rowkey only options prints additional stuff as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713779840 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 19s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 48s | master passed | | +1 :green_heart: | compile | 1m 31s | master passed | | +1 :green_heart: | shadedjars | 7m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 2m 11s | root in the patch failed. | | -1 :x: | compile | 0m 29s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 29s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 19s | patch has 14 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 41s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 0m 29s | hbase-server in the patch failed. | | | | 27m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1e7b59684ffb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 2020-01-14 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/testReport/ | | Max. process+thread count | 297 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713778287 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 7m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 50s | root in the patch failed. | | -1 :x: | compile | 0m 26s | hbase-server in the patch failed. | | -0 :warning: | javac | 0m 26s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 5m 11s | patch has 14 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 35s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 0m 27s | hbase-server in the patch failed. | | | | 25m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 639b647289bc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk8-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/testReport/ | | Max. process+thread count | 318 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509554540
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
Review comment:
It seems SpotBugs does have a bug description for the improper use of
`GuardedBy`,
https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html#is-field-not-guarded-against-concurrent-access-is-field-not-guarded.
I wonder if it's actually implemented ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713776373 Rebase and remove taking the ReadLock from `SimpleRegionNormalizer#setConf`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713776445 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 54s | master passed | | +1 :green_heart: | checkstyle | 1m 26s | master passed | | +1 :green_heart: | spotbugs | 2m 36s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 46s | root in the patch failed. | | -0 :warning: | checkstyle | 0m 10s | hbase-hadoop-compat: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -0 :warning: | checkstyle | 1m 14s | hbase-server: The patch generated 6 new + 283 unchanged - 0 fixed = 289 total (was 283) | | -0 :warning: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 :x: | hadoopcheck | 1m 55s | The patch causes 14 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 3m 53s | The patch causes 14 errors with Hadoop v3.2.1. | | -1 :x: | hadoopcheck | 5m 49s | The patch causes 14 errors with Hadoop v3.3.0. | | -1 :x: | spotbugs | 0m 25s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 18s | The patch does not generate ASF License warnings. | | | | 21m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ce89f8f62b1c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/diff-checkstyle-hbase-hadoop-compat.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/whitespace-eol.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/patch-javac-3.3.0.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/13/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
sandeepvinayak commented on a change in pull request #2556:
URL: https://github.com/apache/hbase/pull/2556#discussion_r509548831
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
##
@@ -301,15 +318,12 @@ public void processFile(final Configuration conf, final
Path p)
List> actions = new ArrayList<>();
for (Cell cell : edit.getCells()) {
// add atomic operation to txn
- Map op = new HashMap<>(toStringMap(cell,
outputOnlyRowKey));
- if (outputValues) {
-op.put("value", Bytes.toStringBinary(CellUtil.cloneValue(cell)));
- }
- // check row output filter
- if (row == null || ((String) op.get("row")).equals(row)) {
Review comment:
@Apache9 Yes, initially I thought the same way and did that. But
@wchevreuil brings the valid concern that rowprefix filter and row filter
should both be available.
Example from @wchevreuil :
If there are rows `a`, `aa`, `aaa` and user just wants row `a`, the
rowPrefix will not help in this case. Hope it answers your concern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
sandeepvinayak commented on a change in pull request #2556:
URL: https://github.com/apache/hbase/pull/2556#discussion_r509545829
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java
##
@@ -326,15 +340,19 @@ public void processFile(final Configuration conf, final
Path p)
out.print(GSON.toJson(txn));
} else {
// Pretty output, complete with indentation by atomic action
- out.println(String.format(outputTmpl,
+ if (!outputOnlyRowKey) {
Review comment:
yes, I should mention it in jira and I will add it, this option
`outputOnlyRowKey` was to print only rowkey but we were still printing
additional stuff.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509538496
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
+ @GuardedBy("configUpdateLock") private boolean splitEnabled;
+ @GuardedBy("configUpdateLock") private boolean mergeEnabled;
+ @GuardedBy("configUpdateLock") private int minRegionCount;
+ @GuardedBy("configUpdateLock") private Period mergeMinRegionAge;
+ @GuardedBy("configUpdateLock") private int mergeMinRegionSizeMb;
public SimpleRegionNormalizer() {
-splitEnabled = DEFAULT_SPLIT_ENABLED;
-mergeEnabled = DEFAULT_MERGE_ENABLED;
-minRegionCount = DEFAULT_MIN_REGION_COUNT;
-mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
-mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
+try {
+ splitEnabled = DEFAULT_SPLIT_ENABLED;
+ mergeEnabled = DEFAULT_MERGE_ENABLED;
+ minRegionCount = DEFAULT_MIN_REGION_COUNT;
+ mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
+ mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+} finally {
+ writeLock.unlock();
+}
}
@Override
public Configuration getConf() {
-return conf;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return conf;
+} finally {
+ readLock.unlock();
+}
}
@Override
public void setConf(final Configuration conf) {
if (conf == null) {
return;
}
-this.conf = conf;
-splitEnabled = conf.getBoolean(SPLIT_ENABLED_KEY, DEFAULT_SPLIT_ENABLED);
-mergeEnabled = conf.getBoolean(MERGE_ENABLED_KEY, DEFAULT_MERGE_ENABLED);
-minRegionCount = parseMinRegionCount(conf);
-mergeMinRegionAge = parseMergeMinRegionAge(conf);
-mergeMinRegionSizeMb = parseMergeMinRegionSizeMb(conf);
+
+final Lock readLock = configUpdateLock.readLock();
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock(); // "a writer can acquire the read lock, but not
vice-versa."
+readLock.lock();
Review comment:
> Why is readLock.lock() called here? Thought writeLock is enough.
Initially I thought so too, but reading the docs on `ReadWriteLock`, it
never explicitly said that the `WriteLock` is exclusive for both readers and
writers, it only says that it's exclusive. The best I can find is from _Java
Concurrency in Practice_, which says (emphasis mine):
> ... Mutual exclusion is a conservative locking strategy that prevents
writer/writer and writer/reader overlap, but also prevents reader/reader
overlap. In many cases, data structures are “read-mostly”—they are mutable and
are sometimes modified, but most accesses involve only reading. In these cases,
it would be nice to relax the locking requirements to allow multiple readers to
access the data structure at once. As long as each thread is guaranteed an
up-to-date view of the data and no other thread modifies the data while the
readers are viewing it, there will be no problems. **This is what read-write
locks allow: a resource can be accessed by multiple readers or a single writer
at a time, but not both.**
So I think you're correct @huaxiangsun , the write lock should be sufficient.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713759128 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 20s | master passed | | +1 :green_heart: | compile | 1m 36s | master passed | | +1 :green_heart: | shadedjars | 8m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 2m 41s | root in the patch failed. | | -1 :x: | compile | 1m 8s | hbase-server in the patch failed. | | -0 :warning: | javac | 1m 8s | hbase-server in the patch failed. | | -1 :x: | shadedjars | 6m 16s | patch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 43s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 1m 7s | hbase-server in the patch failed. | | | | 33m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 603017ffe32d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2cd5f914af | | Default Java | 1.8.0_232 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/testReport/ | | Max. process+thread count | 315 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/12/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2374: HBASE-25003 Backport HBASE-24350 and HBASE-24779 to branch-2.3
Apache-HBase commented on pull request #2374: URL: https://github.com/apache/hbase/pull/2374#issuecomment-71371 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 56s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 28s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 4m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 8s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | +1 :green_heart: | shadedjars | 5m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 25s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 0m 27s | hbase-hadoop2-compat in the patch passed. | | +1 :green_heart: | unit | 141m 47s | hbase-server in the patch passed. | | | | 169m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2374/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2374 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9e48209051c4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 7c4c3d0685 | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2374/3/testReport/ | | Max. process+thread count | 3859 (vs. ulimit of 12500) | | modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2374/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
