[GitHub] [hbase] Apache-HBase commented on pull request #3362: HBASE-22708 Remove the deprecated methods in Hbck interface
Apache-HBase commented on pull request #3362: URL: https://github.com/apache/hbase/pull/3362#issuecomment-855678626 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +1 :green_heart: | compile | 4m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 30s | master passed | | +1 :green_heart: | spotbugs | 3m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 4m 4s | the patch passed | | +1 :green_heart: | javac | 4m 4s | the patch passed | | +1 :green_heart: | checkstyle | 1m 29s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 14s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 52m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3362/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3362 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux aedf5aa18fd9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 456c7f964a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3362/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3321: HBASE-25914 Provide slow/large logs on RegionServer UI
Apache-HBase commented on pull request #3321: URL: https://github.com/apache/hbase/pull/3321#issuecomment-855742462 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 42s | branch-2 passed | | -0 :warning: | patch | 5m 14s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 134m 18s | hbase-server in the patch passed. | | | | 146m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3321 | | Optional Tests | javac javadoc unit | | uname | Linux ef6ddc2b3cd7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e4dd4901a8 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/3/testReport/ | | Max. process+thread count | 3963 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache-HBase commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-855755958 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 21s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | master passed | | +1 :green_heart: | compile | 2m 57s | master passed | | +1 :green_heart: | shadedjars | 8m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 54s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 2m 58s | the patch passed | | +1 :green_heart: | javac | 2m 58s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 188m 6s | root in the patch passed. | | | | 230m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3317 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1355e9ea37d0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 456c7f964a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/2/testReport/ | | Max. process+thread count | 6399 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25963) HBaseCluster should be marked as IA.Public
[
https://issues.apache.org/jira/browse/HBASE-25963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358485#comment-17358485
]
Hudson commented on HBASE-25963:
Results for branch branch-2.3
[build #232 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/232/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/232/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/232/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/232/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/232/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/232//console].
> HBaseCluster should be marked as IA.Public
> --
>
> Key: HBASE-25963
> URL: https://issues.apache.org/jira/browse/HBASE-25963
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> Its sub class MiniHBaseCluster is marked as IA.Public, so it should also be
> IA.Public.
> And in general, HBaseCluster just provides some abstract method so I think it
> is OK to expose it as IA.Public.
> But for MiniHBaseCluster, it exposes several internal stuff such as
> RegionServerThread so maybe it should not be marked as IA.Public.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3321: HBASE-25914 Provide slow/large logs on RegionServer UI
Apache-HBase commented on pull request #3321: URL: https://github.com/apache/hbase/pull/3321#issuecomment-855795847 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | | -0 :warning: | patch | 4m 51s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 208m 31s | hbase-server in the patch passed. | | | | 220m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3321 | | Optional Tests | javac javadoc unit | | uname | Linux 0aa207343dc6 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e4dd4901a8 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/3/testReport/ | | Max. process+thread count | 2743 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3321: HBASE-25914 Provide slow/large logs on RegionServer UI
Apache-HBase commented on pull request #3321: URL: https://github.com/apache/hbase/pull/3321#issuecomment-855804302 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | branch-2 passed | | -0 :warning: | patch | 4m 1s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 10m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3321 | | Optional Tests | dupname asflicense javac | | uname | Linux 418da6722414 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e4dd4901a8 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 65 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3362: HBASE-22708 Remove the deprecated methods in Hbck interface
Apache-HBase commented on pull request #3362: URL: https://github.com/apache/hbase/pull/3362#issuecomment-855823793 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 1m 27s | master passed | | +1 :green_heart: | shadedjars | 8m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 29s | the patch passed | | +1 :green_heart: | javac | 1m 29s | the patch passed | | +1 :green_heart: | shadedjars | 9m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 29s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 210m 30s | hbase-server in the patch passed. | | | | 246m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3362/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3362 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0a81736ae05d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 456c7f964a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3362/2/testReport/ | | Max. process+thread count | 2797 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3362/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache-HBase commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-855862702 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 2m 42s | master passed | | +1 :green_heart: | shadedjars | 8m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 20s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | compile | 2m 38s | the patch passed | | +1 :green_heart: | javac | 2m 38s | the patch passed | | +1 :green_heart: | shadedjars | 8m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 20s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 358m 1s | root in the patch failed. | | | | 396m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3317 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1a26234eaa71 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 456c7f964a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/2/testReport/ | | Max. process+thread count | 5606 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-13126) Provide alternate mini cluster classes other than HBTU for downstream users to write unit tests
[ https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358554#comment-17358554 ] Sean Busbey commented on HBASE-13126: - Yeah that sounds good. > Provide alternate mini cluster classes other than HBTU for downstream users > to write unit tests > --- > > Key: HBASE-13126 > URL: https://issues.apache.org/jira/browse/HBASE-13126 > Project: HBase > Issue Type: Umbrella > Components: API >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU > methods wasn't intended for public consumption. > Can we build a list of such methods across the API, appropriately annotate > them for 2.0, and deprecate them in earlier versions with a warning that > they're going to be restricted? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3321: HBASE-25914 Provide slow/large logs on RegionServer UI
Apache-HBase commented on pull request #3321: URL: https://github.com/apache/hbase/pull/3321#issuecomment-855887742 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 43s | branch-2 passed | | -0 :warning: | patch | 5m 16s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 133m 36s | hbase-server in the patch passed. | | | | 146m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3321 | | Optional Tests | javac javadoc unit | | uname | Linux e6688fd02684 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e4dd4901a8 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/testReport/ | | Max. process+thread count | 4036 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-filesystem] steveloughran opened a new pull request #23: HBASE-25900. HBoss tests compile/failure against Hadoop 3.3.1
steveloughran opened a new pull request #23: URL: https://github.com/apache/hbase-filesystem/pull/23 Fixes the build to * optionally look in the ASF snapshots and staging repos * build with hadoop-3.3.1 * tests to work with hadoop-3.3.1 * assertJ added to the classpath (something is odd with hadoop-common-test here; seen elsewhere. Will need to revisit). This fixes the s3 client factory to use the new parameter object to configure the client. This will allow the hadoop code to add new options (it already has related to regions), without HBoss builds failing. We've also tagged the API as limited private and mentioned HBoss as users of it, so future maintainers will know to be careful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-filesystem] steveloughran commented on pull request #23: HBASE-25900. HBoss tests compile/failure against Hadoop 3.3.1
steveloughran commented on pull request #23: URL: https://github.com/apache/hbase-filesystem/pull/23#issuecomment-855897275 Note: this PR fixes things to build, but only against 3.3.1, which is an RC. So: not ready for merge. And it doesn't have a story for the fact the changes won't build on hadoop < 3.3.1. Really 1. there will need to be separate source branches for the mock clients; the rest of the test suite can be shared. 2. changes in the s3a semantics related to rename() mean that there'll need to be a different s3a.xml too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache-HBase commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-855937680 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 8m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 8m 37s | the patch passed | | -0 :warning: | javac | 8m 37s | root generated 1 new + 1547 unchanged - 1 fixed = 1548 total (was 1548) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 20m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 54m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3317 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux e74db3dbea1a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 456c7f964a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | Max. process+thread count | 126 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25963) HBaseCluster should be marked as IA.Public
[
https://issues.apache.org/jira/browse/HBASE-25963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358609#comment-17358609
]
Hudson commented on HBASE-25963:
Results for branch master
[build #317 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBaseCluster should be marked as IA.Public
> --
>
> Key: HBASE-25963
> URL: https://issues.apache.org/jira/browse/HBASE-25963
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> Its sub class MiniHBaseCluster is marked as IA.Public, so it should also be
> IA.Public.
> And in general, HBaseCluster just provides some abstract method so I think it
> is OK to expose it as IA.Public.
> But for MiniHBaseCluster, it exposes several internal stuff such as
> RegionServerThread so maybe it should not be marked as IA.Public.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25977) Remove 2.2.7 from download page
[
https://issues.apache.org/jira/browse/HBASE-25977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358610#comment-17358610
]
Hudson commented on HBASE-25977:
Results for branch master
[build #317 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/317/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove 2.2.7 from download page
> ---
>
> Key: HBASE-25977
> URL: https://issues.apache.org/jira/browse/HBASE-25977
> Project: HBase
> Issue Type: Sub-task
> Components: website
>Reporter: Duo Zhang
>Assignee: Zhuoyue Huang
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> I think it is time to do this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3321: HBASE-25914 Provide slow/large logs on RegionServer UI
Apache-HBase commented on pull request #3321: URL: https://github.com/apache/hbase/pull/3321#issuecomment-855948130 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 36s | branch-2 passed | | -0 :warning: | patch | 4m 42s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 209m 38s | hbase-server in the patch passed. | | | | 221m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3321 | | Optional Tests | javac javadoc unit | | uname | Linux c407097296d7 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e4dd4901a8 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/testReport/ | | Max. process+thread count | 2336 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3321/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-25973) Balancer should explain progress in a better way in log
[ https://issues.apache.org/jira/browse/HBASE-25973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-25973: --- Assignee: Clara Xiong > Balancer should explain progress in a better way in log > --- > > Key: HBASE-25973 > URL: https://issues.apache.org/jira/browse/HBASE-25973 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 3.0.0-alpha-1 >Reporter: Clara Xiong >Assignee: Clara Xiong >Priority: Major > > In the log, balancer logs at info level at the beginning of run: > > balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer, > initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction : > (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0, > 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0); > TableSkewCostFunction : (1000.0, 0.0019704142954972883); > StoreFileCostFunction : (200.0, 0.3668512059459341); computedMaxSteps: > 42270438200 > the cost is reported without context, it is hard for operator to understand > how unbalanced the cluster is for balancer and how much progress we are > making. > For a large cluster, the calculation can take a long time, we also need to > let operator understand that it will take up to the max time to complete the > calculation. > At the end of computation: > balancer.StochasticLoadBalancer: Finished computing new load balance plan. > Computation took PT40M0.006S to try 1036409 different iterations. Found a > solution that moves 161926 regions; Going from a computed cost of > 118.75715593924485 to a new cost of 1.5509126920967042 > The time to compute the plan is also printed in a format that is not human > readable. we also need to let operator understand that balancer is just > submitting the plan and it be up to execution to complete the move. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25973) Balancer should explain progress in a better way in log
[ https://issues.apache.org/jira/browse/HBASE-25973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-25973: Status: Patch Available (was: Open) > Balancer should explain progress in a better way in log > --- > > Key: HBASE-25973 > URL: https://issues.apache.org/jira/browse/HBASE-25973 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 3.0.0-alpha-1 >Reporter: Clara Xiong >Assignee: Clara Xiong >Priority: Major > > In the log, balancer logs at info level at the beginning of run: > > balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer, > initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction : > (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0, > 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0); > TableSkewCostFunction : (1000.0, 0.0019704142954972883); > StoreFileCostFunction : (200.0, 0.3668512059459341); computedMaxSteps: > 42270438200 > the cost is reported without context, it is hard for operator to understand > how unbalanced the cluster is for balancer and how much progress we are > making. > For a large cluster, the calculation can take a long time, we also need to > let operator understand that it will take up to the max time to complete the > calculation. > At the end of computation: > balancer.StochasticLoadBalancer: Finished computing new load balance plan. > Computation took PT40M0.006S to try 1036409 different iterations. Found a > solution that moves 161926 regions; Going from a computed cost of > 118.75715593924485 to a new cost of 1.5509126920967042 > The time to compute the plan is also printed in a format that is not human > readable. we also need to let operator understand that balancer is just > submitting the plan and it be up to execution to complete the move. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25973) Balancer should explain progress in a better way in log
[
https://issues.apache.org/jira/browse/HBASE-25973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-25973:
Description:
In the log, balancer logs at info level at the beginning of run:
{code}
balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction : (500.0,
0.3749771215224234); ServerLocalityCostFunction : (25.0, 0.5807483226644186);
RackLocalityCostFunction : (15.0, 0.0); TableSkewCostFunction : (1000.0,
0.0019704142954972883); StoreFileCostFunction : (200.0, 0.3668512059459341);
computedMaxSteps: 42270438200
{code}
the cost is reported without context, it is hard for operator to understand how
unbalanced the cluster is for balancer and how much progress we are making.
For a large cluster, the calculation can take a long time, we also need to let
operator understand that it will take up to the max time to complete the
calculation.
At the end of computation:
{code}
balancer.StochasticLoadBalancer: Finished computing new load balance plan.
Computation took PT40M0.006S to try 1036409 different iterations. Found a
solution that moves 161926 regions; Going from a computed cost of
118.75715593924485 to a new cost of 1.5509126920967042
{code}
The time to compute the plan is also printed in a format that is not human
readable. we also need to let operator understand that balancer is just
submitting the plan and it be up to execution to complete the move.
was:
In the log, balancer logs at info level at the beginning of run:
balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction : (500.0,
0.3749771215224234); ServerLocalityCostFunction : (25.0, 0.5807483226644186);
RackLocalityCostFunction : (15.0, 0.0); TableSkewCostFunction : (1000.0,
0.0019704142954972883); StoreFileCostFunction : (200.0, 0.3668512059459341);
computedMaxSteps: 42270438200
the cost is reported without context, it is hard for operator to understand how
unbalanced the cluster is for balancer and how much progress we are making.
For a large cluster, the calculation can take a long time, we also need to let
operator understand that it will take up to the max time to complete the
calculation.
At the end of computation:
balancer.StochasticLoadBalancer: Finished computing new load balance plan.
Computation took PT40M0.006S to try 1036409 different iterations. Found a
solution that moves 161926 regions; Going from a computed cost of
118.75715593924485 to a new cost of 1.5509126920967042
The time to compute the plan is also printed in a format that is not human
readable. we also need to let operator understand that balancer is just
submitting the plan and it be up to execution to complete the move.
> Balancer should explain progress in a better way in log
> ---
>
> Key: HBASE-25973
> URL: https://issues.apache.org/jira/browse/HBASE-25973
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 3.0.0-alpha-1
>Reporter: Clara Xiong
>Assignee: Clara Xiong
>Priority: Major
>
> In the log, balancer logs at info level at the beginning of run:
> {code}
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0,
> 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (1000.0, 0.0019704142954972883);
> StoreFileCostFunction : (200.0, 0.3668512059459341); computedMaxSteps:
> 42270438200
> {code}
> the cost is reported without context, it is hard for operator to understand
> how unbalanced the cluster is for balancer and how much progress we are
> making.
> For a large cluster, the calculation can take a long time, we also need to
> let operator understand that it will take up to the max time to complete the
> calculation.
> At the end of computation:
> {code}
> balancer.StochasticLoadBalancer: Finished computing new load balance plan.
> Computation took PT40M0.006S to try 1036409 different iterations. Found a
> solution that moves 161926 regions; Going from a computed cost of
> 118.75715593924485 to a new cost of 1.5509126920967042
> {code}
> The time to compute the plan is also printed in a format that is not human
> readable. we also need to let operator understand that balancer is just
> submitting the plan and it be up to execution to complete the move.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
busbey commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646624197
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
+"Start calculating moving plan without logging info for up to {} secs",
Review comment:
This is a potential value to fill into a parameter in the first string
given to `LOG.info`, does having a parameter to expand in here work?
maybe a comment on this PR with some examples of the log messages we end up
with post-patch would help?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
Review comment:
What's the rationale for moving this from debug logging to info logging?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
Review comment:
you mean beyond the given cost, multiplier, and needed threshold? you
thinking of the individual cost functions?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -470,17 +470,17 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
updateStochasticCosts(tableName, curOverallCost, curFunctionCosts);
if (initCost > currentCost) {
List plans = createRegionPlans(cluster);
- LOG.info("Finished computing new load balance plan. Computation took {}"
+
-" to try {} different iterations. Found a solution that moves " +
-"{} regions; Going from a computed cost of {}" +
-" to a new cost of {}", java.time.Duration.ofMillis(endTime -
startTime),
-step, plans.size(), initCost, currentCost);
+ LOG.info("Finished computing new load balance plan. Computation took {}
ms" +
+ " to try {} different iterations. Found a solution that moves " +
+ "{} regions; Going from a computed cost of {}" +
+ " to a new cost of {}. Move plan to be submitted to master to
execute",
Review comment:
It's not accurate to say we're executing the plan at this point though.
What ware we trying to convey with this addition? From the issue description
I think it's that the act of selecting a plan for the cluster does not mean
we've execu
[GitHub] [hbase] busbey commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
busbey commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646628873
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -470,17 +470,17 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
updateStochasticCosts(tableName, curOverallCost, curFunctionCosts);
if (initCost > currentCost) {
List plans = createRegionPlans(cluster);
- LOG.info("Finished computing new load balance plan. Computation took {}"
+
-" to try {} different iterations. Found a solution that moves " +
-"{} regions; Going from a computed cost of {}" +
-" to a new cost of {}", java.time.Duration.ofMillis(endTime -
startTime),
-step, plans.size(), initCost, currentCost);
+ LOG.info("Finished computing new load balance plan. Computation took {}
ms" +
+ " to try {} different iterations. Found a solution that moves " +
+ "{} regions; Going from a computed cost of {}" +
+ " to a new cost of {}. Move plan to be submitted to master to
execute",
Review comment:
It's not accurate to say we're executing the plan at this point though.
What are we trying to convey with this addition? From the issue description
I think it's that the act of selecting a plan for the cluster does not mean
we've executed that plan. is that correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25969) Cleanup netty-all transitive includes
[ https://issues.apache.org/jira/browse/HBASE-25969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358653#comment-17358653 ] Michael Stack commented on HBASE-25969: --- bq. We can shade our own usage of netty, but if it is used by hadoop or other libs, it is their duty to not introduce netty-all dependencies to downstreamers? Of course. I'm trying to narrow where and why it leaks in as a transitive. Almost done here. > Cleanup netty-all transitive includes > - > > Key: HBASE-25969 > URL: https://issues.apache.org/jira/browse/HBASE-25969 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > > Our releases include lib/netty-all.jar as a transitive include from hadoop. > Purge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
saintstack commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646688776
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
Review comment:
I'm thinking of the times when balancer says the cluster is balanced but
when when the operator looks at the Master UI, it plainly is not For
example, if the regions-per-server are not aligned, this message would explain
why it thinks the cluster balanced though to the the operator, the cluster
looks off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25588) Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true always."
[
https://issues.apache.org/jira/browse/HBASE-25588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358697#comment-17358697
]
Lokesh Khurana commented on HBASE-25588:
[~zhangduo] Sorry, for past couple of months I haven't been able to work, if it
requires immediate attention, please feel free to reassign it!
> Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true
> always."
> --
>
> Key: HBASE-25588
> URL: https://issues.apache.org/jira/browse/HBASE-25588
> Project: HBase
> Issue Type: Bug
> Components: logging, Operability, Replication
>Affects Versions: 2.4.1
>Reporter: Andrew Kyle Purtell
>Assignee: Lokesh Khurana
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Log line printed frequently, can be as often as once per second:
> {noformat}
> 21:56:46.381 WARN
> [RS_REFRESH_PEER-regionserver/ip-172-31-51-4:8120-0.replicationSource,
> 1.replicationSource.shipperip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.ip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.default,1]
>org.apache.hadoop.hbase.zookeeper.ZKUtil - hbase.zookeeper.useMulti is
> deprecated. Default to true always.
> {noformat}
> The old configuration in place had hbase.zookeeper.useMulti set to true.
> This is the default and while I understand this setting is deprecated,
> constantly warning about it, especially when it conforms to the default, is
> log spam.
> Warn about this once over the lifetime of the replication source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache-HBase commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-856086535 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | compile | 3m 4s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 47s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 3m 4s | the patch passed | | +1 :green_heart: | javac | 3m 4s | the patch passed | | +1 :green_heart: | shadedjars | 8m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 187m 50s | root in the patch passed. | | | | 228m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3317 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 36acd1f68103 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 456c7f964a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/testReport/ | | Max. process+thread count | 6861 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on a change in pull request #3319: HBASE-25914 Provide slow/large logs on RegionServer UI
pankaj72981 commented on a change in pull request #3319: URL: https://github.com/apache/hbase/pull/3319#discussion_r646790004 ## File path: hbase-server/src/main/resources/hbase-webapps/regionserver/rsOperationDetails.jsp ## @@ -0,0 +1,179 @@ +<%-- +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +--%> +<%@ page contentType="text/html;charset=UTF-8" + import="java.util.Date" + import="java.util.List" + import="java.util.Arrays" + import="java.util.Collections" + import="org.apache.hadoop.conf.Configuration" + import="org.apache.hadoop.hbase.client.SnapshotDescription" Review comment: Removed unused imports. Example, Arrays, SnapshotDescription etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3353: HBASE-25969 Purge netty-all transitive includes
Apache-HBase commented on pull request #3353: URL: https://github.com/apache/hbase/pull/3353#issuecomment-856115154 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | branch-2.4 passed | | +1 :green_heart: | compile | 9m 13s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 9m 20s | the patch passed | | -0 :warning: | javac | 9m 20s | root generated 1 new + 1844 unchanged - 1 fixed = 1845 total (was 1845) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 18m 54s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 53m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3353 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux 3fe19d355d38 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 6d1d64caf7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | Max. process+thread count | 126 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25963) HBaseCluster should be marked as IA.Public
[
https://issues.apache.org/jira/browse/HBASE-25963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358779#comment-17358779
]
Hudson commented on HBASE-25963:
Results for branch branch-2
[build #270 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/270/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/270/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/270/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/270/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/270/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBaseCluster should be marked as IA.Public
> --
>
> Key: HBASE-25963
> URL: https://issues.apache.org/jira/browse/HBASE-25963
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> Its sub class MiniHBaseCluster is marked as IA.Public, so it should also be
> IA.Public.
> And in general, HBaseCluster just provides some abstract method so I think it
> is OK to expose it as IA.Public.
> But for MiniHBaseCluster, it exposes several internal stuff such as
> RegionServerThread so maybe it should not be marked as IA.Public.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25969) Cleanup netty-all transitive includes
[ https://issues.apache.org/jira/browse/HBASE-25969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25969: -- Description: Our releases include lib/netty-all.jar as a transitive include from hadoop. -Purge.- ... looks like I can't purge the transitive netty-all includes just yet, not w/o moving MR out of hbase core. The transitively included netty-all w/ the old version is needed to run the tests that put up a MR cluster. was:Our releases include lib/netty-all.jar as a transitive include from hadoop. Purge. > Cleanup netty-all transitive includes > - > > Key: HBASE-25969 > URL: https://issues.apache.org/jira/browse/HBASE-25969 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > > Our releases include lib/netty-all.jar as a transitive include from hadoop. > -Purge.- > ... looks like I can't purge the transitive netty-all includes just yet, not > w/o moving MR out of hbase core. The transitively included netty-all w/ the > old version is needed to run the tests that put up a MR cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646840065
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
Review comment:
There is a function that explain the reason of skipping. But we need to
provide the big picture why t stops here. Will update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646840748
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
+"Start calculating moving plan without logging info for up to {} secs",
+ isByTable ? String.format("table (%s)", tableName) : "cluster",
Review comment:
It only prints is by table is turned on. otherwise it only prints
cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646861474
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
Review comment:
Hmm, right. Need to explain cost /multiplier is the measurement how
balanced the cluster for balancer at a scale on [0,1] 0 being the perfectly
balanced and 1 is the worst it could be. But the actual value still doesn't
seem very intuitive to operator. We can explain if we demand the max of
absolute number of unbalanced nodes to be constant, the bigger the cluster, the
smaller of this value would be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646862825
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -470,17 +470,17 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
updateStochasticCosts(tableName, curOverallCost, curFunctionCosts);
if (initCost > currentCost) {
List plans = createRegionPlans(cluster);
- LOG.info("Finished computing new load balance plan. Computation took {}"
+
-" to try {} different iterations. Found a solution that moves " +
-"{} regions; Going from a computed cost of {}" +
-" to a new cost of {}", java.time.Duration.ofMillis(endTime -
startTime),
-step, plans.size(), initCost, currentCost);
+ LOG.info("Finished computing new load balance plan. Computation took {}
ms" +
+ " to try {} different iterations. Found a solution that moves " +
+ "{} regions; Going from a computed cost of {}" +
+ " to a new cost of {}. Move plan to be submitted to master to
execute",
Review comment:
Yes. @busbey after submitting the moves, balancer will rest for another
time period before it resumes. If the moves are complete, it will look again to
see if we need to move more. otherwise, it will skip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646864080
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
+"Start calculating moving plan without logging info for up to {} secs",
Review comment:
Good callout. will massage it. and will show the example logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646864358
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
Review comment:
Because we want to expose it. and this wouldn't cause log spam.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646864645
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
+"Start calculating moving plan without logging info for up to {} secs",
+ isByTable ? String.format("table (%s)", tableName) : "cluster",
+ total, sumMultiplier, minCostNeedBalance, maxRunningTime / 1000);
+if (LOG.isTraceEnabled()) {
+ LOG.trace("Balance decision detailed function costs={}", functionCost());
}
Review comment:
That makes sense. yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache-HBase commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-856190881 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | compile | 2m 35s | master passed | | +1 :green_heart: | shadedjars | 8m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 14s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 2m 47s | the patch passed | | +1 :green_heart: | javac | 2m 47s | the patch passed | | +1 :green_heart: | shadedjars | 8m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 15s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 353m 38s | root in the patch failed. | | | | 390m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3317 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5c3719365bd3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 456c7f964a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/testReport/ | | Max. process+thread count | 5159 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3317/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3353: HBASE-25969 Purge netty-all transitive includes
Apache-HBase commented on pull request #3353: URL: https://github.com/apache/hbase/pull/3353#issuecomment-856207021 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 46s | branch-2.4 passed | | +1 :green_heart: | compile | 2m 57s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 49s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 50s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 2m 45s | the patch passed | | +1 :green_heart: | javac | 2m 45s | the patch passed | | +1 :green_heart: | shadedjars | 6m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 40s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 164m 9s | root in the patch failed. | | | | 203m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3353 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 23ba441ee46a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 6d1d64caf7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/testReport/ | | Max. process+thread count | 5395 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r646895341
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -343,15 +343,15 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
costFunctions);
}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+LOG.info("{} {}; total cost={}, sum multiplier={}; cost/multiplier to need
a balance is {}",
+ balanced ? "Skipping load balancing because balanced" :
Review comment:
https://issues.apache.org/jira/browse/HBASE-25666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #3354: HBASE-25391 Flush directly into data directory, skip rename when comm…
wchevreuil commented on pull request #3354: URL: https://github.com/apache/hbase/pull/3354#issuecomment-856244728 I believe the latest UTs are flakey, have the same passing in my local build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25980) Master table.jsp pointed at meta throws 500 when no all replicas are online
Nick Dimiduk created HBASE-25980:
Summary: Master table.jsp pointed at meta throws 500 when no all
replicas are online
Key: HBASE-25980
URL: https://issues.apache.org/jira/browse/HBASE-25980
Project: HBase
Issue Type: Bug
Components: master, meta replicas, UI
Affects Versions: 2.3.4
Reporter: Nick Dimiduk
With a replica in a transition state, the UI renders,
{noformat}
HTTP ERROR 500
Problem accessing /table.jsp. Reason:
Server Error
Caused by:
org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out; 1ms
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.waitMetaRegionLocation(MetaTableLocator.java:190)
at
org.apache.hadoop.hbase.generated.master.table_jsp._jspService(table_jsp.java:264)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25980) Master table.jsp pointed at meta throws 500 when no all replicas are online
[
https://issues.apache.org/jira/browse/HBASE-25980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25980:
-
Affects Version/s: (was: 2.3.4)
2.3.5
> Master table.jsp pointed at meta throws 500 when no all replicas are online
> ---
>
> Key: HBASE-25980
> URL: https://issues.apache.org/jira/browse/HBASE-25980
> Project: HBase
> Issue Type: Bug
> Components: master, meta replicas, UI
>Affects Versions: 2.3.5
>Reporter: Nick Dimiduk
>Priority: Major
>
> With a replica in a transition state, the UI renders,
> {noformat}
> HTTP ERROR 500
> Problem accessing /table.jsp. Reason:
> Server Error
> Caused by:
> org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out; 1ms
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.waitMetaRegionLocation(MetaTableLocator.java:190)
> at
> org.apache.hadoop.hbase.generated.master.table_jsp._jspService(table_jsp.java:264)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25343) Avoid the failed meta replica region temporarily in Load Balance mode
[ https://issues.apache.org/jira/browse/HBASE-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25343: Fix Version/s: (was: 2.4.4) 2.4.5 > Avoid the failed meta replica region temporarily in Load Balance mode > - > > Key: HBASE-25343 > URL: https://issues.apache.org/jira/browse/HBASE-25343 > Project: HBase > Issue Type: Sub-task > Components: meta replicas >Affects Versions: 2.4.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > This is a follow-up enhancement with Stack, Duo. With the newly introduced > meta replica LoadBalance mode, if there is something wrong with one of meta > replica regions, the current logic is that it keeps trying until the meta > replica region is onlined again or it reports error, i.e, there is no HA at > LoadBalance mode. HA can be implemented if it reports timeout with one meta > replica region and tries another meta replica region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25433) There is no limit on the table name length when creating a table
[ https://issues.apache.org/jira/browse/HBASE-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25433: Fix Version/s: (was: 2.4.4) 2.4.5 > There is no limit on the table name length when creating a table > > > Key: HBASE-25433 > URL: https://issues.apache.org/jira/browse/HBASE-25433 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.2.3 >Reporter: ZouWeiMing >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5 > > Attachments: attachment_tableNameLengthLimit.html > > > It occurs > Exeception(ERROR：java.util.concurrent.ExecutionExceptionjava.io.IOExpcetionException > in createDir) because of 8000 bytes length limits on HDFS directory name if > the length of HBase table name misses validation when creating. > So,we decide to verify the length of table name when creating a > table.IllegalArgumentExcetion will be thrown when length is larger than 8000 > bytes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25642) Fix or stop warning about already cached block
[
https://issues.apache.org/jira/browse/HBASE-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25642:
Fix Version/s: (was: 2.4.4)
2.4.5
> Fix or stop warning about already cached block
> --
>
> Key: HBASE-25642
> URL: https://issues.apache.org/jira/browse/HBASE-25642
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Operability, regionserver
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> Our logs have as a fairly common occurrence:
> {noformat}
> 2021-03-05 22:24:31,034 WARN [StoreFileOpener-foo-1] hfile.BlockCacheUtil:
> Caching an already cached block: blah.bub. This is harmless and can happen in
> rare cases (see HBASE-8547)
> {noformat}
> If this is really harmless, why do we log? If it's not actually harmless,
> let's take another pass at fixing this situation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25613) [Branch-2 and Master]Handle the NoNode exception in remove log replication in a better way then just log WARN
[ https://issues.apache.org/jira/browse/HBASE-25613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25613: Fix Version/s: (was: 2.4.4) 2.4.5 > [Branch-2 and Master]Handle the NoNode exception in remove log replication in > a better way then just log WARN > - > > Key: HBASE-25613 > URL: https://issues.apache.org/jira/browse/HBASE-25613 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > Currently, when we see the NoNodeException in the replication source while > removing a log from ZK, we swallow that exception and log WARN. > > In certain cases, we might have the peer removed and corresponding logs > removing as well but the replication source continuous to run because of an > RPC failure or anything. > In stead of just log WARN we should check if the peer is removed, if it is > the case, we should terminate the source or try to execute the removePeer > workflow again. > > This would prevent the orphaned source execution infinitely. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: (was: 2.4.4) 2.4.5 > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Assignee: Prathyusha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 1.8.0, 2.4.5 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25694) Improve the shell's 'status replication' command output
[ https://issues.apache.org/jira/browse/HBASE-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25694: Fix Version/s: (was: 2.4.4) 2.4.5 > Improve the shell's 'status replication' command output > --- > > Key: HBASE-25694 > URL: https://issues.apache.org/jira/browse/HBASE-25694 > Project: HBase > Issue Type: Task > Components: Operability, Replication, shell >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Assignee: Divyesh Chandra >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > The output of the shell's 'status "replication"' command could use some > cleaning up. > The basic format is: > version VERSION > N live servers > SERVER_1: >SOURCE: PeerID=PEERID, AgeOfLastShippedOp=160347, SizeOfLogQueue=49, > TimeStampsOfLastShippedOp=Thu Mar 25 01:55:21 UTC 2021, Replication Lag=161013 >SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Mar 25 > 01:49:53 UTC 2021 > ... > Issues: > - Per line format is KEY=VALUE, ... . Most KEYs are CamelCase but some have > spaces. Should all be camel case for consistency. > - Age is printed as long and timestamp is string (local TZ) format. I think > the string formatting of the timestamp makes it difficult to read. Lets just > print timestamps as longs. Or provide a non-default option for string > formatted timestamps. > - There is no sense of effective rate or bandwidth. Each source and sink line > emitted should provide some sense of current workload: rpcs/sec, bytes/sec, > cells/sec... something. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25521) Change ChoreService and ScheduledChore to IA.Private
[ https://issues.apache.org/jira/browse/HBASE-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25521: Fix Version/s: (was: 2.4.4) 2.4.5 > Change ChoreService and ScheduledChore to IA.Private > > > Key: HBASE-25521 > URL: https://issues.apache.org/jira/browse/HBASE-25521 > Project: HBase > Issue Type: Task > Components: util >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > Follow up issue for HBASE-25509. > We are not providing a general scheduling library, so these two classes > should not be IA.Public. User can just use other more powerful scheduling > libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open when using TinyLfuBlockCache
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25698:
Fix Version/s: (was: 2.4.4)
2.4.5
> Persistent IllegalReferenceCountException at scanner open when using
> TinyLfuBlockCache
> --
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25588) Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true always."
[
https://issues.apache.org/jira/browse/HBASE-25588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25588:
Fix Version/s: (was: 2.4.4)
2.4.5
> Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true
> always."
> --
>
> Key: HBASE-25588
> URL: https://issues.apache.org/jira/browse/HBASE-25588
> Project: HBase
> Issue Type: Bug
> Components: logging, Operability, Replication
>Affects Versions: 2.4.1
>Reporter: Andrew Kyle Purtell
>Assignee: Lokesh Khurana
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> Log line printed frequently, can be as often as once per second:
> {noformat}
> 21:56:46.381 WARN
> [RS_REFRESH_PEER-regionserver/ip-172-31-51-4:8120-0.replicationSource,
> 1.replicationSource.shipperip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.ip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.default,1]
>org.apache.hadoop.hbase.zookeeper.ZKUtil - hbase.zookeeper.useMulti is
> deprecated. Default to true always.
> {noformat}
> The old configuration in place had hbase.zookeeper.useMulti set to true.
> This is the default and while I understand this setting is deprecated,
> constantly warning about it, especially when it conforms to the default, is
> log spam.
> Warn about this once over the lifetime of the replication source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25729: Fix Version/s: (was: 2.4.4) 2.4.5 > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25915) ‘mobdir’ overlaps in HFileLink.mobPath
[ https://issues.apache.org/jira/browse/HBASE-25915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25915: Fix Version/s: (was: 2.4.4) 2.4.5 > ‘mobdir’ overlaps in HFileLink.mobPath > -- > > Key: HBASE-25915 > URL: https://issues.apache.org/jira/browse/HBASE-25915 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.3 >Reporter: wenfeiyi666 >Assignee: wenfeiyi666 >Priority: Minor > Fix For: 3.0.0-alpha-1, 1.4.14, 2.4.5 > > > ‘mobdir’ overlaps in HFileLink.mobPath, like > '/root/mobdir/mobdir/data/table/...' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25823) TestSlowLogAccessor.testHigherSlowLogs repeatable failure
[
https://issues.apache.org/jira/browse/HBASE-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25823:
Fix Version/s: (was: 2.4.4)
2.4.5
> TestSlowLogAccessor.testHigherSlowLogs repeatable failure
> -
>
> Key: HBASE-25823
> URL: https://issues.apache.org/jira/browse/HBASE-25823
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 2.4.5
>
>
> {noformat}
> [ERROR] TestSlowLogAccessor.testHigherSlowLogs:211 Waiting timed out after
> [7,000] msec{noformat}
> Repeatable failure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25947) Backport 'HBASE-25894 Improve the performance for region load and region count related cost functions' to branch-2.4 and branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25947: Fix Version/s: (was: 2.4.4) 2.4.5 > Backport 'HBASE-25894 Improve the performance for region load and region > count related cost functions' to branch-2.4 and branch-2.3 > --- > > Key: HBASE-25947 > URL: https://issues.apache.org/jira/browse/HBASE-25947 > Project: HBase > Issue Type: Sub-task > Components: Balancer, Performance >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.3.6, 2.4.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
[
https://issues.apache.org/jira/browse/HBASE-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25828:
Fix Version/s: (was: 2.4.4)
2.4.5
> CompactionProgress WARNS: "totalCompactingKVs=N less than
> currentCompactedKVs=M"
>
>
> Key: HBASE-25828
> URL: https://issues.apache.org/jira/browse/HBASE-25828
> Project: HBase
> Issue Type: Bug
> Components: Compaction, Operability, regionserver
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> Similar to HBASE-25642, but compaction progress warnings.
> Lots of warnings like:
> {noformat}
> 2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=15589 less than
> currentCompactedKVs=21411
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=21916 less than
> currentCompactedKVs=33328
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=89731 less than
> currentCompactedKVs=92808
> {noformat}
> A couple of issues:
> - Is there a way to make CompactionProgress more reliable? I seem to recall
> this is the second or third time around the block on this.
> - This is annoying because compaction progress isn't always accurate, but
> this information is not used to determine anything significant, so WARN level
> is a bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358901#comment-17358901 ] Michael Stack commented on HBASE-25729: --- You talking about upgrading to 3.5.1 Andrew? On the one hand, I think it great idea... It would fix HBASE-25960. On other hand I was worried that the bump in versions in thirdparty a little risky for a point release so was thinking of a 3.4.2 release for 2.4 and 2.3 branches... that just fixed the .so issue. Going to 3.5.1 would be less work. What you think? > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3353: HBASE-25969 Purge netty-all transitive includes
Apache-HBase commented on pull request #3353: URL: https://github.com/apache/hbase/pull/3353#issuecomment-856322869 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | branch-2.4 passed | | +1 :green_heart: | compile | 2m 28s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 18s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 2m 27s | the patch passed | | +1 :green_heart: | javac | 2m 27s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 383m 57s | root in the patch passed. | | | | 417m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3353 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d22deaa57be2 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 6d1d64caf7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/testReport/ | | Max. process+thread count | 3133 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358951#comment-17358951 ] Bharath Vissapragada commented on HBASE-24440: -- Thanks for the detailed results. > These implementations were microbenchmarked to identify the superior option, > which seems to be BoundedIncrementYieldAdvancingClock. Kind of surprised. I guessed it would be IncrementAdvancingClock theoretically (since there is no overhead to maintain boundedness), still trying to wrap my head around the subtle details in the patch. > We are tracking whether or not a row has already been committed to in the > current clock tick. Or, we are tracking per region a set of rows per clock > tick. Why not track that explicitly? This is an interesting idea, trying to think about it a bit more. IIUC the idea here is to maximize the commit throughput as much as possible in a single tick by picking from the available pool of disjoint row keys that can share the same tick. Let me take a look at the patch.. > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3353: HBASE-25969 Purge netty-all transitive includes
Apache-HBase commented on pull request #3353: URL: https://github.com/apache/hbase/pull/3353#issuecomment-856344814 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | branch-2.4 passed | | +1 :green_heart: | compile | 9m 12s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | compile | 9m 11s | the patch passed | | -0 :warning: | javac | 9m 11s | root generated 1 new + 1844 unchanged - 1 fixed = 1845 total (was 1845) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 18m 53s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 54m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3353 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux b883855d549f 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 09eebe5cd0 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/5/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | Max. process+thread count | 127 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25729: Fix Version/s: (was: 2.4.5) > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358960#comment-17358960 ] Andrew Kyle Purtell edited comment on HBASE-25729 at 6/8/21, 12:38 AM: --- bq. On other hand I was worried that the bump in versions in thirdparty a little risky for a point release so was thinking of a 3.4.2 release for 2.4 and 2.3 branches... that just fixed the .so issue. Yeah, I've been holding off on this for 2.4 for that reason. Removed that fix version. +1 to the idea of a smaller scoped fix for point releases. was (Author: apurtell): bq. On other hand I was worried that the bump in versions in thirdparty a little risky for a point release so was thinking of a 3.4.2 release for 2.4 and 2.3 branches... that just fixed the .so issue. Yeah, I've been holding off on this for 2.4 for that reason. Removed that fix version. > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358960#comment-17358960 ] Andrew Kyle Purtell commented on HBASE-25729: - bq. On other hand I was worried that the bump in versions in thirdparty a little risky for a point release so was thinking of a 3.4.2 release for 2.4 and 2.3 branches... that just fixed the .so issue. Yeah, I've been holding off on this for 2.4 for that reason. Removed that fix version. > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358203#comment-17358203 ] Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:39 AM: --- In HBASE-25913 I considered serialization of commits within a given clock tick using the system clock, by extending the interface EnvironmentEdge, which already dealt with time. Consider an API for getting a Clock, by name, from a concurrent map of such things, and then using this clock to get _advancing time_, which is the current system time, like System.currentTimeMillis, but with an additional semantic: When you call getCurrentAdvancing() you will always get a value that has advanced forward from the last time someone looked at this particular clock's time. The particular implementation of Clock is free to maintain that invariant with different strategies, and I explored several different implementations: - IncrementAdvancingClock: like a poor man's hybrid clock, if the system time hasn't advanced, manually advance it, and ensure other API calls to the same clock don't give a different, earlier time - SpinAdvancingClock: if the system time hasn't advanced, spin until it does - YieldAdvancingClock: if the system time hasn't advanced, yield the thread with a small sleep until it does - BoundedIncrementSpinAdvancingClock: advance ahead of system time up to some bound (like 1 second) but then spin until the system time catches up - BoundedIncrementYieldAdvancingClock: advance ahead of system time up to some bound (like 1 second) but then yield with a small sleep until the system time catches up These implementations were microbenchmarked to identify the superior option, which seems to be IncrementAdvancingClock. The main issue with HBASE-25913 is serializing commits with the act of getting the time requires that everything that might mutate the resource you are trying to protect must be within that Clock's scope. For a single row update the scope would be a row, and highly granular, so performance will not be affected that much. For batch mutations, which are typical, the scope is multiple rows, so therefore the region (_strictly speaking, it is the union set of the rows in the batch mutation_ ... I will come back to this later), and serializing commits to a region is expensive. Unfortunately it is always the case that the region must be the scope of control. Consider multiple RPC handlers executing in parallel. One is a single row mutation for rowA. Another is a batch mutation for rowA, rowB, and rowC. If we "optimize" single row updates without respect to concurrent batch requests, with two separate clock instances for row(rowA) and region(rowA,rowB,rowC), then rowA in this scenario might get two commits in the same system clock tick, because two clocks could have different notions of when time last advanced. We've done a lot of work to gain no assurance. So that is an invalid optimization and so the HBASE-25913 change creates a Clock per region and all timekeeping for the region goes through that Clock. With BoundedIncrementYieldAdvancingClock that means the region can take a burst of commits within a single clock tick up to the limit and then there is a performance discontinuity while we yield until the system time catches up (which can be up to 1 second). Other threads on other regions will make progress, but for the client attempting to commit a burst with the yielding operation, performance is a concern. With YieldAdvancingClock, the region can take only one commit for an entire clock tick, which is way worse. The potential performance impacts were known at the outset of the work so are not surprising. With some implementations now the impacts can be measured. This should be managed at the row scope, not the region scope, but then what to do about batch mutations? I struggled for a while with various forms of *hierarchical clocks* where one might get a clock for the region, then use that clock to get child clocks for a row, and the hierarchy would make sure that time would not run backwards for anyone even if the row clocks were of the IncrementXXX types (because spinning or yielding is simply too expensive to do outright). This was not successful in the way I would like because they always ended up serializing _something_ at region scope in a way that I thought would impact performance rather negatively. So, let's take a step back: What is it we are really tracking? We are tracking whether or not a row has already been committed to in the current clock tick. Or, _we are tracking per region a set of rows per clock tick_. Why not track that explicitly? This idea is being explored with HBASE-25975, which does that with a region level data structure based on atomics because access to it will be highly multithreaded. In some
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358962#comment-17358962
]
Andrew Kyle Purtell commented on HBASE-24440:
-
{quote}
bq. These implementations were microbenchmarked to identify the superior
option, which seems to be BoundedIncrementYieldAdvancingClock.
Kind of surprised. I guessed it would be IncrementAdvancingClock theoretically
(since there is no overhead to maintain boundedness), still trying to wrap my
head around the subtle details in the patch.
{quote}
My mistake, that's a typo. Let me fix it.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell commented on HBASE-24440:
-
{quote}
This is an interesting idea, trying to think about it a bit more. IIUC the idea
here is to maximize the commit throughput as much as possible in a single tick
by picking from the available pool of disjoint row keys that can share the same
tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back on that subtask when I post an update
there.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:41 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back on that subtask when I post an update
there.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:41 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back on that subtask when I post an update
there.
was (Author: apurtell):
{quote}
This is an interesting idea, trying to think about it a bit more. IIUC the idea
here is to maximize the commit throughput as much as possible in a single tick
by picking from the available pool of disjoint row keys that can share the same
tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back on that subtask when I post an update
there.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:44 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Run through all of the rows involved in the pending mutation. The check adds
each row to the set, but allows the pending mutation if none of the rows hit
while adding.
- If the pending mutation is allowed, return the time where we did the atomic
clock advance check.
- Otherwise, yield the thread, and try again.
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:45 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Run through all of the rows involved in the pending mutation. The check adds
each row to the set, but allows the pending mutation as long as each row did
not exist in the set when it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Run through all of the rows involved in the pending mutation. The check adds
each row to the set, but allows the pending mutation if none of the rows hit
while adding.
- If the pending mutation is allowed, return the time where we did the atomic
clock advance check.
- Otherwise, yield the thread, and try again.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:47 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Run through all of the rows involved in the pending mutation. The check adds
each row to the set, but allows the pending mutation as long as each row did
not exist in the set when it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> wit
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:49 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple.
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:50 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I don't want a Set, I want a Map of
reference counts.
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the c
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:52 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I don't want a Set, I want a Map where the
values can be used to implement a tri-state check: row does not conflict
(pass), row does conflict (yield and retry), row might conflict (retry
immediately)
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I don't want a Set, I want a Map of
reference counts.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple soluti
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:53 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I need to implement a tri-state check: row
does not conflict (pass), row does conflict (yield and retry), row might
conflict (retry immediately)
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I don't want a Set, I want a Map where the
values can be used to implement a tri-state check: row does not conflict
(pass), row does conflict (yield and retry), row might conflict (retry
immediately)
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anoma
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 12:54 AM:
---
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check as the time we will make timestamp
substitutions with.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I need to implement a tri-state check: row
does not conflict (pass), row does conflict (yield and retry), row might
conflict (retry immediately)
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I need to implement a tri-state check: row
does not conflict (pass), row does conflict (yield and retry), row might
conflict (retry immediately)
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomal
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 1:01 AM:
--
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check as the time we will make timestamp
substitutions with.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple.
Edit: The straightforward thing to do is a synchronized section that does all
of the existence tests with the set but doesn't mutate the set until we have a
pass or yield decision, at which point the result is merged to the set or not.
Hmm,..
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check as the time we will make timestamp
substitutions with.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple. I need to implement a tri-state check: row
does not conflict (pass), row does conflict (yield and retry), row might
conflict (retry immediately)
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedenc
[jira] [Commented] (HBASE-25975) Row commit sequencer
[ https://issues.apache.org/jira/browse/HBASE-25975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358967#comment-17358967 ] Andrew Kyle Purtell commented on HBASE-25975: - Some discussion copied from parent The basic idea is this: First, check if the clock has advanced, atomically. If it has, then clear the row set, also atomically, with respect to the clock advance check. Get a reference to the row set for the current tick. Run through all of the rows involved in the pending mutation. The check adds each row to the set, but allows the pending mutation as long as each row did not exist in the set when it was added. If the pending mutation is going to be allowed to go forward, return the time where we did the atomic clock advance check as the time we will make timestamp substitutions with. Otherwise, yield the thread, and try again, which will keep the handler in queue until the row set is finally disjoint from any other (per the check logic described above) One thing I realized I'm missing while typing this up is some kind of rollback if we added rows to the set for something which overlaps elsewhere, and so might block some other pending operation on rows for this mutation that is going to yield instead of go forward. That's what I was alluding to above... what I have right now is too simple. Edit: The straightforward thing to do is a synchronized section that does all of the existence tests with the set but doesn't mutate the set until we have a pass or yield decision, at which point the result is merged to the set or not. Hmm,.. > Row commit sequencer > > > Key: HBASE-25975 > URL: https://issues.apache.org/jira/browse/HBASE-25975 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Use a row commit sequencer in HRegion to ensure that only the operations that > mutate disjoint sets of rows are able to commit within the same clock tick. > This maintains the invariant that more than one mutation to a given row will > never be committed in the same clock tick. > Callers will first acquire row locks for the row(s) the pending mutation will > mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that > the set of rows about to be mutated do not overlap with those for any other > pending mutations in the current clock tick. If an overlap is identified, > getRowSequence will yield and loop until there is no longer an overlap and > the caller's pending mutation can succeed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358964#comment-17358964
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 6/8/21, 1:13 AM:
--
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check as the time we will make timestamp
substitutions with.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple.
Edit: The straightforward thing to do is a synchronized section that does all
of the existence tests with the set but doesn't mutate the set until we have a
pass or yield decision, at which point the result is merged to the set or not.
Hmm,..
Edit2: I like this phrasing ("picking from the available pool of disjoint row
keys that can share the same tick") but it's first come first serve. We don't
pick among alternatives. (That feels like constraint solving... all for one
tick?) It's just a one pass filter.
was (Author: apurtell):
{quote}
IIUC the idea here is to maximize the commit throughput as much as possible in
a single tick by picking from the available pool of disjoint row keys that can
share the same tick. Let me take a look at the patch..
{quote}
This is the HBASE-25975 subtask. I still need to add a test that proves it even
works. :-) You might want to come back to that subtask in a bit, when I post an
update there.
The basic idea is this:
- First, check if the clock has advanced, atomically. If it has, then clear the
row set, also atomically, with respect to the clock advance check.
- Get a reference to the row set for the current tick. Run through all of the
rows involved in the pending mutation. The check adds each row to the set, but
allows the pending mutation as long as each row did not exist in the set when
it was added.
- If the pending mutation is going to be allowed to go forward, return the time
where we did the atomic clock advance check as the time we will make timestamp
substitutions with.
- Otherwise, yield the thread, and try again, which will keep the handler in
queue until the row set is finally disjoint from any other (per the check logic
described above)
One thing I realized I'm missing while typing this up is some kind of rollback
if we added rows to the set for something which overlaps elsewhere, and so
might block some other pending operation on rows for this mutation that is
going to yield instead of go forward. That's what I was alluding to above...
what I have right now is too simple.
Edit: The straightforward thing to do is a synchronized section that does all
of the existence tests with the set but doesn't mutate the set until we have a
pass or yield decision, at which point the result is merged to the set or not.
Hmm,..
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When
[jira] [Commented] (HBASE-25947) Backport 'HBASE-25894 Improve the performance for region load and region count related cost functions' to branch-2.4 and branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358970#comment-17358970 ] Duo Zhang commented on HBASE-25947: --- [~apurtell] The PR is already ready for reviewing, mind taking a look? [~ndimiduk] has already reviewed. Thanks. > Backport 'HBASE-25894 Improve the performance for region load and region > count related cost functions' to branch-2.4 and branch-2.3 > --- > > Key: HBASE-25947 > URL: https://issues.apache.org/jira/browse/HBASE-25947 > Project: HBase > Issue Type: Sub-task > Components: Balancer, Performance >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.3.6, 2.4.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #3362: HBASE-22708 Remove the deprecated methods in Hbck interface
Apache9 merged pull request #3362: URL: https://github.com/apache/hbase/pull/3362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-22708) Remove the deprecated methods in Hbck interface
[ https://issues.apache.org/jira/browse/HBASE-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-22708. --- Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Resolution: Fixed Merged to master. Thanks [~GeorryHuang] for contributing. Do not forget to fill the release note about the breaking change [~GeorryHuang]. > Remove the deprecated methods in Hbck interface > --- > > Key: HBASE-22708 > URL: https://issues.apache.org/jira/browse/HBASE-22708 > Project: HBase > Issue Type: Bug > Components: hbck2 >Affects Versions: 3.0.0-alpha-1 >Reporter: Guanghao Zhang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Only for 3.0.0. Remove the methods which mark deprecated in HBASE-22673. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache9 commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-856372689 The failed UTs are only in backup module and jdk8. Since it is not in any releases and known to be flaky sometimes, let's merge this PR first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache9 merged pull request #3317: URL: https://github.com/apache/hbase/pull/3317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3353: HBASE-25969 Purge netty-all transitive includes
Apache-HBase commented on pull request #3353: URL: https://github.com/apache/hbase/pull/3353#issuecomment-856393928 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | branch-2.4 passed | | +1 :green_heart: | compile | 2m 43s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 42s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 2m 46s | the patch passed | | +1 :green_heart: | javac | 2m 46s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 43s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 165m 6s | root in the patch failed. | | | | 201m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3353 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e19a2ba81ba0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 09eebe5cd0 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/5/testReport/ | | Max. process+thread count | 5272 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3353/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Kumar updated HBASE-25918: - Fix Version/s: 3.0.0-alpha-1 > Upgrade hbase-thirdparty dependency to 3.5.1 > > > Key: HBASE-25918 > URL: https://issues.apache.org/jira/browse/HBASE-25918 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > Recently we have fixed multiple CVEs from jetty & netty as part of > HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdparty jar > version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] pankaj72981 commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
pankaj72981 commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-856426977 Thanks @Apache9 for merging this PR, let me open a new PR for branch-2 and see the QA report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 opened a new pull request #3363: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
pankaj72981 opened a new pull request #3363: URL: https://github.com/apache/hbase/pull/3363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache9 commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-856456528 > Thanks @Apache9 for merging this PR, let me open a new PR for branch-2 and see the QA report. I think we could just cherry-pick it and keep an eye on the night result. We have more testing way in nightly build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on pull request #3317: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
pankaj72981 commented on pull request #3317: URL: https://github.com/apache/hbase/pull/3317#issuecomment-856457835 Ok, I have created a PR. Let me close that and cherry pick in 2.4+ branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 closed pull request #3363: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
pankaj72981 closed pull request #3363: URL: https://github.com/apache/hbase/pull/3363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Kumar resolved HBASE-25918. -- Fix Version/s: 2.4.5 2.5.0 Resolution: Fixed Thanks [~zhangduo] for the review and merge. > Upgrade hbase-thirdparty dependency to 3.5.1 > > > Key: HBASE-25918 > URL: https://issues.apache.org/jira/browse/HBASE-25918 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > Recently we have fixed multiple CVEs from jetty & netty as part of > HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdparty jar > version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Kumar resolved HBASE-25729. -- Fix Version/s: 2.4.5 Resolution: Fixed > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359048#comment-17359048 ] Duo Zhang commented on HBASE-25918: --- We should also include this for 2.3.x? > Upgrade hbase-thirdparty dependency to 3.5.1 > > > Key: HBASE-25918 > URL: https://issues.apache.org/jira/browse/HBASE-25918 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > Recently we have fixed multiple CVEs from jetty & netty as part of > HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdparty jar > version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359053#comment-17359053 ] Pankaj Kumar commented on HBASE-25729: -- Pardon me, I wasn't aware of above conversation. Shall I revert back HBASE-25918 from branch-2.4? > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25729) Upgrade to latest hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Kumar reopened HBASE-25729: -- > Upgrade to latest hbase-thirdparty > -- > > Key: HBASE-25729 > URL: https://issues.apache.org/jira/browse/HBASE-25729 > Project: HBase > Issue Type: Sub-task > Components: build, thirdparty >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359057#comment-17359057 ] Pankaj Kumar commented on HBASE-25918: -- There is some discussion in HBASE-25729 for upgrading hbase-thirdparty in 2.3 & 2.4 branch, We may need to revert this from 2.4 or cherry-pick to 2.3 after [~stack] & [~apurtell] confirmation. > Upgrade hbase-thirdparty dependency to 3.5.1 > > > Key: HBASE-25918 > URL: https://issues.apache.org/jira/browse/HBASE-25918 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > Recently we have fixed multiple CVEs from jetty & netty as part of > HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdparty jar > version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25981) JVM crash when displaying regionserver UI
Xiaolin Ha created HBASE-25981:
--
Summary: JVM crash when displaying regionserver UI
Key: HBASE-25981
URL: https://issues.apache.org/jira/browse/HBASE-25981
Project: HBase
Issue Type: Bug
Components: rpc, UI
Affects Versions: 2.0.0, 3.0.0-alpha-1
Reporter: Xiaolin Ha
Assignee: Xiaolin Ha
Attachments: hs_err_pid116190.log-gha-data-hbase-cat0085-ui
The MonitoredRPCHandlerImpl refers to the params of a request, and will show
them when we call 'toJson()'. But the running RPC call may be cleaned up and
the ByteBuffer be released before the displaying in UI. We need to let the
life cycle of RPC status monitor be inner the life cycle of RPC.
{code:java}
J 19267 C2
org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printMessage(Lorg/apache/hbase/thirdparty/com/google/protobuf/MessageOrBuilder;Lorg/apache/hbase/thirdparty/com/google/protobuf/TextFormat$TextGenerator;)V
(73 bytes) @ 0x7f1ac7e54640 [0x7f1ac7e53f60+0x6e0]
J 20932 C2
org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.print(Lorg/apache/hbase/thirdparty/com/google/protobuf/MessageOrBuilder;Lorg/apache/hbase/thirdparty/com/google/protobuf/TextFormat$TextGenerator;)V
(34 bytes) @ 0x7f1ac68ab9b0 [0x7f1ac68ab880+0x130]
J 21843 C1
org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage.toString()Ljava/lang/String;
(8 bytes) @ 0x7f1ac620e14c [0x7f1ac620dba0+0x5ac]
J 21835 C1
org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.toMap()Ljava/util/Map;
(240 bytes) @ 0x7f1ac5009bf4 [0x7f1ac50071c0+0x2a34]
J 21833 C1
org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.toJSON()Ljava/lang/String;
(5 bytes) @ 0x7f1ac74efb74 [0x7f1ac74efaa0+0xd4]
j
org.apache.hadoop.hbase.tmpl.common.TaskMonitorTmplImpl.renderNoFlush(Ljava/io/Writer;)V+259
j
org.apache.hadoop.hbase.tmpl.common.TaskMonitorTmpl.renderNoFlush(Ljava/io/Writer;)V+16
j
org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(Ljava/io/Writer;)V+129
{code}
[^hs_err_pid116190.log-gha-data-hbase-cat0085-ui]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359061#comment-17359061 ] Duo Zhang commented on HBASE-25918: --- OK, no problem. > Upgrade hbase-thirdparty dependency to 3.5.1 > > > Key: HBASE-25918 > URL: https://issues.apache.org/jira/browse/HBASE-25918 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > Recently we have fixed multiple CVEs from jetty & netty as part of > HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdparty jar > version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3363: HBASE-25918 Upgrade hbase-thirdparty dependency to 3.5.1
Apache-HBase commented on pull request #3363: URL: https://github.com/apache/hbase/pull/3363#issuecomment-856477343 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | branch-2 passed | | +1 :green_heart: | compile | 8m 42s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | compile | 8m 43s | the patch passed | | -0 :warning: | javac | 8m 43s | root generated 1 new + 1886 unchanged - 1 fixed = 1887 total (was 1887) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 11m 56s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 45m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3363/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3363 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux 9c732496afc0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e4dd4901a8 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3363/1/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | Max. process+thread count | 141 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3363/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
