Re: [PR] HBASE-28384 Client ingegration tests fails for branch-2/branch-2.6 [hbase]
Apache9 commented on PR #5714: URL: https://github.com/apache/hbase/pull/5714#issuecomment-1968400901 https://github.com/apache/hbase/commit/b3892cb3eb3c0073ae64da5704653aaef27bf37a There is an extra commit on that branch to skip all other checks :) It is not included in the PR here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28404 Use "set -x" when running release script in debug mode [hbase]
Apache-HBase commented on PR #5715: URL: https://github.com/apache/hbase/pull/5715#issuecomment-1968360748 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 39s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 3m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5715/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5715 | | Optional Tests | dupname asflicense spotless shellcheck shelldocs | | uname | Linux c14ae67f1db7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4b552434db | | Max. process+thread count | 43 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5715/1/console | | versions | git=2.34.1 maven=3.8.6 shellcheck=0.8.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28404 Use "set -x" when running release script in debug mode [hbase]
Apache-HBase commented on PR #5715: URL: https://github.com/apache/hbase/pull/5715#issuecomment-1968359220 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 5s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 1m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5715/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5715 | | Optional Tests | | | uname | Linux ff92e8349446 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4b552434db | | Max. process+thread count | 41 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5715/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28404 Use "set -x" when running release script in debug mode [hbase]
Apache-HBase commented on PR #5715: URL: https://github.com/apache/hbase/pull/5715#issuecomment-1968359015 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 1m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5715/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5715 | | Optional Tests | | | uname | Linux 2e9ee592d2c9 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4b552434db | | Max. process+thread count | 28 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5715/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28404 Use "set -x" when running release script in debug mode [hbase]
stoty commented on PR #5715: URL: https://github.com/apache/hbase/pull/5715#issuecomment-1968347640 FYI @ndimiduk @virajjasani @saintstack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-28404) Use "set -x" when running release script in debug mode
[ https://issues.apache.org/jira/browse/HBASE-28404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth reassigned HBASE-28404: --- Assignee: Istvan Toth > Use "set -x" when running release script in debug mode > -- > > Key: HBASE-28404 > URL: https://issues.apache.org/jira/browse/HBASE-28404 > Project: HBase > Issue Type: Improvement > Components: scripts >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > Labels: pull-request-available > > Phoenix release scripts are forked from HBase. > I found using the bash "set -x" command very useful when diagnosing > problems. > It is implemented as part of PHOENIX-7236, and could be very easily ported > back to HBase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28404) Use "set -x" when running release script in debug mode
[ https://issues.apache.org/jira/browse/HBASE-28404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated HBASE-28404: Status: Patch Available (was: Open) > Use "set -x" when running release script in debug mode > -- > > Key: HBASE-28404 > URL: https://issues.apache.org/jira/browse/HBASE-28404 > Project: HBase > Issue Type: Improvement > Components: scripts >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > > Phoenix release scripts are forked from HBase. > I found using the bash "set -x" command very useful when diagnosing > problems. > It is implemented as part of PHOENIX-7236, and could be very easily ported > back to HBase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28404) Use "set -x" when running release script in debug mode
[ https://issues.apache.org/jira/browse/HBASE-28404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-28404: --- Labels: pull-request-available (was: ) > Use "set -x" when running release script in debug mode > -- > > Key: HBASE-28404 > URL: https://issues.apache.org/jira/browse/HBASE-28404 > Project: HBase > Issue Type: Improvement > Components: scripts >Reporter: Istvan Toth >Priority: Minor > Labels: pull-request-available > > Phoenix release scripts are forked from HBase. > I found using the bash "set -x" command very useful when diagnosing > problems. > It is implemented as part of PHOENIX-7236, and could be very easily ported > back to HBase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] HBASE-28404 Use "set -x" when running release script in debug mode [hbase]
stoty opened a new pull request, #5715: URL: https://github.com/apache/hbase/pull/5715 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28384 Client ingegration tests fails for branch-2/branch-2.6 [hbase]
Apache-HBase commented on PR #5714: URL: https://github.com/apache/hbase/pull/5714#issuecomment-1968344564 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | spotless | 0m 59s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | | -0 :warning: | shellcheck | 0m 0s | The patch generated 19 new + 0 unchanged - 1 fixed = 19 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 44s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 3m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5714/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5714 | | Optional Tests | dupname asflicense spotless shellcheck shelldocs | | uname | Linux 18e23d2846f6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / db6c4b2883 | | shellcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5714/1/artifact/yetus-general-check/output/diff-patch-shellcheck.txt | | Max. process+thread count | 43 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5714/1/console | | versions | git=2.34.1 maven=3.8.6 shellcheck=0.8.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28384 Client ingegration tests fails for branch-2/branch-2.6 [hbase]
Apache-HBase commented on PR #5714: URL: https://github.com/apache/hbase/pull/5714#issuecomment-1968342245 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 48s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 5s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 1m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5714/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5714 | | Optional Tests | | | uname | Linux 124ca1857393 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / db6c4b2883 | | Max. process+thread count | 39 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5714/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28384 Client ingegration tests fails for branch-2/branch-2.6 [hbase]
Apache-HBase commented on PR #5714: URL: https://github.com/apache/hbase/pull/5714#issuecomment-1968342212 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 1m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5714/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5714 | | Optional Tests | | | uname | Linux 6b3b710f5751 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / db6c4b2883 | | Max. process+thread count | 28 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5714/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28384 Client ingegration tests fails for branch-2/branch-2.6 [hbase]
Apache9 commented on PR #5714: URL: https://github.com/apache/hbase/pull/5714#issuecomment-1968335523 The solution is to also build a hadoop3 tarball when doing source release, and then use it to run hadoop 3 related integration test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[ https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-28384: --- Labels: pull-request-available (was: ) > Client ingegration tests fails for branch-2/branch-2.6 > -- > > Key: HBASE-28384 > URL: https://issues.apache.org/jira/browse/HBASE-28384 > Project: HBase > Issue Type: Bug > Components: hadoop3, jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > > In HBASE-28331, we fixed the problem that we should use the latest hadoop3 > tarballs, for generating hdfs-site.xml. > But for branch-2.x, we still use hadoop2 when compiling hbase by default, so > there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop > 3.3.x will generate hdfs-site.xml with this config, then leads to the failure > of hbase start up. > Should we change to build with hadoop3 while running on top of hadoop3? Since > for branch-2.x, we will publish artifacts for hadoop3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28384 Client ingegration tests fails for branch-2/branch-2.6 [hbase]
Apache9 commented on PR #5714: URL: https://github.com/apache/hbase/pull/5714#issuecomment-1968335032 https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/8/ Finally we passed the client integration test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] HBASE-28384 Client ingegration tests fails for branch-2/branch-2.6 [hbase]
Apache9 opened a new pull request, #5714: URL: https://github.com/apache/hbase/pull/5714 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[
https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821524#comment-17821524
]
Hudson commented on HBASE-28384:
Results for branch HBASE-28384-branch-2
[build #8 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/8/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Client ingegration tests fails for branch-2/branch-2.6
> --
>
> Key: HBASE-28384
> URL: https://issues.apache.org/jira/browse/HBASE-28384
> Project: HBase
> Issue Type: Bug
> Components: hadoop3, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> In HBASE-28331, we fixed the problem that we should use the latest hadoop3
> tarballs, for generating hdfs-site.xml.
> But for branch-2.x, we still use hadoop2 when compiling hbase by default, so
> there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop
> 3.3.x will generate hdfs-site.xml with this config, then leads to the failure
> of hbase start up.
> Should we change to build with hadoop3 while running on top of hadoop3? Since
> for branch-2.x, we will publish artifacts for hadoop3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
[
https://issues.apache.org/jira/browse/HBASE-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821504#comment-17821504
]
Hudson commented on HBASE-28391:
Results for branch branch-2.4
[build #701 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/701/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/701/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/701/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/701/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/701/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove the need for ADMIN permissions for listDecommissionedRegionServers
> -
>
> Key: HBASE-28391
> URL: https://issues.apache.org/jira/browse/HBASE-28391
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 2.4.17, 2.5.7
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> Why we need {{ADMIN}} permissions for
> {{AccessController#preListDecommissionedRegionServers}} ?
> From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
> argument {{excludeDecommissionedRS}} is set to true. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
> If {{excludeDecommissionedRS}} is set to true and if we have
> {{AccessController}} co-proc attached, it requires ADMIN permissions to
> execute {{listDecommissionedRegionServers}} RPC. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
>
> {code:java}
> @Override
> public void
> preListDecommissionedRegionServers(ObserverContext
> ctx)
> throws IOException {
> requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
> }
> {code}
> I understand that we need ADMIN permissions for
> _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it
> changes the membership of regionservers but I don’t see any need for ADMIN
> permissions for _listDecommissionedRegionServers_. Do you think we can
> remove need for ADMIN permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[
https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821496#comment-17821496
]
Hudson commented on HBASE-28384:
Results for branch HBASE-28384-branch-2
[build #7 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/7//console].
> Client ingegration tests fails for branch-2/branch-2.6
> --
>
> Key: HBASE-28384
> URL: https://issues.apache.org/jira/browse/HBASE-28384
> Project: HBase
> Issue Type: Bug
> Components: hadoop3, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> In HBASE-28331, we fixed the problem that we should use the latest hadoop3
> tarballs, for generating hdfs-site.xml.
> But for branch-2.x, we still use hadoop2 when compiling hbase by default, so
> there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop
> 3.3.x will generate hdfs-site.xml with this config, then leads to the failure
> of hbase start up.
> Should we change to build with hadoop3 while running on top of hadoop3? Since
> for branch-2.x, we will publish artifacts for hadoop3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[
https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821492#comment-17821492
]
Hudson commented on HBASE-28384:
Results for branch HBASE-28384-branch-2
[build #6 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/6/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/6//console].
> Client ingegration tests fails for branch-2/branch-2.6
> --
>
> Key: HBASE-28384
> URL: https://issues.apache.org/jira/browse/HBASE-28384
> Project: HBase
> Issue Type: Bug
> Components: hadoop3, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> In HBASE-28331, we fixed the problem that we should use the latest hadoop3
> tarballs, for generating hdfs-site.xml.
> But for branch-2.x, we still use hadoop2 when compiling hbase by default, so
> there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop
> 3.3.x will generate hdfs-site.xml with this config, then leads to the failure
> of hbase start up.
> Should we change to build with hadoop3 while running on top of hadoop3? Since
> for branch-2.x, we will publish artifacts for hadoop3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28399 region size can be wrong from RegionSizeCalculator [hbase]
frostruan commented on code in PR #5700:
URL: https://github.com/apache/hbase/pull/5700#discussion_r1504486835
##
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/RegionSizeCalculator.java:
##
@@ -82,8 +81,8 @@ private void init(RegionLocator regionLocator, Admin admin)
throws IOException {
regionLocator.getName())) {
byte[] regionId = regionLoad.getRegionName();
-long regionSizeBytes =
- ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) *
MEGABYTE;
+long regionSizeBytes = (long)
regionLoad.getMemStoreSize().get(Size.Unit.BYTE)
Review Comment:
Yes. otherwise the data in memstore will be lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[
https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821489#comment-17821489
]
Hudson commented on HBASE-28384:
Results for branch HBASE-28384-branch-2
[build #5 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/5/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/5//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Client ingegration tests fails for branch-2/branch-2.6
> --
>
> Key: HBASE-28384
> URL: https://issues.apache.org/jira/browse/HBASE-28384
> Project: HBase
> Issue Type: Bug
> Components: hadoop3, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> In HBASE-28331, we fixed the problem that we should use the latest hadoop3
> tarballs, for generating hdfs-site.xml.
> But for branch-2.x, we still use hadoop2 when compiling hbase by default, so
> there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop
> 3.3.x will generate hdfs-site.xml with this config, then leads to the failure
> of hbase start up.
> Should we change to build with hadoop3 while running on top of hadoop3? Since
> for branch-2.x, we will publish artifacts for hadoop3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28407 [thirdparty] Update release instructions [hbase-thirdparty]
Apache9 commented on code in PR #113: URL: https://github.com/apache/hbase-thirdparty/pull/113#discussion_r1505264727 ## README.md: ## @@ -0,0 +1,115 @@ +# HBase Thirdparty + + +This project packages relocated third-party libraries used by Apache HBase. + +> DISCLAIMER: This project is for Apache HBase internal use. Included libs +> and/or their versions are subject to change at the dictate of hbase without +> regard to the concern of others! + +We have a number of submodules, one per ornery lib -- protobuf, netty, &c. -- Review Comment: OK, got it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28407 [thirdparty] Update release instructions [hbase-thirdparty]
busbey commented on code in PR #113: URL: https://github.com/apache/hbase-thirdparty/pull/113#discussion_r1505263844 ## README.md: ## @@ -0,0 +1,115 @@ +# HBase Thirdparty + + +This project packages relocated third-party libraries used by Apache HBase. + +> DISCLAIMER: This project is for Apache HBase internal use. Included libs +> and/or their versions are subject to change at the dictate of hbase without +> regard to the concern of others! + +We have a number of submodules, one per ornery lib -- protobuf, netty, &c. -- Review Comment: It's a way to say "etcetera" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28407 [thirdparty] Update release instructions [hbase-thirdparty]
Apache9 commented on code in PR #113: URL: https://github.com/apache/hbase-thirdparty/pull/113#discussion_r1505261164 ## README.md: ## @@ -0,0 +1,115 @@ +# HBase Thirdparty + + +This project packages relocated third-party libraries used by Apache HBase. + +> DISCLAIMER: This project is for Apache HBase internal use. Included libs +> and/or their versions are subject to change at the dictate of hbase without +> regard to the concern of others! + +We have a number of submodules, one per ornery lib -- protobuf, netty, &c. -- Review Comment: What does the '&c' mean? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28379 Upgrade thirdparty dep to 4.1.6 [hbase]
Apache-HBase commented on PR #5693: URL: https://github.com/apache/hbase/pull/5693#issuecomment-1967794800 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 29s | master passed | | +1 :green_heart: | compile | 1m 56s | master passed | | +1 :green_heart: | shadedjars | 6m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 11s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 58s | the patch passed | | +1 :green_heart: | compile | 2m 15s | the patch passed | | +1 :green_heart: | javac | 2m 15s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 54s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 389m 15s | root in the patch failed. | | | | 423m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5693/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5693 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3767f763a320 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c4a02f7fcd | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5693/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5693/5/testReport/ | | Max. process+thread count | 5818 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-examples hbase-shaded . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5693/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Backport "HBASE-28379 Upgrade thirdparty dep to 4.1.6 (#5693)" to branch-2.6 [hbase]
Apache-HBase commented on PR #5710: URL: https://github.com/apache/hbase/pull/5710#issuecomment-1967763497 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.6 Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 12s | branch-2.6 passed | | +1 :green_heart: | compile | 1m 59s | branch-2.6 passed | | +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 21s | branch-2.6 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 51s | the patch passed | | +1 :green_heart: | compile | 2m 4s | the patch passed | | +1 :green_heart: | javac | 2m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 20s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 378m 1s | root in the patch failed. | | | | 411m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5710 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 690c56a5623b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.6 / 9ae26c3685 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/testReport/ | | Max. process+thread count | 4723 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-shaded . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28385 make Scan estimates more realistic [hbase]
Apache-HBase commented on PR #5713: URL: https://github.com/apache/hbase/pull/5713#issuecomment-1967685269 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 8s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 6m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 6s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 6m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 296m 36s | hbase-server in the patch failed. | | | | 324m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5713 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 42dbbe60012e 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c4a02f7fcd | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/testReport/ | | Max. process+thread count | 4875 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28385 make Scan estimates more realistic [hbase]
Apache-HBase commented on PR #5713: URL: https://github.com/apache/hbase/pull/5713#issuecomment-1967675730 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 6m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 291m 10s | hbase-server in the patch passed. | | | | 320m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5713 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 413eed8289f1 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c4a02f7fcd | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/testReport/ | | Max. process+thread count | 4640 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Backport "HBASE-28379 Upgrade thirdparty dep to 4.1.6 (#5693)" to branch-2.6 [hbase]
Apache-HBase commented on PR #5710: URL: https://github.com/apache/hbase/pull/5710#issuecomment-1967560346 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.6 Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | branch-2.6 passed | | +1 :green_heart: | compile | 2m 21s | branch-2.6 passed | | +1 :green_heart: | shadedjars | 6m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 48s | branch-2.6 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | compile | 2m 30s | the patch passed | | +1 :green_heart: | javac | 2m 30s | the patch passed | | +1 :green_heart: | shadedjars | 6m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 25s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 275m 5s | root in the patch failed. | | | | 310m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5710 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f4881483f777 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.6 / 9ae26c3685 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/testReport/ | | Max. process+thread count | 5420 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-shaded . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
[
https://issues.apache.org/jira/browse/HBASE-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821400#comment-17821400
]
Rushabh Shah commented on HBASE-28391:
--
Thank you [~nihaljain.cs] and [~zhangduo] for the review. Merged the patch to
all active branches: trunk, branch-3, branch-2, branch-2.6, branch-2.5 and
branch-2.4
> Remove the need for ADMIN permissions for listDecommissionedRegionServers
> -
>
> Key: HBASE-28391
> URL: https://issues.apache.org/jira/browse/HBASE-28391
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 2.4.17, 2.5.7
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> Why we need {{ADMIN}} permissions for
> {{AccessController#preListDecommissionedRegionServers}} ?
> From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
> argument {{excludeDecommissionedRS}} is set to true. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
> If {{excludeDecommissionedRS}} is set to true and if we have
> {{AccessController}} co-proc attached, it requires ADMIN permissions to
> execute {{listDecommissionedRegionServers}} RPC. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
>
> {code:java}
> @Override
> public void
> preListDecommissionedRegionServers(ObserverContext
> ctx)
> throws IOException {
> requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
> }
> {code}
> I understand that we need ADMIN permissions for
> _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it
> changes the membership of regionservers but I don’t see any need for ADMIN
> permissions for _listDecommissionedRegionServers_. Do you think we can
> remove need for ADMIN permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
[
https://issues.apache.org/jira/browse/HBASE-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah resolved HBASE-28391.
--
Fix Version/s: 2.6.0
2.4.18
4.0.0-alpha-1
2.7.0
2.5.8
3.0.0-beta-2
Resolution: Fixed
> Remove the need for ADMIN permissions for listDecommissionedRegionServers
> -
>
> Key: HBASE-28391
> URL: https://issues.apache.org/jira/browse/HBASE-28391
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 2.4.17, 2.5.7
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> Why we need {{ADMIN}} permissions for
> {{AccessController#preListDecommissionedRegionServers}} ?
> From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
> argument {{excludeDecommissionedRS}} is set to true. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
> If {{excludeDecommissionedRS}} is set to true and if we have
> {{AccessController}} co-proc attached, it requires ADMIN permissions to
> execute {{listDecommissionedRegionServers}} RPC. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
>
> {code:java}
> @Override
> public void
> preListDecommissionedRegionServers(ObserverContext
> ctx)
> throws IOException {
> requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
> }
> {code}
> I understand that we need ADMIN permissions for
> _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it
> changes the membership of regionservers but I don’t see any need for ADMIN
> permissions for _listDecommissionedRegionServers_. Do you think we can
> remove need for ADMIN permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28385 make Scan estimates more realistic [hbase]
rmdmattingly commented on code in PR #5713:
URL: https://github.com/apache/hbase/pull/5713#discussion_r1504733844
##
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##
@@ -153,24 +175,60 @@ public void addMutation(final Mutation mutation) {
* Update estimate quota(read/write size/capacityUnits) which will be
consumed
* @param numWrites the number of write requests
* @param numReads the number of read requests
- * @param numScans the number of scan requests
*/
- protected void updateEstimateConsumeQuota(int numWrites, int numReads, int
numScans) {
+ protected void updateEstimateConsumeBatchQuota(int numWrites, int numReads) {
writeConsumed = estimateConsume(OperationType.MUTATE, numWrites, 100);
if (useResultSizeBytes) {
readConsumed = estimateConsume(OperationType.GET, numReads, 100);
- readConsumed += estimateConsume(OperationType.SCAN, numScans, 1000);
} else {
// assume 1 block required for reads. this is probably a low estimate,
which is okay
readConsumed = numReads > 0 ? blockSizeBytes : 0;
- readConsumed += numScans > 0 ? blockSizeBytes : 0;
}
writeCapacityUnitConsumed = calculateWriteCapacityUnit(writeConsumed);
readCapacityUnitConsumed = calculateReadCapacityUnit(readConsumed);
}
+ /**
+ * Update estimate quota(read/write size/capacityUnits) which will be
consumed
+ * @param scanRequest the scan to be executed
+ * @param maxScannerResultSize the maximum bytes to be returned by the
scanner
+ * @param maxBlockBytesScanned the maximum bytes scanned in a single RPC
call by the scanner
+ */
+ protected void updateEstimateConsumeScanQuota(ClientProtos.ScanRequest
scanRequest,
+long maxScannerResultSize, long maxBlockBytesScanned) {
+if (useResultSizeBytes) {
+ readConsumed = estimateConsume(OperationType.GET, 1, 1000);
+} else {
+ /*
+ * Estimating scan workload is more complicated, and if we severely
underestimate workloads
+ * then throttled clients will exhaust retries too quickly, and could
saturate the RPC layer.
+ * We have access to the ScanRequest's nextCallSeq number, the
maxScannerResultSize, and the
+ * maxBlockBytesScanned by every relevant Scanner#next call. With these
inputs we can make a
+ * more informed estimate about the scan's workload.
+ */
+ long estimate;
+ if (scanRequest.getNextCallSeq() == 0) {
+// start scanners with an optimistic 1 block IO estimate
+// it is better to underestimate a large scan in the beginning
+// than to overestimate, and block, a small scan
+estimate = blockSizeBytes;
+ } else {
+// scanner result sizes will be limited by quota availability,
regardless of
+// maxScannerResultSize. This means that we cannot safely assume that
a long-running
+// scan with a small maxBlockBytesScanned would not prefer to pull down
+// a larger payload. So we should estimate with the assumption that
long-running scans
+// are appropriately configured to approach their maxScannerResultSize
per RPC call
+estimate =
+ Math.min(maxScannerResultSize, scanRequest.getNextCallSeq() *
maxBlockBytesScanned);
Review Comment:
After some more testing, I'm realizing that we should also cap this estimate
at some maximum proportion of the quota. If a request will otherwise never
succeed based on the estimate, then we should allow it and hope for the best
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821361#comment-17821361
]
Andrew Kyle Purtell edited comment on HBASE-28405 at 2/27/24 5:49 PM:
--
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
In the rollback we can set MERGING state back to OPEN.
Also, there is a related and interesting finding.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It would be ideal if the master does not make redundant requests, but if it
does make one, the RS should handle the request and return success to the
master because the request to open an already open region on the same server is
idempotent with the earlier request that caused the region to be opened there
in the first place. So why would this hang the rollback? Maybe because today
the RS won't ack the new request? So we can change the RS code to do that if so.
{quote}If we assign the region to the same RS, it will hang the rollback, a
worse scenario is we try to assign it to another region server, then it will
lead to double assign and cause data loss
{quote}
For sure we must insure a region already OPEN on one server is never assigned
to a different server concurrently.
was (Author: apurtell):
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It would be ideal if the master does not make redundant requests, but if it
does make one, the RS should handle the request and return success to the
master because the request to open an already open region on the same server is
idempotent with the earlier request that caused the region to be opened there
in the first place. So why would this hang the rollback? Maybe because today
the RS won't ack the new request? So we can change the RS code to do that if so.
{quote}If we assign the region to the same RS, it will hang the rollback, a
worse scenario is we try to assign it to another region server, then it will
lead to double assign and cause data loss
{quote}
For sure we must insure a region already OPEN on one server is never assigned
to a different server concurrently.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbas
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821361#comment-17821361
]
Andrew Kyle Purtell edited comment on HBASE-28405 at 2/27/24 5:48 PM:
--
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It would be ideal if the master does not make redundant requests, but if it
does make one, the RS should handle the request and return success to the
master because the request to open an already open region on the same server is
idempotent with the earlier request that caused the region to be opened there
in the first place. So why would this hang the rollback? Maybe because today
the RS won't ack the new request? So we can change the RS code to do that if so.
{quote}If we assign the region to the same RS, it will hang the rollback, a
worse scenario is we try to assign it to another region server, then it will
lead to double assign and cause data loss
{quote}
For sure we must insure a region already OPEN on one server is never assigned
to a different server concurrently.
was (Author: apurtell):
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It would be ideal if the master does not make redundant requests, but if it
does make one, the RS should handle the request and return success to the
master because the request to open an already open region on the same server is
idempotent with the earlier request that caused the region to be opened there
in the first place. So why would this hang the rollback? Maybe because today
the RS won't ack the new request? So we can change the RS code to do that if so.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachi
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821361#comment-17821361
]
Andrew Kyle Purtell edited comment on HBASE-28405 at 2/27/24 5:46 PM:
--
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It would be ideal if the master does not make redundant requests, but if it
does make one, the RS should handle the request and return success to the
master because the request to open an already open region on the same server is
idempotent with the earlier request that caused the region to be opened there
in the first place. So why would this hang the rollback? Maybe because today
the RS won't ack the new request? So we can change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
was (Author: apurtell):
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It is unnecessary to make a new TRSP to assign a region already online on the
regionserver, but if we do make one, the RS should handle the request and
return success to the master because the request to open an already open region
on the same server is idempotent with the earlier request that caused the
region to be opened there in the first place. So why would this hang the
rollback? Maybe because today the RS won't ack the new request? So we can
change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecu
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821361#comment-17821361
]
Andrew Kyle Purtell edited comment on HBASE-28405 at 2/27/24 5:46 PM:
--
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It would be ideal if the master does not make redundant requests, but if it
does make one, the RS should handle the request and return success to the
master because the request to open an already open region on the same server is
idempotent with the earlier request that caused the region to be opened there
in the first place. So why would this hang the rollback? Maybe because today
the RS won't ack the new request? So we can change the RS code to do that if so.
was (Author: apurtell):
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It would be ideal if the master does not make redundant requests, but if it
does make one, the RS should handle the request and return success to the
master because the request to open an already open region on the same server is
idempotent with the earlier request that caused the region to be opened there
in the first place. So why would this hang the rollback? Maybe because today
the RS won't ack the new request? So we can change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecuto
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821361#comment-17821361
]
Andrew Kyle Purtell edited comment on HBASE-28405 at 2/27/24 5:45 PM:
--
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo]
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It is unnecessary to make a new TRSP to assign a region already online on the
regionserver, but if we do make one, the RS should handle the request and
return success to the master because the request to open an already open region
on the same server is idempotent with the earlier request that caused the
region to be opened there in the first place. So why would this hang the
rollback? Maybe because today the RS won't ack the new request? So we can
change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
was (Author: apurtell):
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo] I don't understand this part:
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It is unnecessary to make a new TRSP to assign a region already online on the
regionserver, but if we do make one, the RS should handle the request and
return success to the master because the request to open an already open region
on the same server is idempotent with the earlier request that caused the
region to be opened there in the first place. So why would this hang the
rollback? Maybe because today the RS won't ack the new request? So we can
change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821361#comment-17821361
]
Andrew Kyle Purtell edited comment on HBASE-28405 at 2/27/24 5:45 PM:
--
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
The best solution is fixing the state in the master to reflect the region is
still online on the RS after the failed merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo] I don't understand this part:
{quote}So the problem is that, we should not issue a TRSP if the region is
already online, when rollbacking the MergeTableRegionsProcedure. If we assign
the region to the same RS, it will hang the rollback
{quote}
It is unnecessary to make a new TRSP to assign a region already online on the
regionserver, but if we do make one, the RS should handle the request and
return success to the master because the request to open an already open region
on the same server is idempotent with the earlier request that caused the
region to be opened there in the first place. So why would this hang the
rollback? Maybe because today the RS won't ack the new request? So we can
change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
was (Author: apurtell):
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
+1, I think this is what Duo was getting at. The solution is fixing the state
in the master to reflect the region is still online on the RS after the failed
merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo] I don't understand this part:
bq. So the problem is that, we should not issue a TRSP if the region is already
online, when rollbacking the MergeTableRegionsProcedure. If we assign the
region to the same RS, it will hang the rollback
It is unnecessary to make a new TRSP to assign a region already online on the
regionserver, but if we do make one, the RS should handle the request and
return success to the master because the request to open an already open region
on the same server is idempotent with the earlier request that caused the
region to be opened there in the first place. So why would this hang the
rollback? Maybe because today the RS won't ack the new request. So we can
change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateM
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821361#comment-17821361
]
Andrew Kyle Purtell commented on HBASE-28405:
-
{quote}Maybe since region is online just changing the state in region state
node (meta) from MERGING to OPEN would have sifficed in such cases.
{quote}
+1, I think this is what Duo was getting at. The solution is fixing the state
in the master to reflect the region is still online on the RS after the failed
merge.
Here:
{noformat}
2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler -
Received OPEN for table1,r1,1685436252488.a92008b76ccae47d55c590930b837036.
which is already online
{noformat}
One option here is the RS can tell the master the assign succeeded, because the
OPEN request is idempotent when the region is already open on the RS.
[~zhangduo] I don't understand this part:
bq. So the problem is that, we should not issue a TRSP if the region is already
online, when rollbacking the MergeTableRegionsProcedure. If we assign the
region to the same RS, it will hang the rollback
It is unnecessary to make a new TRSP to assign a region already online on the
regionserver, but if we do make one, the RS should handle the request and
return success to the master because the request to open an already open region
on the same server is idempotent with the earlier request that caused the
region to be opened there in the first place. So why would this hang the
rollback? Maybe because today the RS won't ack the new request. So we can
change the RS code to do that if so.
Although it would be good to optimize the master so it isn't making redundant
requests.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821350#comment-17821350
]
Aman Poonia edited comment on HBASE-28405 at 2/27/24 5:23 PM:
--
[~zhangduo] Thanks for the insight.
{noformat}
The logic at RS side is that, at the end of assign a region, it will retry
forever on reporting this to master. So if we find out that the region is
already online, we should just ignore it, as we can make sure that there is
someone else will finally report it to master, to avoid double report and cause
issues.
{noformat}
When i checked the log of OpenRegionProcedure on RS there were no logs for that
rpocedure. Similarly when we look at the master logs there were no logs about
the parent procedure and its progress. SO we were stuck in this state infinitely
One another though, we had to execute TRSP but maybe not the assign because the
state of region was merging in region state node.
Dfference - when we check for state in region we use regionstatenode and when
we check if region is online on RS we use the onlineregions map of RS to see if
region is online. So basically we are looking at two different places in same
flow.
Maybe since region is online just changing the state in region state node
(meta) from MERGING to OPEN would have sifficed in such cases.
was (Author: mnpoonia):
[~zhangduo] Thanks for the insight.
{noformat}
The logic at RS side is that, at the end of assign a region, it will retry
forever on reporting this to master. So if we find out that the region is
already online, we should just ignore it, as we can make sure that there is
someone else will finally report it to master, to avoid double report and cause
issues.
{noformat}
When i checked the log of OpenRegionProcedure on RS there were no logs for that
rpocedure. Similarly when we look at the master logs there were no logs about
the parent procedure and its progress. SO we were stuck in this state infinitely
One another though, we had to execute TRSP but maybe not the assign because the
state of region was merging in region state node.
Dfference - when we check for state in region we use regionstatenode and when
we check if region is online on RS we use the onlineregions map of RS to see if
region is online. So basically we are looking at two different places in same
flow.
Maybe since region is online we just change the state in region state node
(meta) from MERGING to OPEN
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a is
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821350#comment-17821350
]
Aman Poonia edited comment on HBASE-28405 at 2/27/24 5:08 PM:
--
[~zhangduo] Thanks for the insight.
{noformat}
The logic at RS side is that, at the end of assign a region, it will retry
forever on reporting this to master. So if we find out that the region is
already online, we should just ignore it, as we can make sure that there is
someone else will finally report it to master, to avoid double report and cause
issues.
{noformat}
When i checked the log of OpenRegionProcedure on RS there were no logs for that
rpocedure. Similarly when we look at the master logs there were no logs about
the parent procedure and its progress. SO we were stuck in this state infinitely
One another though, we had to execute TRSP but maybe not the assign because the
state of region was merging in region state node.
Dfference - when we check for state in region we use regionstatenode and when
we check if region is online on RS we use the onlineregions map of RS to see if
region is online. So basically we are looking at two different places in same
flow.
Maybe since region is online we just change the state in region state node
(meta) from MERGING to OPEN
was (Author: mnpoonia):
[~zhangduo] Thanks for the insight.
{noformat}
The logic at RS side is that, at the end of assign a region, it will retry
forever on reporting this to master. So if we find out that the region is
already online, we should just ignore it, as we can make sure that there is
someone else will finally report it to master, to avoid double report and cause
issues.
{noformat}
When i checked the log of OpenRegionProcedure on RS there were no logs for that
rpocedure. Similarly when we look at the master logs there were no logs about
the parent procedure and its progress. SO we were stuck in this state infinitely
One another though we had to execute TRSP but maybe not the assign because the
state of region was merging in region state node.
This is the difference. when we check for state in region we use
regionstatenode and when we check if region is online on RS we use the
onlineregions map of RS to see if region is online. So basically we are looking
at two different places in same flow.
Maybe since region is online we just change the state in region state node
(meta) from MERGING to OPEN
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821350#comment-17821350
]
Aman Poonia commented on HBASE-28405:
-
[~zhangduo] Thanks for the insight.
{noformat}
The logic at RS side is that, at the end of assign a region, it will retry
forever on reporting this to master. So if we find out that the region is
already online, we should just ignore it, as we can make sure that there is
someone else will finally report it to master, to avoid double report and cause
issues.
{noformat}
When i checked the log of OpenRegionProcedure on RS there were no logs for that
rpocedure. Similarly when we look at the master logs there were no logs about
the parent procedure and its progress. SO we were stuck in this state infinitely
One another though we had to execute TRSP but maybe not the assign because the
state of region was merging in region state node.
This is the difference. when we check for state in region we use
regionstatenode and when we check if region is online on RS we use the
onlineregions map of RS to see if region is online. So basically we are looking
at two different places in same flow.
Maybe since region is online we just change the state in region state node
(meta) from MERGING to OPEN
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to open the region a92008b76ccae47d55c590930b837036.
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of pr
Re: [PR] HBASE-28385 make Scan estimates more realistic [hbase]
Apache-HBase commented on PR #5713: URL: https://github.com/apache/hbase/pull/5713#issuecomment-1967159348 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 2m 27s | master passed | | +1 :green_heart: | checkstyle | 0m 38s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 33s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | -1 :x: | compile | 2m 23s | hbase-server in the patch failed. | | -0 :warning: | javac | 2m 23s | hbase-server in the patch failed. | | -0 :warning: | checkstyle | 0m 36s | hbase-server: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 55s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | -1 :x: | spotbugs | 1m 43s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 28m 23s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerHolder.getMaxBlockBytesScanned() is unsynchronized, org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerHolder.setMaxBlockBytesScanned(long) is synchronized At RSRpcServices.java:synchronized At RSRpcServices.java:[line 457] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5713 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 25b7de5aabd8 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c4a02f7fcd | | Default Java | Eclipse Adoptium-11.0.17+8 | | compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5713/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28391 Remove the need for ADMIN permissions for listDecommissionedRegionServers [hbase]
shahrs87 merged PR #5695: URL: https://github.com/apache/hbase/pull/5695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28391 Remove the need for ADMIN permissions for listDecommissionedRegionServers [hbase]
shahrs87 commented on code in PR #5695: URL: https://github.com/apache/hbase/pull/5695#discussion_r1504611977 ## hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java: ## @@ -1203,12 +1203,6 @@ public void preDecommissionRegionServers(ObserverContext
Re: [PR] HBASE-28407 [thirdparty] Update release instructions [hbase-thirdparty]
Apache-HBase commented on PR #113: URL: https://github.com/apache/hbase-thirdparty/pull/113#issuecomment-1966996039 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 18s | ASF License check generated no output? | | | | 2m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-113/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/113 | | Optional Tests | dupname asflicense markdownlint | | uname | Linux 90af66ffd0c2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 293203f | | Max. process+thread count | 8 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-113/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28385) Quota estimates are too optimistic for large scans
[ https://issues.apache.org/jira/browse/HBASE-28385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-28385: --- Labels: pull-request-available (was: ) > Quota estimates are too optimistic for large scans > -- > > Key: HBASE-28385 > URL: https://issues.apache.org/jira/browse/HBASE-28385 > Project: HBase > Issue Type: Improvement >Reporter: Ray Mattingly >Assignee: Ray Mattingly >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0 > > > Let's say you're running a table scan with a throttle of 100MB/sec per > RegionServer. Ideally your scans are going to pull down large results, often > containing hundreds or thousands of blocks. > You will estimate each scan as costing a single block of read capacity, and > if your quota is already exhausted then the server will evaluate the backoff > required for your estimated consumption (1 block) to be available. This will > often be ~1ms, causing your retries to basically be immediate. > Obviously it will routinely take much longer than 1ms for 100MB of IO to > become available in the given configuration, so your retries will be destined > to fail. At worst this can cause a saturation of your server's RPC layer, and > at best this causes erroneous exhaustion of the client's retries. > We should find a way to make these estimates a bit smarter for large scans. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] HBASE-28385 make Scan estimates more realistic [hbase]
rmdmattingly opened a new pull request, #5713: URL: https://github.com/apache/hbase/pull/5713 ## Background https://issues.apache.org/jira/browse/HBASE-28385 Let's say you're running a table scan with a throttle of 100MB/sec per RegionServer. Ideally your scans are going to pull down large results, often containing hundreds or thousands of blocks. You will estimate each scan as costing a single block of read capacity, and if your quota is already exhausted then the server will evaluate the backoff required for your estimated consumption (1 block) to be available. This will often be ~1ms, causing your retries to basically be immediate. Obviously it will routinely take much longer than 1ms for 100MB of IO to become available in the given configuration, so your retries will be destined to fail. At worst this can cause a saturation of your server's RPC layer, and at best this causes erroneous exhaustion of the client's retries. ## Proposal This makes two major changes: 1. It introduces a minimum waitInterval of 100ms. Throttling with a near-zero backoff is not useful. If we’ve reached the point of quota saturation then there should be a minimum backoff that must at least be thrown. 2. It introduces a more complex estimation of scan workloads. Specifically: * We continue to estimate initial scan calls very optimistically at, or near, 1 block of IO. * We begin tracking the max block bytes scanned (`maxBlockBytesScanned`) by a single `Scanner#next` call for each scanner in the `RegionScannerHolder`. * Keep in mind that there is already a server configured maxResultSize for scanners, and a call sequence number which increments with each `Scanner#next` call, beginning with 0 * With all of these inputs, we estimate scan workload to be `Math.min(maxResultSize, nextSeqNum*maxBlockBytesScanned)` ## Open questions - [ ] Should the new minimum waitInterval be configurable? ## Testing I've deployed this to a test cluster and confirmed that large scan estimates quickly produced useful estimates and, consequently, meaningful waitInterval backoffs. I can also try to write a unit test which confirms this behavior. cc @hgromer @eab148 @bozzkar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28407) [thirdparty] Update release instructions
[ https://issues.apache.org/jira/browse/HBASE-28407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-28407: - Status: Patch Available (was: Open) > [thirdparty] Update release instructions > > > Key: HBASE-28407 > URL: https://issues.apache.org/jira/browse/HBASE-28407 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Our release instructions for hbase-thirdparty are out of date. Update them > based on recent experience. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] HBASE-28407 [thirdparty] Update release instructions [hbase-thirdparty]
ndimiduk opened a new pull request, #113: URL: https://github.com/apache/hbase-thirdparty/pull/113 @Apache9 @busbey This is the processes that I (eventually) sorted out. Does it look about right from your recollection? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Backport "HBASE-28379 Upgrade thirdparty dep to 4.1.6 (#5693)" to branch-2.6 [hbase]
Apache-HBase commented on PR #5710: URL: https://github.com/apache/hbase/pull/5710#issuecomment-1966922160 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 55s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.6 Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 26s | branch-2.6 passed | | +1 :green_heart: | compile | 7m 25s | branch-2.6 passed | | +1 :green_heart: | spotless | 0m 54s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 7m 23s | the patch passed | | +1 :green_heart: | javac | 7m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 5s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 10m 52s | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6. | | +1 :green_heart: | spotless | 0m 52s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 40m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5710 | | Optional Tests | dupname asflicense javac hadoopcheck spotless xml compile | | uname | Linux bcf1077de020 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.6 / 9ae26c3685 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-shaded . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5710/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28379 Upgrade thirdparty dep to 4.1.6 [hbase]
Apache-HBase commented on PR #5693: URL: https://github.com/apache/hbase/pull/5693#issuecomment-1966921294 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 52s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 25s | master passed | | +1 :green_heart: | compile | 7m 8s | master passed | | +1 :green_heart: | spotless | 0m 52s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 9s | the patch passed | | +1 :green_heart: | compile | 6m 47s | the patch passed | | +1 :green_heart: | javac | 6m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 10s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 5m 54s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 51s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 39m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5693/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5693 | | Optional Tests | dupname asflicense javac hadoopcheck spotless xml compile | | uname | Linux cff1a651f05d 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c4a02f7fcd | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-examples hbase-shaded . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5693/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-28407) [thirdparty] Update release instructions
Nick Dimiduk created HBASE-28407: Summary: [thirdparty] Update release instructions Key: HBASE-28407 URL: https://issues.apache.org/jira/browse/HBASE-28407 Project: HBase Issue Type: Task Components: thirdparty Reporter: Nick Dimiduk Assignee: Nick Dimiduk Our release instructions for hbase-thirdparty are out of date. Update them based on recent experience. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28321 RpcConnectionRegistry is broken when security is enabled … [hbase]
Apache-HBase commented on PR #5707: URL: https://github.com/apache/hbase/pull/5707#issuecomment-1966895219 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 55s | branch-2 passed | | +1 :green_heart: | compile | 2m 6s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 11s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 48s | the patch passed | | +1 :green_heart: | compile | 2m 9s | the patch passed | | +1 :green_heart: | javac | 2m 9s | the patch passed | | +1 :green_heart: | shadedjars | 6m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 45s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 7m 41s | hbase-client in the patch passed. | | -1 :x: | unit | 271m 57s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 2m 45s | hbase-examples in the patch passed. | | | | 314m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5707 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9b12e074c66d 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ac01d72d94 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/testReport/ | | Max. process+thread count | 4195 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-examples U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28321 RpcConnectionRegistry is broken when security is enabled … [hbase]
Apache-HBase commented on PR #5707: URL: https://github.com/apache/hbase/pull/5707#issuecomment-1966868699 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 16s | branch-2 passed | | +1 :green_heart: | compile | 1m 47s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 13s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 8s | the patch passed | | +1 :green_heart: | compile | 2m 16s | the patch passed | | +1 :green_heart: | javac | 2m 16s | the patch passed | | +1 :green_heart: | shadedjars | 5m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 35s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 7m 46s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 262m 12s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 2m 45s | hbase-examples in the patch passed. | | | | 302m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5707 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 14b28c0a72d6 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ac01d72d94 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/testReport/ | | Max. process+thread count | 3944 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-examples U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] (HBASE-28399) region size can be wrong from RegionSizeCalculator
[ https://issues.apache.org/jira/browse/HBASE-28399 ]
ruanhui deleted comment on HBASE-28399:
-
was (Author: frostruan):
HBASE-26609 may have solved this problem. A little strange Let me digger
more
> region size can be wrong from RegionSizeCalculator
> --
>
> Key: HBASE-28399
> URL: https://issues.apache.org/jira/browse/HBASE-28399
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 3.0.0-beta-1
>Reporter: ruanhui
>Assignee: ruanhui
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0-beta-2
>
>
> The RegionSizeCalculator calculates region byte size using the following
> method
> {code:java}
> private static final long MEGABYTE = 1024L * 1024L;
> long regionSizeBytes =
> ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) * MEGABYTE;
> {code}
> However, this method will lose accuracy. For example, the result of
> {code:java}
> ((long) new Size(1, Size.Unit.BYTE).get(Size.Unit.MEGABYTE)) * MEGABYTE {code}
> is 0. This will result in a TableInputSplit with a length of 0, but in fact
> this TableInputSplit has a small amount of data.
>
> This TableInputSplit will be ignored if we enable
> spark.hadoopRDD.ignoreEmptySplits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28399 region size can be wrong from RegionSizeCalculator [hbase]
frostruan commented on code in PR #5700:
URL: https://github.com/apache/hbase/pull/5700#discussion_r1504486835
##
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/RegionSizeCalculator.java:
##
@@ -82,8 +81,8 @@ private void init(RegionLocator regionLocator, Admin admin)
throws IOException {
regionLocator.getName())) {
byte[] regionId = regionLoad.getRegionName();
-long regionSizeBytes =
- ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) *
MEGABYTE;
+long regionSizeBytes = (long)
regionLoad.getMemStoreSize().get(Size.Unit.BYTE)
Review Comment:
Yes. The data in memstore will be lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-28399) region size can be wrong from RegionSizeCalculator
[
https://issues.apache.org/jira/browse/HBASE-28399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821304#comment-17821304
]
ruanhui commented on HBASE-28399:
-
HBASE-26609 may have solved this problem. A little strange Let me digger
more
> region size can be wrong from RegionSizeCalculator
> --
>
> Key: HBASE-28399
> URL: https://issues.apache.org/jira/browse/HBASE-28399
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 3.0.0-beta-1
>Reporter: ruanhui
>Assignee: ruanhui
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0-beta-2
>
>
> The RegionSizeCalculator calculates region byte size using the following
> method
> {code:java}
> private static final long MEGABYTE = 1024L * 1024L;
> long regionSizeBytes =
> ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) * MEGABYTE;
> {code}
> However, this method will lose accuracy. For example, the result of
> {code:java}
> ((long) new Size(1, Size.Unit.BYTE).get(Size.Unit.MEGABYTE)) * MEGABYTE {code}
> is 0. This will result in a TableInputSplit with a length of 0, but in fact
> this TableInputSplit has a small amount of data.
>
> This TableInputSplit will be ignored if we enable
> spark.hadoopRDD.ignoreEmptySplits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28399) region size can be wrong from RegionSizeCalculator
[
https://issues.apache.org/jira/browse/HBASE-28399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821287#comment-17821287
]
Duo Zhang commented on HBASE-28399:
---
In general, the precision we used to calcuate the store file size metrics is
per MB, so using a smaller unit does not help here...
> region size can be wrong from RegionSizeCalculator
> --
>
> Key: HBASE-28399
> URL: https://issues.apache.org/jira/browse/HBASE-28399
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 3.0.0-beta-1
>Reporter: ruanhui
>Assignee: ruanhui
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0-beta-2
>
>
> The RegionSizeCalculator calculates region byte size using the following
> method
> {code:java}
> private static final long MEGABYTE = 1024L * 1024L;
> long regionSizeBytes =
> ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) * MEGABYTE;
> {code}
> However, this method will lose accuracy. For example, the result of
> {code:java}
> ((long) new Size(1, Size.Unit.BYTE).get(Size.Unit.MEGABYTE)) * MEGABYTE {code}
> is 0. This will result in a TableInputSplit with a length of 0, but in fact
> this TableInputSplit has a small amount of data.
>
> This TableInputSplit will be ignored if we enable
> spark.hadoopRDD.ignoreEmptySplits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28399 region size can be wrong from RegionSizeCalculator [hbase]
Apache9 commented on code in PR #5700:
URL: https://github.com/apache/hbase/pull/5700#discussion_r1504404795
##
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/RegionSizeCalculator.java:
##
@@ -82,8 +81,8 @@ private void init(RegionLocator regionLocator, Admin admin)
throws IOException {
regionLocator.getName())) {
byte[] regionId = regionLoad.getRegionName();
-long regionSizeBytes =
- ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) *
MEGABYTE;
+long regionSizeBytes = (long)
regionLoad.getMemStoreSize().get(Size.Unit.BYTE)
Review Comment:
But anyway, I think the problem here is we also need to account memstore
size when calculating region size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28399 region size can be wrong from RegionSizeCalculator [hbase]
Apache9 commented on code in PR #5700:
URL: https://github.com/apache/hbase/pull/5700#discussion_r1504402271
##
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/RegionSizeCalculator.java:
##
@@ -82,8 +81,8 @@ private void init(RegionLocator regionLocator, Admin admin)
throws IOException {
regionLocator.getName())) {
byte[] regionId = regionLoad.getRegionName();
-long regionSizeBytes =
- ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) *
MEGABYTE;
+long regionSizeBytes = (long)
regionLoad.getMemStoreSize().get(Size.Unit.BYTE)
Review Comment:
OK, checked the code on how we constructor the store file size and memstore
size, the default unit is MB, so it is useless to pass an unit less than MB
here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28354 RegionSizeCalculator throws NPE when regions are in transition [hbase]
frostruan commented on PR #5699: URL: https://github.com/apache/hbase/pull/5699#issuecomment-1966711401 > 1. If it held 1, it would mean that regions that no longer exist would still get reported with size (maybe?) I don't think regions in state transition and regions no longer exist are same thing. When we call regionLocator.getAllRegionLocations(), we will exlcude offlined split parent regions. You can see https://github.com/apache/hbase/blob/rel/3.0.0-beta-1/hbase-client/src/main/java/org/apache/hadoop/hbase/ClientMetaTableAccessor.java#L172 for details. 2. I don't think it is appropriate to return the average size of other regions, we should not make any decisions for the user. > 3. What if there are currently no regions available? I don't think this is a big deal. As @ndimiduk mentioned, it's just a hint for MapReduce scheduler. Thanks. @aalhour -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[
https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821259#comment-17821259
]
Hudson commented on HBASE-28384:
Results for branch HBASE-28384-branch-2
[build #4 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/4//console].
> Client ingegration tests fails for branch-2/branch-2.6
> --
>
> Key: HBASE-28384
> URL: https://issues.apache.org/jira/browse/HBASE-28384
> Project: HBase
> Issue Type: Bug
> Components: hadoop3, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> In HBASE-28331, we fixed the problem that we should use the latest hadoop3
> tarballs, for generating hdfs-site.xml.
> But for branch-2.x, we still use hadoop2 when compiling hbase by default, so
> there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop
> 3.3.x will generate hdfs-site.xml with this config, then leads to the failure
> of hbase start up.
> Should we change to build with hadoop3 while running on top of hadoop3? Since
> for branch-2.x, we will publish artifacts for hadoop3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28149 move_servers_namespaces_rsgroup is not changing the new r… [hbase]
chrajeshbabu commented on PR #5661: URL: https://github.com/apache/hbase/pull/5661#issuecomment-1966574426 > > @Apache9 Thanks for response. Even I am thinking the same while moving namespaces to the new RS group would be better to have following options. > > ``` > > 1. **Move All Tables:** This option involves moving all tables within a namespace to the new RS group, regardless of the current RS group they belong to. This would effectively consolidate all tables of the namespace into the new RS group. This is the same as the current implementation so it can be default behavior. > > > > 2. **Move Tables present in current RS group**: With this option, only tables belonging to the current RS group of the namespace would be moved to the new RS group. This provides more granular control, allowing users to choose specific tables to move based on their needs. If the namespace does not belong to any RS group, the namespace tables in the default RS group would be moved to the new RS group. > > > > 3. **Move No Tables:** This option involves changing the RS group of the namespace without moving any tables. Existing tables would remain in their current RS group. This could be useful if there's a desire to separate the namespaces by RS group but without immediately moving the tables. > > ``` > > > > > > > > > > > > > > > > > > > > > > > > With this options we can give control to user choose proper option based on the requirements. Will start working on it. Happy to add if any new options possible. > > We wanted to use RS Groups for managing multiple tenants data in same cluster currently a namespace belongs to a tenant so first option is best fit for us. > > @Apache9 wdyt? Will create separate JIRA for this and work on it if it's fine. created HBASE-28406 to support this. Can we merge this to support existing behaviour till then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-28406) Move namespace to RS group can support multiple options to move tables under it
Rajeshbabu Chintaguntla created HBASE-28406:
---
Summary: Move namespace to RS group can support multiple options
to move tables under it
Key: HBASE-28406
URL: https://issues.apache.org/jira/browse/HBASE-28406
Project: HBase
Issue Type: Improvement
Components: rsgroup
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Fix For: 4.0.0-alpha-1, 3.0.0-beta-2
As discussed here
[https://github.com/apache/hbase/pull/5661#issuecomment-1949090826]
Move namespaces to rsgroup can have multiple options like:
# *Move All Tables:* This option involves moving all tables within a namespace
to the new RS group, regardless of the current RS group they belong to. This
would effectively consolidate all tables of the namespace into the new RS
group. This is the same as the current implementation so it can be default
behavior.
# {*}Move Tables present in current RS group{*}: With this option, only tables
belonging to the current RS group of the namespace would be moved to the new RS
group. This provides more granular control, allowing users to choose specific
tables to move based on their needs. If the namespace does not belong to any RS
group, the namespace tables in the default RS group would be moved to the new
RS group.
# *Move No Tables:* This option involves changing the RS group of the
namespace without moving any tables. Existing tables would remain in their
current RS group. This could be useful if there's a desire to separate the
namespaces by RS group but without immediately moving the tables.
With this options we can give control to user choose proper option based on the
requirements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[
https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821234#comment-17821234
]
Hudson commented on HBASE-28384:
Results for branch HBASE-28384-branch-2
[build #3 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/3/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/3//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Client ingegration tests fails for branch-2/branch-2.6
> --
>
> Key: HBASE-28384
> URL: https://issues.apache.org/jira/browse/HBASE-28384
> Project: HBase
> Issue Type: Bug
> Components: hadoop3, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> In HBASE-28331, we fixed the problem that we should use the latest hadoop3
> tarballs, for generating hdfs-site.xml.
> But for branch-2.x, we still use hadoop2 when compiling hbase by default, so
> there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop
> 3.3.x will generate hdfs-site.xml with this config, then leads to the failure
> of hbase start up.
> Should we change to build with hadoop3 while running on top of hadoop3? Since
> for branch-2.x, we will publish artifacts for hadoop3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821232#comment-17821232
]
Duo Zhang commented on HBASE-28405:
---
The logic at RS side is that, at the end of assign a region, it will retry
forever on reporting this to master. So if we find out that the region is
already online, we should just ignore it, as we can make sure that there is
someone else will finally report it to master, to avoid double report and cause
issues.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to open the region a92008b76ccae47d55c590930b837036.
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of procedure. When we run TRSP
> it sends a OpenRegionProcedure which is handled by AssignRegionHandler. This
> handlers on execution suggests that region is already online*
> Sequence of events are as follow
> _2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore -
> pid=26674602 updating hbase:meta row=a92008b76ccae47d55c590930b837036,
> regionState=OPENING, regionLocation=rs-210,60020,1707596461539_
> _2024-02-11 10:53:58,920 INFO [PEWorker-58] procedure2.ProcedureExecutor -
> Initialized subprocedures=[\\{pid=26675798, ppid=26674602, state=RUNNABLE;
> OpenRegionProcedure a92008b76ccae47d55c590930b837036,
> server=rs-210,60020,1707596461539}]_
> _2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
> handler.AssignRegionHandler - Received OPEN for
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821231#comment-17821231
]
Duo Zhang commented on HBASE-28405:
---
So the problem is that, we should not issue a TRSP if the region is already
online, when rollbacking the MergeTableRegionsProcedure. If we assign the
region to the same RS, it will hang the rollback, a worse scenario is we try to
assign it to another region server, then it will lead to double assign and
cause data loss...
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to open the region a92008b76ccae47d55c590930b837036.
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of procedure. When we run TRSP
> it sends a OpenRegionProcedure which is handled by AssignRegionHandler. This
> handlers on execution suggests that region is already online*
> Sequence of events are as follow
> _2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore -
> pid=26674602 updating hbase:meta row=a92008b76ccae47d55c590930b837036,
> regionState=OPENING, regionLocation=rs-210,60020,1707596461539_
> _2024-02-11 10:53:58,920 INFO [PEWorker-58] procedure2.ProcedureExecutor -
> Initialized subprocedures=[\\{pid=26675798, ppid=26674602, state=RUNNABLE;
> OpenRegionProcedure a92008b76ccae47d55c590930b837036,
> server=rs-210,60020,1707596461539}]_
> _2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
> handler.AssignRegionHandler - Received
[jira] [Comment Edited] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821227#comment-17821227
]
Aman Poonia edited comment on HBASE-28405 at 2/27/24 1:14 PM:
--
If we had executed postDeploy here the region would have come out of RIT.
{code:java}
// code placeholder
// From here on out, this is PONR. We can not revert back. The only way to
address an
// exception from here on out is to abort the region server.
rs.postOpenDeployTasks(new PostOpenDeployContext(region, openProcId,
masterSystemTime));
rs.addRegion(region);
LOG.info("Opened {}", regionName);
// Cache the open region procedure id after report region transition succeed.
rs.finishRegionProcedure(openProcId);
Boolean current =
rs.getRegionsInTransitionInRS().remove(regionInfo.getEncodedNameAsBytes());
if (current == null) {
// Should NEVER happen, but let's be paranoid.
LOG.error("Bad state: we've just opened {} which was NOT in transition",
regionName);
} else if (!current) {
// Should NEVER happen, but let's be paranoid.
LOG.error("Bad state: we've just opened {} which was closing", regionName);
} {code}
I don't see any harm in using the above piece of code in this scenario. Even
if we do this multiple times this piece of code seems idempotent.
was (Author: mnpoonia):
If we had executed postDeploy here the region would have come out of RIT.
{code:java}
// code placeholder
// From here on out, this is PONR. We can not revert back. The only way to
address an
// exception from here on out is to abort the region server.
rs.postOpenDeployTasks(new PostOpenDeployContext(region, openProcId,
masterSystemTime));
rs.addRegion(region);
LOG.info("Opened {}", regionName);
// Cache the open region procedure id after report region transition succeed.
rs.finishRegionProcedure(openProcId);
Boolean current =
rs.getRegionsInTransitionInRS().remove(regionInfo.getEncodedNameAsBytes());
if (current == null) {
// Should NEVER happen, but let's be paranoid.
LOG.error("Bad state: we've just opened {} which was NOT in transition",
regionName);
} else if (!current) {
// Should NEVER happen, but let's be paranoid.
LOG.error("Bad state: we've just opened {} which was closing", regionName);
} {code}
I don't see any harm in executing the above piece of code in this scenario.
Even if we do this multiple times this piece of code seems idempotent.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
>
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821227#comment-17821227
]
Aman Poonia commented on HBASE-28405:
-
If we had executed postDeploy here the region would have come out of RIT.
{code:java}
// code placeholder
// From here on out, this is PONR. We can not revert back. The only way to
address an
// exception from here on out is to abort the region server.
rs.postOpenDeployTasks(new PostOpenDeployContext(region, openProcId,
masterSystemTime));
rs.addRegion(region);
LOG.info("Opened {}", regionName);
// Cache the open region procedure id after report region transition succeed.
rs.finishRegionProcedure(openProcId);
Boolean current =
rs.getRegionsInTransitionInRS().remove(regionInfo.getEncodedNameAsBytes());
if (current == null) {
// Should NEVER happen, but let's be paranoid.
LOG.error("Bad state: we've just opened {} which was NOT in transition",
regionName);
} else if (!current) {
// Should NEVER happen, but let's be paranoid.
LOG.error("Bad state: we've just opened {} which was closing", regionName);
} {code}
I don't see any harm in executing the above piece of code in this scenario.
Even if we do this multiple times this piece of code seems idempotent.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to open the region a92008b76ccae47d55c590930b837036.
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of procedure. When we run TRSP
>
[jira] [Updated] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Poonia updated HBASE-28405:
Description:
*We had a scenario in production where a merge operation had failed as below*
_2024-02-11 10:53:57,715 ERROR [PEWorker-31]
assignment.MergeTableRegionsProcedure - Error trying to merge
[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in table1
(in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
_org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
location=rs-229,60020,1707587658182, table=table1,
region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
_at
org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
_at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
_at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
_at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
*Now when we do rollback of failed merge operation we see a issue where region
is in state opened until the RS holding it stopped.*
Rollback create a TRSP as below
_2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
Stored [pid=26674602,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
TransitRegionStateProcedure table=table1,
region=a92008b76ccae47d55c590930b837036, ASSIGN]_
*and rollback finished successfully*
_2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
Rolled back pid=26673594, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.HBaseIOException via
master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
MergeTableRegionsProcedure table=table1,
regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
force=false exec-time=1.4820 sec_
*We create a procedure to open the region a92008b76ccae47d55c590930b837036.
Intrestingly we didnt close the region as creation of procedure to close
regions had thrown exception and not execution of procedure. When we run TRSP
it sends a OpenRegionProcedure which is handled by AssignRegionHandler. This
handlers on execution suggests that region is already online*
Sequence of events are as follow
_2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore -
pid=26674602 updating hbase:meta row=a92008b76ccae47d55c590930b837036,
regionState=OPENING, regionLocation=rs-210,60020,1707596461539_
_2024-02-11 10:53:58,920 INFO [PEWorker-58] procedure2.ProcedureExecutor -
Initialized subprocedures=[\\{pid=26675798, ppid=26674602, state=RUNNABLE;
OpenRegionProcedure a92008b76ccae47d55c590930b837036,
server=rs-210,60020,1707596461539}]_
_2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler - Received OPEN for
table1,r1,1685436252488.a92008b76ccae47d55c590930b837036. which is already
online_
was:
We had a scenario in production where a merge operation had failed as below
_2024-02-11 10:53:57,715 ERROR [PEWorker-31]
assignment.MergeTableRegionsProcedure - Error trying to merge
[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in table1
(in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
_org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
location=rs-229,60020,1707587658182, table=table1,
region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
_at
org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableReg
Re: [PR] HBASE-28354 RegionSizeCalculator throws NPE when regions are in transition [hbase]
frostruan commented on PR #5699: URL: https://github.com/apache/hbase/pull/5699#issuecomment-1966499225 I prefer to use 1 byte to represent unknown region size for two reasons: 1. it cannot be a valid region size because even a minimal keyvalue is bigger than that. 2. some computing engines, like spark, will filter out splits with size less than 0. You can see this for details. https://github.com/apache/spark/blob/v3.5.1-rc2/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L138 I'll try to answer your questions later, sorry have to catch the shuttle bus. Thanks. @aalhour -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821214#comment-17821214
]
Aman Poonia commented on HBASE-28405:
-
Looking at the code comments it says that it might be a mistake and master is
checking on RS again and last response has not reached to HMaster yet. But our
current issue seems to be little different. Here first call itself is returned
without any reportRegionStateTransition
{code:java}
// code placeholder
String regionName = regionInfo.getRegionNameAsString();
Region onlineRegion = rs.getRegion(encodedName);
if (onlineRegion != null) {
LOG.warn("Received OPEN for {} which is already online", regionName);
// Just follow the old behavior, do we need to call
reportRegionStateTransition? Maybe not?
// For normal case, it could happen that the rpc call to schedule this
handler is succeeded,
// but before returning to master the connection is broken. And when master
tries again, we
// have already finished the opening. For this case we do not need to call
// reportRegionStateTransition any more.
return;
}
Boolean previous =
rs.getRegionsInTransitionInRS().putIfAbsent(encodedNameBytes, Boolean.TRUE);
{code}
[~apurtell] [~vjasani] FYI
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> We had a scenario in production where a merge operation had failed as below
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> Now when we do rollback of failed merge operation we see a issue where region
> is in state opened until the RS holding it stopped.
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> and rollback finished successfully
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> We create a procedure to open the region a92008b76ccae47d55c590930b837036
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of procedure.
> Now when we run TRSP it sends a OpenRegion
[jira] [Updated] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Poonia updated HBASE-28405:
Summary: Region open procedure silently returns without notifying the
parent proc (was: Region open procedure silently does nothing without
notifying the parent proc)
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.5.7
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
>
> We had a scenario in production where a merge operation had failed as below
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> Now when we do rollback of failed merge operation we see a issue where region
> is in state opened until the RS holding it stopped.
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> and rollback finished successfully
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> We create a procedure to open the region a92008b76ccae47d55c590930b837036
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of procedure.
> Now when we run TRSP it sends a OpenRegionProcedure which is handled by
> AssignRegionHandler
> This handlers on execution suggests that region is already online
> Sequence of events are as follow
> _2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore -
> pid=26674602 updating hbase:meta row=a92008b76ccae47d55c590930b837036,
> regionState=OPENING, regionLocation=rs-210,60020,1707596461539_
> _2024-02-11 10:53:58,920 INFO [PEWorker-58] procedure2.ProcedureExecutor -
> Initialized subprocedures=[\{pid=26675798, ppid=26674602, state=RUNNABLE;
> OpenRegionProcedure a92008b76ccae47d55c590930b837036,
> server=rs-210,60020,1707596461539}]_
> _2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
> handler.AssignRegionHandler - Received OPEN for
> table1,r1,1685436252488.a92008b76ccae47d55c590930b837036. which is already
> online_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28405) Region open procedure silently does nothing without notifying the parent proc
Aman Poonia created HBASE-28405:
---
Summary: Region open procedure silently does nothing without
notifying the parent proc
Key: HBASE-28405
URL: https://issues.apache.org/jira/browse/HBASE-28405
Project: HBase
Issue Type: Bug
Components: proc-v2
Affects Versions: 2.5.7
Reporter: Aman Poonia
Assignee: Aman Poonia
We had a scenario in production where a merge operation had failed as below
_2024-02-11 10:53:57,715 ERROR [PEWorker-31]
assignment.MergeTableRegionsProcedure - Error trying to merge
[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in table1
(in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
_org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
location=rs-229,60020,1707587658182, table=table1,
region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
_at
org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
_at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
_at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
_at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
_at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
_at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
Now when we do rollback of failed merge operation we see a issue where region
is in state opened until the RS holding it stopped.
Rollback create a TRSP as below
_2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
Stored [pid=26674602,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
TransitRegionStateProcedure table=table1,
region=a92008b76ccae47d55c590930b837036, ASSIGN]_
and rollback finished successfully
_2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
Rolled back pid=26673594, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.HBaseIOException via
master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
MergeTableRegionsProcedure table=table1,
regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
force=false exec-time=1.4820 sec_
We create a procedure to open the region a92008b76ccae47d55c590930b837036
Intrestingly we didnt close the region as creation of procedure to close
regions had thrown exception and not execution of procedure.
Now when we run TRSP it sends a OpenRegionProcedure which is handled by
AssignRegionHandler
This handlers on execution suggests that region is already online
Sequence of events are as follow
_2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore -
pid=26674602 updating hbase:meta row=a92008b76ccae47d55c590930b837036,
regionState=OPENING, regionLocation=rs-210,60020,1707596461539_
_2024-02-11 10:53:58,920 INFO [PEWorker-58] procedure2.ProcedureExecutor -
Initialized subprocedures=[\{pid=26675798, ppid=26674602, state=RUNNABLE;
OpenRegionProcedure a92008b76ccae47d55c590930b837036,
server=rs-210,60020,1707596461539}]_
_2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
handler.AssignRegionHandler - Received OPEN for
table1,r1,1685436252488.a92008b76ccae47d55c590930b837036. which is already
online_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28354 RegionSizeCalculator throws NPE when regions are in transition [hbase]
aalhour commented on PR #5699: URL: https://github.com/apache/hbase/pull/5699#issuecomment-1966464798 I'm wondering what the byte constant would hold, other than `0`: - If it held `1`, it would mean that regions that no longer exist would still get reported with size (maybe?) - I'm having trouble understanding tests about inexistent regions and how HBase reacts to them eventually - If it held `-1`, we'd need to refactor the callers to handle the `< 0` case, which will make "UNKNOWN REGIONS" more explicit with comments Alternatively, if we return the average size of all other regions might be good but it will push the idea of a region in transition too deep into the RegionSizeCalculator and other areas won't know about it, maybe they should? What if there are currently no regions available? What if there is one (or more) region in transition and the size map is empty? What if the size map only contains a region that's empty? I am not sure if these cases are realistic, I'm still new to the codebase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28292 Make Delay prefetch property to be dynamically configured [hbase]
wchevreuil commented on code in PR #5605:
URL: https://github.com/apache/hbase/pull/5605#discussion_r1504055111
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java:
##
@@ -78,7 +85,10 @@ public Thread newThread(Runnable r) {
+ Path.SEPARATOR_CHAR + ")|(" + Path.SEPARATOR_CHAR
+ HConstants.HREGION_COMPACTIONDIR_NAME.replace(".", "\\.") +
Path.SEPARATOR_CHAR + ")");
- public static void request(Path path, Runnable runnable) {
+ // For tests. Contains computed prefetch delay
+ private static long computedPrefetchDelay;
+
+ public static void request(Path path, boolean isInterrupted, Runnable
runnable) {
Review Comment:
Why do we need the additional parameter? Just remove the future from the
collection in the "interrupt" method.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java:
##
@@ -127,11 +153,44 @@ public static boolean isCompleted(Path path) {
return true;
}
- private PrefetchExecutor() {
Review Comment:
Why are we making this public now?
##
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java:
##
@@ -106,10 +115,19 @@ public class TestPrefetch {
@Before
public void setUp() throws IOException {
conf = TEST_UTIL.getConfiguration();
+long var = conf.getInt(PREFETCH_DELAY, 1000);
Review Comment:
nit: useless line, please remove it.
##
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java:
##
@@ -106,10 +115,19 @@ public class TestPrefetch {
@Before
public void setUp() throws IOException {
conf = TEST_UTIL.getConfiguration();
+long var = conf.getInt(PREFETCH_DELAY, 1000);
conf.setBoolean(CacheConfig.PREFETCH_BLOCKS_ON_OPEN_KEY, true);
fs = HFileSystem.get(conf);
blockCache = BlockCacheFactory.createBlockCache(conf);
cacheConf = new CacheConfig(conf, blockCache);
+prefetchExecutorNotifier = new PrefetchExecutorNotifier(conf);
+resetTiming();
Review Comment:
Is this irrelevant for non delay related test? If so, remove it from here
and put at the end of the related tests only.
##
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java:
##
@@ -84,6 +87,7 @@
@Category({ IOTests.class, MediumTests.class })
public class TestPrefetch {
private static final Logger LOG =
LoggerFactory.getLogger(TestPrefetch.class);
+ protected PrefetchExecutorNotifier prefetchExecutorNotifier;
Review Comment:
Why is this global? It should be created within the context of each
individual test, so better make it local on each test method.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java:
##
@@ -78,7 +85,10 @@ public Thread newThread(Runnable r) {
+ Path.SEPARATOR_CHAR + ")|(" + Path.SEPARATOR_CHAR
+ HConstants.HREGION_COMPACTIONDIR_NAME.replace(".", "\\.") +
Path.SEPARATOR_CHAR + ")");
- public static void request(Path path, Runnable runnable) {
+ // For tests. Contains computed prefetch delay
+ private static long computedPrefetchDelay;
Review Comment:
Do we really need to calculate and check this on the tests? I guess the
configured delay is a guaranteed minimum. so we could just check 1) before that
minimum no prefetch is running, 2) we check that once prefetch has completed it
took at least the minimum time (specified in the delay).
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/PrefetchExecutorNotifier.java:
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.conf.ConfigurationManager;
+import org.apache.hadoop.hbase.conf.PropagatingConfigurationObserver;
+import org.apache.hadoop.hbase.io.hfile.PrefetchExecutor;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Class to submit requests for PrefetchExecutor depending on configuration
change
+ */
+@InterfaceAudience.Private
+public final cl
Re: [PR] HBASE-28321 RpcConnectionRegistry is broken when security is enabled … [hbase]
Apache-HBase commented on PR #5707: URL: https://github.com/apache/hbase/pull/5707#issuecomment-1966353273 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 42s | branch-2 passed | | +1 :green_heart: | compile | 4m 2s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 4s | branch-2 passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 4m 24s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 0s | the patch passed | | +1 :green_heart: | compile | 4m 53s | the patch passed | | +1 :green_heart: | cc | 4m 53s | the patch passed | | -0 :warning: | javac | 0m 48s | hbase-client generated 1 new + 107 unchanged - 1 fixed = 108 total (was 108) | | +1 :green_heart: | checkstyle | 1m 24s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 22s | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6. | | +1 :green_heart: | hbaseprotoc | 2m 0s | the patch passed | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 5m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. | | | | 46m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5707 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool | | uname | Linux 095f7a845719 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ac01d72d94 | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-client.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-examples U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5707/4/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28354 RegionSizeCalculator throws NPE when regions are in transition [hbase]
frostruan commented on PR #5699: URL: https://github.com/apache/hbase/pull/5699#issuecomment-1966344106 Also ping Duo~ @Apache9 This problem is related to my PR you reviewed yesterday, would you mind taking a look at this too ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28354 RegionSizeCalculator throws NPE when regions are in transition [hbase]
frostruan commented on PR #5699: URL: https://github.com/apache/hbase/pull/5699#issuecomment-1966341246 I think there are two problems here: 1. For in-transition regions, the result of RegionLocation.getServerName() could be null. In the current PR, we just filter out these regions. With in-transition region missing, we may lost some data. 2. We first get all region servers with regions of target table, then we request each region server to get RegionMetrics. If any region is moved from this server during this period, this region will be missed too. For the first problem, I think maybe we can return something indicates that we can not know the specific data size now, so in the previous discussion, I propose introducing a new constant UNKNOWN_SIZE with value of 1 byte. For the second problem, I think maybe we'd better use a snapshot of cluster metrics to make sure we will not miss any region. What do you think ? Thanks. @aalhour -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28384) Client ingegration tests fails for branch-2/branch-2.6
[
https://issues.apache.org/jira/browse/HBASE-28384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821185#comment-17821185
]
Hudson commented on HBASE-28384:
Results for branch HBASE-28384-branch-2
[build #2 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/2/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-28384-branch-2/2//console].
> Client ingegration tests fails for branch-2/branch-2.6
> --
>
> Key: HBASE-28384
> URL: https://issues.apache.org/jira/browse/HBASE-28384
> Project: HBase
> Issue Type: Bug
> Components: hadoop3, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> In HBASE-28331, we fixed the problem that we should use the latest hadoop3
> tarballs, for generating hdfs-site.xml.
> But for branch-2.x, we still use hadoop2 when compiling hbase by default, so
> there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop
> 3.3.x will generate hdfs-site.xml with this config, then leads to the failure
> of hbase start up.
> Should we change to build with hadoop3 while running on top of hadoop3? Since
> for branch-2.x, we will publish artifacts for hadoop3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28354 RegionSizeCalculator throws NPE when regions are in transition [hbase]
aalhour commented on PR #5699: URL: https://github.com/apache/hbase/pull/5699#issuecomment-1966144703 @frostruan, which part of the code would handle that byte? Also, should that be added to this PR or yours (https://github.com/apache/hbase/pull/5700)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-28354) RegionSizeCalculator throws NPE when regions are in transition
[ https://issues.apache.org/jira/browse/HBASE-28354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-28354 started by Ahmad Alhour. > RegionSizeCalculator throws NPE when regions are in transition > -- > > Key: HBASE-28354 > URL: https://issues.apache.org/jira/browse/HBASE-28354 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Ahmad Alhour >Priority: Major > Labels: pull-request-available > > When a region is in transition, it may briefly have a null ServerName in > meta. The RegionSizeCalculator calls RegionLocator.getAllRegionLocations() > and does not handle the possibility that a RegionLocation.getServerName() > could be null. The ServerName is eventually passed into an Admin call, which > results in an NPE. > This has come up in other contexts. For example, taking a look at > getAllRegionLocations() impl, we have checks to ensure that we don't call > null server names. We need to similarly handle the possibility of nulls in > RegionSizeCalculator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28391 Remove the need for ADMIN permissions for listDecommissionedRegionServers [hbase]
Apache-HBase commented on PR #5695: URL: https://github.com/apache/hbase/pull/5695#issuecomment-1966130805 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 58s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 5m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 0m 52s | the patch passed | | +1 :green_heart: | javac | 0m 52s | the patch passed | | +1 :green_heart: | shadedjars | 5m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 238m 42s | hbase-server in the patch passed. | | | | 264m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5695/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5695 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9fb37d3083b1 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c4a02f7fcd | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5695/2/testReport/ | | Max. process+thread count | 4519 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5695/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28391 Remove the need for ADMIN permissions for listDecommissionedRegionServers [hbase]
Apache-HBase commented on PR #5695: URL: https://github.com/apache/hbase/pull/5695#issuecomment-1966116522 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 42s | master passed | | +1 :green_heart: | compile | 0m 42s | master passed | | +1 :green_heart: | shadedjars | 5m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 25s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 234m 9s | hbase-server in the patch passed. | | | | 256m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5695/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5695 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 29f5df2aa883 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c4a02f7fcd | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5695/2/testReport/ | | Max. process+thread count | 6073 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5695/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28391 Remove the need for ADMIN permissions for listDecommissionedRegionServers [hbase]
NihalJain commented on code in PR #5695: URL: https://github.com/apache/hbase/pull/5695#discussion_r1503858370 ## hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java: ## @@ -1203,12 +1203,6 @@ public void preDecommissionRegionServers(ObserverContext However I am curious to know the difference between removing this method vs having a no-op method? IMO removing a method may send a false impression to a future auditor that there is no Access Rule defined for the method or it is somehow missing. And the person may end up re-adding the method. So it's better to have it, even if no-op. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
