[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842248#comment-17842248
]
Hudson commented on HBASE-28405:
Results for branch branch-2.6
[build #108 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/108/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/108/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/108/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/108/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/108/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2084403726 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 52s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 0m 18s | master passed | | +1 :green_heart: | shadedjars | 6m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 0m 19s | the patch passed | | +1 :green_heart: | javac | 0m 19s | the patch passed | | +1 :green_heart: | shadedjars | 6m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 4s | hbase-rest in the patch passed. | | | | 28m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 34ee03799704 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/testReport/ | | Max. process+thread count | 1616 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2084403718 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | master passed | | +1 :green_heart: | compile | 0m 24s | master passed | | +1 :green_heart: | shadedjars | 6m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | shadedjars | 6m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 5m 22s | hbase-rest in the patch passed. | | | | 28m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 39d8162d7378 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/testReport/ | | Max. process+thread count | 1753 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842246#comment-17842246
]
Hudson commented on HBASE-28405:
Results for branch branch-2.4
[build #728 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/728/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/728/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/728/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/728/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/728/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2084399772 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 2s | master passed | | +1 :green_heart: | compile | 0m 19s | master passed | | +1 :green_heart: | shadedjars | 5m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | compile | 0m 20s | the patch passed | | +1 :green_heart: | javac | 0m 20s | the patch passed | | +1 :green_heart: | shadedjars | 5m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 58s | hbase-rest in the patch passed. | | | | 24m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 06b580c39a05 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/testReport/ | | Max. process+thread count | 1698 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2084396598 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 0m 30s | master passed | | +1 :green_heart: | checkstyle | 0m 12s | master passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 31s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | -0 :warning: | javac | 0m 28s | hbase-rest generated 1 new + 184 unchanged - 1 fixed = 185 total (was 185) | | +1 :green_heart: | checkstyle | 0m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 53s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 21m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 0a8e51c2f4c6 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-rest.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28556) Reduce memory copying in Rest server when serializing CellModel to Protobuf
[ https://issues.apache.org/jira/browse/HBASE-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated HBASE-28556: Description: The REST server does a lot of unneccessary coping, which could be avoided at least for protobuf encoding. - It uses ByteStringer to handle ByteBuffer backed Cells. However, it uses the client API, so it should never encounter ByteBuffer backed cells. - It clones everything from the cells (sometimes multiple times) before serializing to protbuf. We could mimic the structure in Cell, with array, offset and length for each field, in CellModel and use the appropriate protobuf setters to avoid the extra copies. There may or may not be a way to do the same for JSON and XML via jax-rs, I don't know the frameworks well enough to tell, but if not, we could just do the copying in the getters for them, which would not make things worse. was: The REST server does a lot of unneccessary coping, which could be avoided at least for protobuf encoding. - It uses ByteStringer to handle ByteBuffer backed Cells. However, it uses the client API, so it sjpuld never encounter ByteBuffer backed cells. - It clones everything from the cells (sometimes multiple times) before serializing to protbuf. We could mimic the structure in Cell, with array, offset and length for each field, and use the appropriate protobuf setters to avoid the extra copies. There may or may not be a way to do the same for JSON and XML via jax-rs, I don't know the frameworks well enough to tell, but if not, we could just do the copying in the getters for them. > Reduce memory copying in Rest server when serializing CellModel to Protobuf > --- > > Key: HBASE-28556 > URL: https://issues.apache.org/jira/browse/HBASE-28556 > Project: HBase > Issue Type: Improvement > Components: REST >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > The REST server does a lot of unneccessary coping, which could be avoided at > least for protobuf encoding. > - It uses ByteStringer to handle ByteBuffer backed Cells. However, it uses > the client API, so it should never encounter ByteBuffer backed cells. > - It clones everything from the cells (sometimes multiple times) before > serializing to protbuf. > We could mimic the structure in Cell, with array, offset and length for each > field, in CellModel and use the appropriate protobuf setters to avoid the > extra copies. > There may or may not be a way to do the same for JSON and XML via jax-rs, I > don't know the frameworks well enough to tell, but if not, we could just do > the copying in the getters for them, which would not make things worse. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-28556) Reduce memory copying in Rest server when serializing CellModel to Protobuf
[ https://issues.apache.org/jira/browse/HBASE-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth reassigned HBASE-28556: --- Assignee: Istvan Toth > Reduce memory copying in Rest server when serializing CellModel to Protobuf > --- > > Key: HBASE-28556 > URL: https://issues.apache.org/jira/browse/HBASE-28556 > Project: HBase > Issue Type: Improvement > Components: REST >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > The REST server does a lot of unneccessary coping, which could be avoided at > least for protobuf encoding. > - It uses ByteStringer to handle ByteBuffer backed Cells. However, it uses > the client API, so it sjpuld never encounter ByteBuffer backed cells. > - It clones everything from the cells (sometimes multiple times) before > serializing to protbuf. > We could mimic the structure in Cell, with array, offset and length for each > field, and use the appropriate protobuf setters to avoid the extra copies. > There may or may not be a way to do the same for JSON and XML via jax-rs, I > don't know the frameworks well enough to tell, but if not, we could just do > the copying in the getters for them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
stoty commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2084320167 Looking at the tests, the failure is because of some kind transient maven artifact resolution problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28526) hbase-rest jar does not work with hbase-shaded-client with protobuf encoding
[
https://issues.apache.org/jira/browse/HBASE-28526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842231#comment-17842231
]
Istvan Toth commented on HBASE-28526:
-
The best solution would be splitting the hbase-rest module in two, a client and
server component, and including the client part hbase-shaded-client.
Switching to the shaded protobuf should also help.
> hbase-rest jar does not work with hbase-shaded-client with protobuf encoding
>
>
> Key: HBASE-28526
> URL: https://issues.apache.org/jira/browse/HBASE-28526
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Istvan Toth
>Priority: Major
>
> When trying to decode a protobuf encoded CellSet, I get
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeFrom(Lcom/google/protobuf/Message$Builder;[B)V
> at
> org.apache.hadoop.hbase.rest.model.CellSetModel.getObjectFromMessage(CellSetModel.java:129)
> at RestClientExample.getMulti(RestClientExample.java:191)
> at RestClientExample.start(RestClientExample.java:138)
> at RestClientExample.main(RestClientExample.java:124)
> {noformat}
> Seems to be caused by relocating protobuf 2.5 in hbase-shaded-client.
> It works fine with the unrelcoated client i.e. when using the
> {noformat}
> export CLASSPATH=`hbase --internal-classpath classpath`:
> {noformat}
> command to set up the classpath for the client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28533 - On split procedure rollback revert parent region state … [hbase]
Apache-HBase commented on PR #5863: URL: https://github.com/apache/hbase/pull/5863#issuecomment-2083877171 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | shadedjars | 5m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 6m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 266m 27s | hbase-server in the patch failed. | | | | 293m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5863 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ff17d776d05a 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/testReport/ | | Max. process+thread count | 5265 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28533 - On split procedure rollback revert parent region state … [hbase]
Apache-HBase commented on PR #5863: URL: https://github.com/apache/hbase/pull/5863#issuecomment-2083852085 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 7m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 244m 19s | hbase-server in the patch passed. | | | | 274m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5863 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b5f697369f8f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/testReport/ | | Max. process+thread count | 4650 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28420 update the procedure's field to store for ServerRemoteProcedure [hbase]
Apache-HBase commented on PR #5816: URL: https://github.com/apache/hbase/pull/5816#issuecomment-2083848185 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 58s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed | | +1 :green_heart: | shadedjars | 5m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 6m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 36s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 242m 19s | hbase-server in the patch passed. | | | | 270m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5816 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 0db61131c076 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/testReport/ | | Max. process+thread count | 4463 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28560) Region quotas: Split/merge procedure rollback can lead to inaccurate account of region counts
[ https://issues.apache.org/jira/browse/HBASE-28560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Roudnitsky updated HBASE-28560: -- Labels: Quota (was: quotas) > Region quotas: Split/merge procedure rollback can lead to inaccurate account > of region counts > - > > Key: HBASE-28560 > URL: https://issues.apache.org/jira/browse/HBASE-28560 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-beta-2 >Reporter: Daniel Roudnitsky >Assignee: Daniel Roudnitsky >Priority: Major > Labels: Quota > > When region quotas are enabled, HMaster keeps an in memory account of region > counts through NamespaceStateManager. Region counts in NamespaceStateManager > are incremented/decremented at the beginning stages of split/merge > procedures, in > SPLIT_TABLE_REGION_PRE_OPERATION/MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION > before any region is offlined. If the split/merge procedure gets rolled back > after the region count change in NamespaceStateManager is made, the > split/merge procedure rollback does not revert the region count change in > NamespaceStateManager to reflect that the expected split/merge never > succeeded. This leaves NamespaceStateManager with an inaccurate account of > the number of regions, believing that there are more/less regions than > actually exist. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28560) Region quotas: Split/merge procedure rollback can lead to inaccurate account of region counts
Daniel Roudnitsky created HBASE-28560: - Summary: Region quotas: Split/merge procedure rollback can lead to inaccurate account of region counts Key: HBASE-28560 URL: https://issues.apache.org/jira/browse/HBASE-28560 Project: HBase Issue Type: Bug Affects Versions: 3.0.0-beta-2 Reporter: Daniel Roudnitsky Assignee: Daniel Roudnitsky When region quotas are enabled, HMaster keeps an in memory account of region counts through NamespaceStateManager. Region counts in NamespaceStateManager are incremented/decremented at the beginning stages of split/merge procedures, in SPLIT_TABLE_REGION_PRE_OPERATION/MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION before any region is offlined. If the split/merge procedure gets rolled back after the region count change in NamespaceStateManager is made, the split/merge procedure rollback does not revert the region count change in NamespaceStateManager to reflect that the expected split/merge never succeeded. This leaves NamespaceStateManager with an inaccurate account of the number of regions, believing that there are more/less regions than actually exist. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28420 update the procedure's field to store for ServerRemoteProcedure [hbase]
Apache-HBase commented on PR #5816: URL: https://github.com/apache/hbase/pull/5816#issuecomment-2083842999 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 5m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 23s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. | | -1 :x: | unit | 239m 58s | hbase-server in the patch failed. | | | | 265m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5816 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 7cee1e692533 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/testReport/ | | Max. process+thread count | 5207 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28533 - On split procedure rollback revert parent region state … [hbase]
Apache-HBase commented on PR #5863: URL: https://github.com/apache/hbase/pull/5863#issuecomment-2083841319 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 5m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 31s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | shadedjars | 5m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 240m 31s | hbase-server in the patch passed. | | | | 264m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5863 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dee380dc1553 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/testReport/ | | Max. process+thread count | 5025 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28420 update the procedure's field to store for ServerRemoteProcedure [hbase]
Apache-HBase commented on PR #5816: URL: https://github.com/apache/hbase/pull/5816#issuecomment-2083840692 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 30s | master passed | | +1 :green_heart: | compile | 1m 43s | master passed | | +1 :green_heart: | shadedjars | 6m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 1m 47s | the patch passed | | +1 :green_heart: | javac | 1m 47s | the patch passed | | +1 :green_heart: | shadedjars | 6m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 54s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 230m 51s | hbase-server in the patch passed. | | | | 262m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5816 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 22445dfbb124 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/testReport/ | | Max. process+thread count | 5615 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
Apache-HBase commented on PR #5856: URL: https://github.com/apache/hbase/pull/5856#issuecomment-2083818441 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | HBASE-28463 passed | | +1 :green_heart: | compile | 1m 5s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 6m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 18m 16s | hbase-server in the patch failed. | | | | 43m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5856 | | JIRA Issue | HBASE-28535 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8633fd4eeda0 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / f890281bf9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/testReport/ | | Max. process+thread count | 1708 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
Apache-HBase commented on PR #5856: URL: https://github.com/apache/hbase/pull/5856#issuecomment-2083811378 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 55s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 49s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 7m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 13m 13s | hbase-server in the patch failed. | | | | 37m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5856 | | JIRA Issue | HBASE-28535 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 92750aba1045 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / f890281bf9 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/testReport/ | | Max. process+thread count | 1579 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
Apache-HBase commented on PR #5856: URL: https://github.com/apache/hbase/pull/5856#issuecomment-2083806893 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | HBASE-28463 passed | | +1 :green_heart: | compile | 3m 1s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 49s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 54s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 50s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | compile | 2m 51s | the patch passed | | +1 :green_heart: | javac | 2m 51s | the patch passed | | +1 :green_heart: | checkstyle | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 40s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 33m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5856 | | JIRA Issue | HBASE-28535 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 46a86edd4dfd 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / f890281bf9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/6/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-28559) Region quotas: Multi-region merge causes inaccurate accounting of region counts
Daniel Roudnitsky created HBASE-28559: - Summary: Region quotas: Multi-region merge causes inaccurate accounting of region counts Key: HBASE-28559 URL: https://issues.apache.org/jira/browse/HBASE-28559 Project: HBase Issue Type: Bug Components: Quotas Affects Versions: 3.0.0-beta-2 Reporter: Daniel Roudnitsky Assignee: Daniel Roudnitsky There is support for merging more than two regions in one merge procedure with multi-region merge, but if region quotas are enabled, [NamespaceAuditor assumes that every merge is a two region merge|https://github.com/apache/hbase/blob/branch-3/hbase-server/src/main/java/org/apache/hadoop/hbase/namespace/NamespaceAuditor.java#L128-L129]. This causes an inaccurate in memory accounting of region counts in NamespaceStateManager, leading MasterQuotaManager to believe there are more regions than actually exist if multi-region merge is used. To demonstrate the issue: 1. Start with a table with 3 regions in a namespace with a region quota limit of 3 2. Merge all 3 regions leaving 1 region, NamespaceAuditor assumed it was a 2 region merge and believes the number of regions to be 2. 3. Split a region, number of regions is now 2, and NamespaceAuditor believe it to be 3. 4. Attempt another region split, which will fail because NamespaceAuditor believes the ns to be at its region limit of 3 when there are actually only 2 regions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28459 HFileOutputFormat2 ClassCastException with s3 magic committer [hbase]
Apache-HBase commented on PR #5858: URL: https://github.com/apache/hbase/pull/5858#issuecomment-2083777654 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 1s | branch-2 passed | | +1 :green_heart: | compile | 0m 39s | branch-2 passed | | +1 :green_heart: | shadedjars | 9m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 24s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | shadedjars | 8m 53s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 21m 30s | hbase-mapreduce in the patch passed. | | | | 53m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5858 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c1b5a9fb59c6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a456bfb142 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/testReport/ | | Max. process+thread count | 2853 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
Apache-HBase commented on PR #5856: URL: https://github.com/apache/hbase/pull/5856#issuecomment-2083762814 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 48s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 48s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 5s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 288m 11s | hbase-server in the patch failed. | | | | 313m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5856 | | JIRA Issue | HBASE-28535 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 869284c7189b 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / f890281bf9 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/testReport/ | | Max. process+thread count | 4949 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28459 HFileOutputFormat2 ClassCastException with s3 magic committer [hbase]
Apache-HBase commented on PR #5858: URL: https://github.com/apache/hbase/pull/5858#issuecomment-2083752684 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 19s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 5s | branch-2 passed | | +1 :green_heart: | compile | 0m 21s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 54s | the patch passed | | +1 :green_heart: | compile | 0m 20s | the patch passed | | +1 :green_heart: | javac | 0m 20s | the patch passed | | +1 :green_heart: | shadedjars | 5m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 16m 22s | hbase-mapreduce in the patch passed. | | | | 36m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5858 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c7887d248f82 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a456bfb142 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/testReport/ | | Max. process+thread count | 2982 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28459 HFileOutputFormat2 ClassCastException with s3 magic committer [hbase]
Apache-HBase commented on PR #5858: URL: https://github.com/apache/hbase/pull/5858#issuecomment-2083737936 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | branch-2 passed | | +1 :green_heart: | compile | 0m 39s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 15s | branch-2 passed | | +1 :green_heart: | spotless | 0m 52s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 38s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 0m 34s | the patch passed | | +1 :green_heart: | javac | 0m 34s | the patch passed | | +1 :green_heart: | checkstyle | 0m 12s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 32s | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6. | | +1 :green_heart: | spotless | 0m 51s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 25m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5858 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 8a8ceeff9501 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a456bfb142 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5858/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28554) TestZooKeeperScanPolicyObserver and TestAdminShell fail 100% of times on flaky dashboard
[
https://issues.apache.org/jira/browse/HBASE-28554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842166#comment-17842166
]
Hudson commented on HBASE-28554:
Results for branch master
[build #1064 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestZooKeeperScanPolicyObserver and TestAdminShell fail 100% of times on
> flaky dashboard
>
>
> Key: HBASE-28554
> URL: https://issues.apache.org/jira/browse/HBASE-28554
> Project: HBase
> Issue Type: Bug
> Components: shell, test, Zookeeper
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2, 2.5.9
>
>
> This is for branch-2.5+, need to figure out why before cutting any releases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28482) Reverse scan with tags throws ArrayIndexOutOfBoundsException with DBE
[
https://issues.apache.org/jira/browse/HBASE-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842165#comment-17842165
]
Hudson commented on HBASE-28482:
Results for branch master
[build #1064 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1064/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reverse scan with tags throws ArrayIndexOutOfBoundsException with DBE
> -
>
> Key: HBASE-28482
> URL: https://issues.apache.org/jira/browse/HBASE-28482
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Reporter: Vineet Kumar Maheshwari
>Assignee: Vineet Kumar Maheshwari
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> Facing ArrayIndexOutOfBoundsException when performing reverse scan on a table
> with 30K+ records in single hfile.
> Exception is happening when block changes during seekBefore call.
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray(ByteBufferUtils.java:1326)
> at org.apache.hadoop.hbase.nio.SingleByteBuff.get(SingleByteBuff.java:213)
> at
> org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder$DiffSeekerStateBufferedEncodedSeeker.decode(DiffKeyDeltaEncoder.java:431)
> at
> org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder$DiffSeekerStateBufferedEncodedSeeker.decodeNext(DiffKeyDeltaEncoder.java:502)
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.seekToKeyInBlock(BufferedDataBlockEncoder.java:1012)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$EncodedScanner.loadBlockAndSeekToKey(HFileReaderImpl.java:1605)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekBefore(HFileReaderImpl.java:719)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekBeforeAndSaveKeyToPreviousRow(StoreFileScanner.java:645)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRowWithoutHint(StoreFileScanner.java:570)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRow(StoreFileScanner.java:506)
> at
> org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:126)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:693)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:151){code}
>
> Steps to reproduce:
> Create a table with DataBlockEncoding.DIFF and block size as 1024, write some
> 30K+ puts with setTTL, then do a reverse scan.
> {code:java}
> @Test
> public void testReverseScanWithDBEWhenCurrentBlockUpdates() throws
> IOException {
> byte[] family = Bytes.toBytes("0");
> Configuration conf = new Configuration(TEST_UTIL.getConfiguration());
> conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);
> try (Connection connection = ConnectionFactory.createConnection(conf)) {
> testReverseScanWithDBE(connection, DataBlockEncoding.DIFF, family, 1024,
> 3);
> for (DataBlockEncoding encoding : DataBlockEncoding.values()) {
> testReverseScanWithDBE(connection, encoding, family, 1024, 3);
> }
> }
> }
> private void testReverseScanWithDBE(Connection conn, DataBlockEncoding
> encoding, byte[] family, int blockSize, int maxRows)
> throws IOException {
> LOG.info("Running test with DBE={}", encoding);
> TableName tableName = TableName.valueOf(TEST_NAME.getMethodName() + "-" +
> encoding);
> TEST_UTIL.createTable(TableDescriptorBuilder.newBuilder(tableName)
> .setColumnFamily(
> ColumnFamilyDescriptorBuilder.newBuilder(family).setDataBlockEncoding(encoding).setBlocksize(blockSize).build())
> .build(), null);
> Table table = conn.getTable(tableName);
> byte[] val1 = new byte[10];
> byte[] val2 = new byte[10];
> Bytes.random(val1);
> Bytes.random(val2);
> for (int i = 0; i < maxRows; i++) {
> table.put(new
[jira] [Commented] (HBASE-28554) TestZooKeeperScanPolicyObserver and TestAdminShell fail 100% of times on flaky dashboard
[
https://issues.apache.org/jira/browse/HBASE-28554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842163#comment-17842163
]
Hudson commented on HBASE-28554:
Results for branch branch-3
[build #196 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestZooKeeperScanPolicyObserver and TestAdminShell fail 100% of times on
> flaky dashboard
>
>
> Key: HBASE-28554
> URL: https://issues.apache.org/jira/browse/HBASE-28554
> Project: HBase
> Issue Type: Bug
> Components: shell, test, Zookeeper
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2, 2.5.9
>
>
> This is for branch-2.5+, need to figure out why before cutting any releases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28482) Reverse scan with tags throws ArrayIndexOutOfBoundsException with DBE
[
https://issues.apache.org/jira/browse/HBASE-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842162#comment-17842162
]
Hudson commented on HBASE-28482:
Results for branch branch-3
[build #196 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/196/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reverse scan with tags throws ArrayIndexOutOfBoundsException with DBE
> -
>
> Key: HBASE-28482
> URL: https://issues.apache.org/jira/browse/HBASE-28482
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Reporter: Vineet Kumar Maheshwari
>Assignee: Vineet Kumar Maheshwari
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> Facing ArrayIndexOutOfBoundsException when performing reverse scan on a table
> with 30K+ records in single hfile.
> Exception is happening when block changes during seekBefore call.
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray(ByteBufferUtils.java:1326)
> at org.apache.hadoop.hbase.nio.SingleByteBuff.get(SingleByteBuff.java:213)
> at
> org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder$DiffSeekerStateBufferedEncodedSeeker.decode(DiffKeyDeltaEncoder.java:431)
> at
> org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder$DiffSeekerStateBufferedEncodedSeeker.decodeNext(DiffKeyDeltaEncoder.java:502)
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.seekToKeyInBlock(BufferedDataBlockEncoder.java:1012)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$EncodedScanner.loadBlockAndSeekToKey(HFileReaderImpl.java:1605)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekBefore(HFileReaderImpl.java:719)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekBeforeAndSaveKeyToPreviousRow(StoreFileScanner.java:645)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRowWithoutHint(StoreFileScanner.java:570)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRow(StoreFileScanner.java:506)
> at
> org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:126)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:693)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:151){code}
>
> Steps to reproduce:
> Create a table with DataBlockEncoding.DIFF and block size as 1024, write some
> 30K+ puts with setTTL, then do a reverse scan.
> {code:java}
> @Test
> public void testReverseScanWithDBEWhenCurrentBlockUpdates() throws
> IOException {
> byte[] family = Bytes.toBytes("0");
> Configuration conf = new Configuration(TEST_UTIL.getConfiguration());
> conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);
> try (Connection connection = ConnectionFactory.createConnection(conf)) {
> testReverseScanWithDBE(connection, DataBlockEncoding.DIFF, family, 1024,
> 3);
> for (DataBlockEncoding encoding : DataBlockEncoding.values()) {
> testReverseScanWithDBE(connection, encoding, family, 1024, 3);
> }
> }
> }
> private void testReverseScanWithDBE(Connection conn, DataBlockEncoding
> encoding, byte[] family, int blockSize, int maxRows)
> throws IOException {
> LOG.info("Running test with DBE={}", encoding);
> TableName tableName = TableName.valueOf(TEST_NAME.getMethodName() + "-" +
> encoding);
> TEST_UTIL.createTable(TableDescriptorBuilder.newBuilder(tableName)
> .setColumnFamily(
> ColumnFamilyDescriptorBuilder.newBuilder(family).setDataBlockEncoding(encoding).setBlocksize(blockSize).build())
> .build(), null);
> Table table = conn.getTable(tableName);
> byte[] val1 = new byte[10];
> byte[] val2 = new byte[10];
> Bytes.random(val1);
> Bytes.random(val2);
> for (int i = 0; i < maxRows; i++) {
> table.put(new
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Status: Patch Available (was: In Progress)
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
> Labels: pull-request-available
>
> Depending on where the split procedure fails, SplitTableRegionProcedure
> rollback can leave the parent region's in memory RegionStateNode in SPLITTING
> after rollback is complete, when the parent region is still online on the
> assigned region server and according to meta. This leaves active HMaster
> believing that the parent region is offline according to its RegionStates,
> and causes subsequent procedures that require the region to be online like
> merge/split/move to fail to start. One workaround is to restart active
> HMaster to reset the in memory record of region states.
> Two example scenarios where this can happen:
> * If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
> has a replica which is in transition, the unassign procedure in that step is
> never created/rolled back and we are left with the parent region state in
> splitting.
> * If region quotas are enabled and a split is run for a region whose
> namespace is at its maximum region quota limit we will fail in
> SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
> with the parent region state in splitting
> The region replica case is demonstrated in
> TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
> To reproduce the region quota case in HBase shell:
> {code:java}
> > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> > 'UniformSplit'}
> > region_a =
> > region_b =
> > split region_a, 'x'
> # HMaster will report:
> pid=405, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
> master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
> Region split not possible for : as quota limits are exceeded ;
> SplitTableRegionProcedure table=test_ns:test_table, parent=...
> > merge_region region_a, region_b
> ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
> OPEN; state=SPLITTING
> > stop_master # trigger hmaster failover
> > merge_region region_a, region_b # merge now succeeds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's in memory RegionStateNode in SPLITTING
after rollback is complete, when the parent region is still online on the
assigned region server and according to meta. This leaves active HMaster
believing that the parent region is offline according to its RegionStates, and
causes subsequent procedures that require the region to be online like
merge/split/move to fail to start. One workaround is to restart active HMaster
to reset the in memory record of region states.
Two example scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server and according to meta. This leaves active HMaster believing that
the parent region is offline according to its RegionStates, and causes
subsequent procedures that require the region to be online like
merge/split/move to fail to start. One workaround is to restart active HMaster
to reset the in memory record of region states.
Two example scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842144#comment-17842144
]
Daniel Roudnitsky commented on HBASE-28405:
---
HBASE-28533 provides a partially similar patch for split procedure rollback to
correct the in memory RegionState in HMaster to reflect that the region is
still online on the RS after the failed split. HBASE-19126 seems relevant to
both issues.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to open the region a92008b76ccae47d55c590930b837036.
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of procedure. When we run TRSP
> it sends a OpenRegionProcedure which is handled by AssignRegionHandler. This
> handlers on execution suggests that region is already online*
> Sequence of events are as follow
> _2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore -
> pid=26674602 updating hbase:meta row=a92008b76ccae47d55c590930b837036,
> regionState=OPENING, regionLocation=rs-210,60020,1707596461539_
> _2024-02-11 10:53:58,920 INFO [PEWorker-58] procedure2.ProcedureExecutor -
> Initialized subprocedures=[\\{pid=26675798, ppid=26674602, state=RUNNABLE;
> OpenRegionProcedure a92008b76ccae47d55c590930b837036,
> server=rs-210,60020,1707596461539}]_
> _2024-02-11 10:53:59,074 WARN
[jira] [Commented] (HBASE-28554) TestZooKeeperScanPolicyObserver and TestAdminShell fail 100% of times on flaky dashboard
[
https://issues.apache.org/jira/browse/HBASE-28554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842146#comment-17842146
]
Hudson commented on HBASE-28554:
Results for branch branch-2.5
[build #520 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestZooKeeperScanPolicyObserver and TestAdminShell fail 100% of times on
> flaky dashboard
>
>
> Key: HBASE-28554
> URL: https://issues.apache.org/jira/browse/HBASE-28554
> Project: HBase
> Issue Type: Bug
> Components: shell, test, Zookeeper
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2, 2.5.9
>
>
> This is for branch-2.5+, need to figure out why before cutting any releases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28482) Reverse scan with tags throws ArrayIndexOutOfBoundsException with DBE
[
https://issues.apache.org/jira/browse/HBASE-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842145#comment-17842145
]
Hudson commented on HBASE-28482:
Results for branch branch-2.5
[build #520 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reverse scan with tags throws ArrayIndexOutOfBoundsException with DBE
> -
>
> Key: HBASE-28482
> URL: https://issues.apache.org/jira/browse/HBASE-28482
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Reporter: Vineet Kumar Maheshwari
>Assignee: Vineet Kumar Maheshwari
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> Facing ArrayIndexOutOfBoundsException when performing reverse scan on a table
> with 30K+ records in single hfile.
> Exception is happening when block changes during seekBefore call.
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray(ByteBufferUtils.java:1326)
> at org.apache.hadoop.hbase.nio.SingleByteBuff.get(SingleByteBuff.java:213)
> at
> org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder$DiffSeekerStateBufferedEncodedSeeker.decode(DiffKeyDeltaEncoder.java:431)
> at
> org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder$DiffSeekerStateBufferedEncodedSeeker.decodeNext(DiffKeyDeltaEncoder.java:502)
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.seekToKeyInBlock(BufferedDataBlockEncoder.java:1012)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$EncodedScanner.loadBlockAndSeekToKey(HFileReaderImpl.java:1605)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekBefore(HFileReaderImpl.java:719)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekBeforeAndSaveKeyToPreviousRow(StoreFileScanner.java:645)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRowWithoutHint(StoreFileScanner.java:570)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRow(StoreFileScanner.java:506)
> at
> org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:126)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:693)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:151){code}
>
> Steps to reproduce:
> Create a table with DataBlockEncoding.DIFF and block size as 1024, write some
> 30K+ puts with setTTL, then do a reverse scan.
> {code:java}
> @Test
> public void testReverseScanWithDBEWhenCurrentBlockUpdates() throws
> IOException {
> byte[] family = Bytes.toBytes("0");
> Configuration conf = new Configuration(TEST_UTIL.getConfiguration());
> conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);
> try (Connection connection = ConnectionFactory.createConnection(conf)) {
> testReverseScanWithDBE(connection, DataBlockEncoding.DIFF, family, 1024,
> 3);
> for (DataBlockEncoding encoding : DataBlockEncoding.values()) {
> testReverseScanWithDBE(connection, encoding, family, 1024, 3);
> }
> }
> }
> private void testReverseScanWithDBE(Connection conn, DataBlockEncoding
> encoding, byte[] family, int blockSize, int maxRows)
> throws IOException {
> LOG.info("Running test with DBE={}", encoding);
> TableName tableName = TableName.valueOf(TEST_NAME.getMethodName() + "-" +
> encoding);
> TEST_UTIL.createTable(TableDescriptorBuilder.newBuilder(tableName)
> .setColumnFamily(
> ColumnFamilyDescriptorBuilder.newBuilder(family).setDataBlockEncoding(encoding).setBlocksize(blockSize).build())
> .build(),
[jira] [Commented] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842147#comment-17842147
]
Hudson commented on HBASE-28405:
Results for branch branch-2.5
[build #520 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/520/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to open
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server and according to meta. This leaves active HMaster believing that
the parent region is offline according to its RegionStates, and causes
subsequent procedures that require the region to be online like
merge/split/move to fail to start. One workaround is to restart active HMaster
to reset the in memory record of region states.
Two example scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two example scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority:
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two example scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
> Labels:
Re: [PR] HBASE-28420 update the procedure's field to store for ServerRemoteProcedure [hbase]
Apache-HBase commented on PR #5816: URL: https://github.com/apache/hbase/pull/5816#issuecomment-2083502346 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 48s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 53s | master passed | | +1 :green_heart: | compile | 3m 19s | master passed | | +1 :green_heart: | checkstyle | 0m 45s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 3m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 3m 17s | the patch passed | | +1 :green_heart: | cc | 3m 17s | the patch passed | | +1 :green_heart: | javac | 3m 17s | the patch passed | | -0 :warning: | checkstyle | 0m 37s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 57s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | hbaseprotoc | 1m 11s | the patch passed | | +1 :green_heart: | spotless | 0m 39s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 3m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 36m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5816 | | Optional Tests | dupname asflicense cc hbaseprotoc spotless prototool javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6a946b0bd51d 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 82 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5816/4/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28533 - On split procedure rollback revert parent region state … [hbase]
Apache-HBase commented on PR #5863: URL: https://github.com/apache/hbase/pull/5863#issuecomment-2083489009 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 7s | master passed | | +1 :green_heart: | compile | 2m 43s | master passed | | +1 :green_heart: | checkstyle | 0m 37s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 42s | the patch passed | | +1 :green_heart: | compile | 2m 40s | the patch passed | | +1 :green_heart: | javac | 2m 40s | the patch passed | | +1 :green_heart: | checkstyle | 0m 33s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 8s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 29m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5863 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux a62334d65fdb 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5863/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28428 : ConnectionRegistry APIs should have timeout [hbase]
virajjasani commented on PR #5837: URL: https://github.com/apache/hbase/pull/5837#issuecomment-2083463566 Here, `TimerTask` is used within `addListener()`, hence we can make somewhat similar change in `ZKConnectionRegistry`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28428 : ConnectionRegistry APIs should have timeout [hbase]
virajjasani commented on PR #5837:
URL: https://github.com/apache/hbase/pull/5837#issuecomment-2083461473
The other example is for `execProcedure`:
```
@Override
public CompletableFuture execProcedure(String signature, String
instance,
Map props) {
CompletableFuture future = new CompletableFuture<>();
ProcedureDescription procDesc =
ProtobufUtil.buildProcedureDescription(signature, instance, props);
addListener(this. newMasterCaller()
.action((controller, stub) -> this. call(
controller, stub,
ExecProcedureRequest.newBuilder().setProcedure(procDesc).build(),
(s, c, req, done) -> s.execProcedure(c, req, done), resp ->
resp.getExpectedTimeout()))
.call(), (expectedTimeout, err) -> {
if (err != null) {
future.completeExceptionally(err);
return;
}
TimerTask pollingTask = new TimerTask() {
int tries = 0;
long startTime = EnvironmentEdgeManager.currentTime();
long endTime = startTime + expectedTimeout;
long maxPauseTime = expectedTimeout / maxAttempts;
@Override
public void run(Timeout timeout) throws Exception {
if (EnvironmentEdgeManager.currentTime() < endTime) {
addListener(isProcedureFinished(signature, instance, props),
(done, err2) -> {
if (err2 != null) {
future.completeExceptionally(err2);
return;
}
if (done) {
future.complete(null);
} else {
// retry again after pauseTime.
long pauseTime =
ConnectionUtils.getPauseTime(TimeUnit.NANOSECONDS.toMillis(pauseNs), ++tries);
pauseTime = Math.min(pauseTime, maxPauseTime);
AsyncConnectionImpl.RETRY_TIMER.newTimeout(this, pauseTime,
TimeUnit.MICROSECONDS);
}
});
} else {
future.completeExceptionally(new IOException("Procedure '" +
signature + " : "
+ instance + "' wasn't completed in expectedTime:" +
expectedTimeout + " ms"));
}
}
};
// Queue the polling task into RETRY_TIMER to poll procedure state
asynchronously.
AsyncConnectionImpl.RETRY_TIMER.newTimeout(pollingTask, 1,
TimeUnit.MILLISECONDS);
});
return future;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28428 : ConnectionRegistry APIs should have timeout [hbase]
virajjasani commented on PR #5837:
URL: https://github.com/apache/hbase/pull/5837#issuecomment-2083455858
@Divneet18 here is the sample for TimerTask and how we can introduce timeout:
```
TimerTask pollingTask = new TimerTask() {
int tries = 0;
long startTime = EnvironmentEdgeManager.currentTime();
long endTime = startTime + expectedTimeout;
long maxPauseTime = expectedTimeout / maxAttempts;
@Override
public void run(Timeout timeout) throws Exception {
if (EnvironmentEdgeManager.currentTime() < endTime) {
addListener(isSnapshotFinished(snapshot), (done, err2) -> {
if (err2 != null) {
future.completeExceptionally(err2);
} else if (done) {
future.complete(null);
} else {
// retry again after pauseTime.
long pauseTime =
ConnectionUtils.getPauseTime(TimeUnit.NANOSECONDS.toMillis(pauseNs), ++tries);
pauseTime = Math.min(pauseTime, maxPauseTime);
AsyncConnectionImpl.RETRY_TIMER.newTimeout(this, pauseTime,
TimeUnit.MILLISECONDS);
}
});
} else {
future
.completeExceptionally(new SnapshotCreationException("Snapshot
'" + snapshot.getName()
+ "' wasn't completed in expectedTime:" + expectedTimeout +
" ms", snapshot));
}
}
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28420 update the procedure's field to store for ServerRemoteProcedure [hbase]
Umeshkumar9414 commented on code in PR #5816:
URL: https://github.com/apache/hbase/pull/5816#discussion_r1583578291
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerRemoteProcedure.java:
##
@@ -137,6 +140,10 @@ synchronized void remoteOperationDone(MasterProcedureEnv
env, Throwable error) {
getProcId());
return;
}
+//below persistence is added so that if report goes to last active master,
it throws exception
+state =
MasterProcedureProtos.ServerRemoteProcedureState.SERVER_REMOTE_PROCEDURE_REPORT_SUCCEED;
+
env.getMasterServices().getMasterProcedureExecutor().getStore().update(this);
+
complete(env, error);
Review Comment:
I wanted to call complete in remoteOperationDone so that we didn't encounter
the scenario of a master with old code becoming active. Anyway as we don't need
to consider this, I have updated the PR with new changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28420 persist the procedure's field to store on remoteProcedureDone for ServerRemoteProcedure [hbase]
Umeshkumar9414 commented on PR #5857: URL: https://github.com/apache/hbase/pull/5857#issuecomment-2083410773 added these changes in [PR](https://github.com/apache/hbase/pull/5816) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28420 persist the procedure's field to store on remoteProcedureDone for ServerRemoteProcedure [hbase]
Umeshkumar9414 closed pull request #5857: HBASE-28420 persist the procedure's field to store on remoteProcedureDone for ServerRemoteProcedure URL: https://github.com/apache/hbase/pull/5857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-28533:
---
Labels: pull-request-available (was: )
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
> Labels: pull-request-available
>
> Depending on where the split procedure fails, SplitTableRegionProcedure
> rollback can leave the parent region's RegionStateNode in SPLITTING after
> rollback is complete, when the parent region is still online on the assigned
> region server. This leaves active HMaster believing that the parent region is
> offline according to its RegionStates, and causes subsequent procedures that
> require the region to be online like merge/split/move to fail to start. One
> workaround is to restart active HMaster to reset the in memory record of
> region states.
> Two scenarios where this can happen:
> * If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
> has a replica which is in transition, the unassign procedure in that step is
> never created/rolled back and we are left with the parent region state in
> splitting.
> * If region quotas are enabled and a split is run for a region whose
> namespace is at its maximum region quota limit we will fail in
> SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
> with the parent region state in splitting
> The region replica case is demonstrated in
> TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
> To reproduce the region quota case in HBase shell:
> {code:java}
> > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> > 'UniformSplit'}
> > region_a =
> > region_b =
> > split region_a, 'x'
> # HMaster will report:
> pid=405, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
> master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
> Region split not possible for : as quota limits are exceeded ;
> SplitTableRegionProcedure table=test_ns:test_table, parent=...
> > merge_region region_a, region_b
> ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
> OPEN; state=SPLITTING
> > stop_master # trigger hmaster failover
> > merge_region region_a, region_b # merge now succeeds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583471499
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -402,6 +406,67 @@ public void testBlockEvictionsHotBlocks() throws Exception
{
validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
}
+ @Test
+ public void testFeatureKeyDisabled() throws Exception {
+DataTieringManager.resetForTestingOnly();
+defaultConf.setBoolean(DataTieringManager.GLOBAL_DATA_TIERING_ENABLED_KEY,
false);
+try {
+ assertFalse(DataTieringManager.instantiate(defaultConf,
testOnlineRegions));
+ // Verify that the DataaTieringManager instance is not instantiated in
the
+ // instantiate call above.
+ assertNull(DataTieringManager.getInstance());
+
+ // Also validate that data temperature is not honoured.
+ long capacitySize = 40 * 1024;
+ int writeThreads = 3;
+ int writerQLen = 64;
+ int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+ // Setup: Create a bucket cache with lower capacity
+ BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+ // Create three Cache keys with two hot data blocks and one cold data
block
+ // hStoreFiles.get(0) is a hot data file and hStoreFiles.get(3) is a
cold data file.
+ Set cacheKeys = new HashSet<>();
+ cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+ cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 8192,
true, BlockType.DATA));
+ cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+
+ // Create dummy data to be cached and fill the cache completely.
+ CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+ int blocksIter = 0;
+ for (BlockCacheKey key : cacheKeys) {
+bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+// Ensure that the block is persisted to the file.
+Waiter.waitFor(defaultConf, 1, 100,
+ () -> (bucketCache.getBackingMap().containsKey(key)));
+ }
+
+ // Verify that the bucket cache contains 3 blocks.
+ assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+ // Add an additional block which should evict the only cold block with
an additional hot
+ // block.
+ BlockCacheKey newKey =
+new BlockCacheKey(hStoreFiles.get(2).getPath(), 0, true,
BlockType.DATA);
+ CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+ bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+ Waiter.waitFor(defaultConf, 1, 100,
+() -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+ // Verify that the bucket still contains the 2 cold blocks and one newly
added hot block.
Review Comment:
Currently, in this test, which blocks stay is not deterministic. Any block
may get evicted leading to test failure.
@wchevreuil, @vinayakphegde, any idea to enforce the eviction of hot blocks
and retain the cold blocks in cache to validate that data-tiering is not
effective? If not, we may have to rely on the absence of DataTieringManager
instance only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
Apache-HBase commented on PR #5856: URL: https://github.com/apache/hbase/pull/5856#issuecomment-2083281790 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | HBASE-28463 passed | | +1 :green_heart: | compile | 4m 5s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 50s | HBASE-28463 passed | | +1 :green_heart: | spotless | 1m 0s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 15s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 3m 38s | the patch passed | | +1 :green_heart: | javac | 3m 38s | the patch passed | | +1 :green_heart: | checkstyle | 1m 1s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 7m 39s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 53s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 41m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5856 | | JIRA Issue | HBASE-28535 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux d5be5dc00984 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / f890281bf9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
Apache-HBase commented on PR #5856: URL: https://github.com/apache/hbase/pull/5856#issuecomment-2083267689 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 54s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 50s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 5m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 14m 21s | hbase-server in the patch failed. | | | | 34m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5856 | | JIRA Issue | HBASE-28535 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8aeb04bff2d6 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / f890281bf9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/testReport/ | | Max. process+thread count | 1693 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5856/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583446844
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -61,25 +63,25 @@ private DataTieringManager(Map
onlineRegions) {
* Initializes the DataTieringManager instance with the provided map of
online regions.
* @param onlineRegions A map containing online regions.
*/
- public static synchronized void instantiate(Map
onlineRegions) {
-if (instance == null) {
- instance = new DataTieringManager(onlineRegions);
- LOG.info("DataTieringManager instantiated successfully.");
+ public static synchronized void instantiate(Configuration conf,
Review Comment:
Also, would it make sense to return True if instantiated successfully and
false otherwise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583445711
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -61,25 +63,25 @@ private DataTieringManager(Map
onlineRegions) {
* Initializes the DataTieringManager instance with the provided map of
online regions.
* @param onlineRegions A map containing online regions.
*/
- public static synchronized void instantiate(Map
onlineRegions) {
-if (instance == null) {
- instance = new DataTieringManager(onlineRegions);
- LOG.info("DataTieringManager instantiated successfully.");
+ public static synchronized void instantiate(Configuration conf,
Review Comment:
I have made the function isDataTieringFeatureEnabled private function of the
class. We do not want to give the undue flexiblity to the user to instantiate
the DataTieringManager even when the feature key is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28526) hbase-rest jar does not work with hbase-shaded-client with protobuf encoding
[
https://issues.apache.org/jira/browse/HBASE-28526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Istvan Toth updated HBASE-28526:
Description:
When trying to decode a protobuf encoded CellSet, I get
{noformat}
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeFrom(Lcom/google/protobuf/Message$Builder;[B)V
at
org.apache.hadoop.hbase.rest.model.CellSetModel.getObjectFromMessage(CellSetModel.java:129)
at RestClientExample.getMulti(RestClientExample.java:191)
at RestClientExample.start(RestClientExample.java:138)
at RestClientExample.main(RestClientExample.java:124)
{noformat}
Seems to be caused by relocating protobuf 2.5 in hbase-shaded-client.
It works fine with the unrelcoated client i.e. when using the
{noformat}
export CLASSPATH=`hbase --internal-classpath classpath`:
{noformat}
command to set up the classpath for the client.
was:
When trying to decode a protobof encoded CellSet, I get
{noformat}
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeFrom(Lcom/google/protobuf/Message$Builder;[B)V
at
org.apache.hadoop.hbase.rest.model.CellSetModel.getObjectFromMessage(CellSetModel.java:129)
at RestClientExample.getMulti(RestClientExample.java:191)
at RestClientExample.start(RestClientExample.java:138)
at RestClientExample.main(RestClientExample.java:124)
{noformat}
Seems to be caused by relocating protobuf 2.5 in hbase-shaded-client.
It works fine with the unrelcoated client i.e. when using the
{noformat}
export CLASSPATH=`hbase --internal-classpath classpath`:
{noformat}
command to set up the classpath for the client.
> hbase-rest jar does not work with hbase-shaded-client with protobuf encoding
>
>
> Key: HBASE-28526
> URL: https://issues.apache.org/jira/browse/HBASE-28526
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Istvan Toth
>Priority: Major
>
> When trying to decode a protobuf encoded CellSet, I get
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeFrom(Lcom/google/protobuf/Message$Builder;[B)V
> at
> org.apache.hadoop.hbase.rest.model.CellSetModel.getObjectFromMessage(CellSetModel.java:129)
> at RestClientExample.getMulti(RestClientExample.java:191)
> at RestClientExample.start(RestClientExample.java:138)
> at RestClientExample.main(RestClientExample.java:124)
> {noformat}
> Seems to be caused by relocating protobuf 2.5 in hbase-shaded-client.
> It works fine with the unrelcoated client i.e. when using the
> {noformat}
> export CLASSPATH=`hbase --internal-classpath classpath`:
> {noformat}
> command to set up the classpath for the client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583364899
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -45,6 +45,8 @@
@InterfaceAudience.Private
public class DataTieringManager {
private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATA_TIERING_ENABLED_KEY =
"hbase.hstore.datatiering.enable";
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2083082457 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 0m 20s | master passed | | +1 :green_heart: | shadedjars | 6m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 0m 20s | the patch passed | | +1 :green_heart: | javac | 0m 20s | the patch passed | | +1 :green_heart: | shadedjars | 6m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 4m 31s | hbase-rest in the patch passed. | | | | 26m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3605fab2ebfa 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/testReport/ | | Max. process+thread count | 1639 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2083076969 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 1m 5s | root in master failed. | | -1 :x: | compile | 0m 12s | hbase-rest in master failed. | | +1 :green_heart: | shadedjars | 6m 29s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 8s | hbase-rest in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | shadedjars | 6m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 4m 51s | hbase-rest in the patch passed. | | | | 24m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8461bcf4479c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Temurin-1.8.0_352-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-jdk8-hadoop3-check/output/branch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-jdk8-hadoop3-check/output/branch-compile-hbase-rest.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-jdk8-hadoop3-check/output/branch-javadoc-hbase-rest.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/testReport/ | | Max. process+thread count | 1633 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2083074933 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 0m 20s | master passed | | +1 :green_heart: | shadedjars | 5m 22s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 0m 19s | the patch passed | | +1 :green_heart: | javac | 0m 19s | the patch passed | | +1 :green_heart: | shadedjars | 5m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 56s | hbase-rest in the patch passed. | | | | 23m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 01088176bd5f 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/testReport/ | | Max. process+thread count | 1637 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28523 Use a single get call in REST multiget endpoint [hbase]
Apache-HBase commented on PR #5862: URL: https://github.com/apache/hbase/pull/5862#issuecomment-2083071262 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 0m 30s | master passed | | +1 :green_heart: | checkstyle | 0m 12s | master passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed | | -0 :warning: | javac | 0m 26s | hbase-rest generated 1 new + 184 unchanged - 1 fixed = 185 total (was 185) | | +1 :green_heart: | checkstyle | 0m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 50s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 21m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5862 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 2794a284a952 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-rest.txt | | Max. process+thread count | 76 (vs. ulimit of 3) | | modules | C: hbase-rest U: hbase-rest | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5862/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583300564
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -263,47 +264,56 @@ public void testPickColdDataFiles() {
@Test
public void testBlockEvictions() throws Exception {
long capacitySize = 40 * 1024;
-int writeThreads = 3;
-int writerQLen = 64;
+int writeThreads = 1;
+int writerQLen = 4;
int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
-// Setup: Create a bucket cache with lower capacity
-BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
- 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
- DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
-
-// Create three Cache keys with cold data files and a block with hot data.
-// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
-Set cacheKeys = new HashSet<>();
-cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
-cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
-cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
-
-// Create dummy data to be cached and fill the cache completely.
-CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
-
-int blocksIter = 0;
-for (BlockCacheKey key : cacheKeys) {
- bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
- // Ensure that the block is persisted to the file.
- Waiter.waitFor(defaultConf, 1, 100, () ->
(bucketCache.getBackingMap().containsKey(key)));
-}
+// disable any prefetch in parallel to test execution
+defaultConf.setBoolean(CacheConfig.PREFETCH_BLOCKS_ON_OPEN_KEY, false);
Review Comment:
There have been intermittent failures in these tests where the block counts
different than expected at different points. I did observe the prefetch kicking
in during those failures. Hence, I tried to fix these failures. But I realise,
it is better to keep these test fixes different from this original change. I
will remove these changes in the new patch set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583284258
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -2195,17 +2194,19 @@ public Optional
blockFitsIntoTheCache(HFileBlock block) {
@Override
public Optional shouldCacheFile(HFileInfo hFileInfo, Configuration
conf) {
String fileName = hFileInfo.getHFileContext().getHFileName();
-try {
- DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
if (!dataTieringManager.isHotData(hFileInfo, conf)) {
LOG.debug("Data tiering is enabled for file: '{}' and it is not hot
data", fileName);
return Optional.of(false);
+ } else {
+LOG.debug("Data tiering is enabled for file: '{}' and it is hot data",
fileName);
}
-} catch (IllegalStateException e) {
- LOG.error("Error while getting DataTieringManager instance: {}",
e.getMessage());
+} else {
+ LOG.debug("Data tiering feature is not enabled. "
++ " The file: '{}' will be loaded if not already loaded", fileName);
}
-
-// if we don't have the file in fullyCachedFiles, we should cache it
+// if we don't have the file in fullyCachedFiles, we should cache it.
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583283580
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -2195,17 +2194,19 @@ public Optional
blockFitsIntoTheCache(HFileBlock block) {
@Override
public Optional shouldCacheFile(HFileInfo hFileInfo, Configuration
conf) {
String fileName = hFileInfo.getHFileContext().getHFileName();
-try {
- DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
if (!dataTieringManager.isHotData(hFileInfo, conf)) {
LOG.debug("Data tiering is enabled for file: '{}' and it is not hot
data", fileName);
return Optional.of(false);
+ } else {
+LOG.debug("Data tiering is enabled for file: '{}' and it is hot data",
fileName);
Review Comment:
ack!
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -2195,17 +2194,19 @@ public Optional
blockFitsIntoTheCache(HFileBlock block) {
@Override
public Optional shouldCacheFile(HFileInfo hFileInfo, Configuration
conf) {
String fileName = hFileInfo.getHFileContext().getHFileName();
-try {
- DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
if (!dataTieringManager.isHotData(hFileInfo, conf)) {
LOG.debug("Data tiering is enabled for file: '{}' and it is not hot
data", fileName);
return Optional.of(false);
+ } else {
+LOG.debug("Data tiering is enabled for file: '{}' and it is hot data",
fileName);
}
-} catch (IllegalStateException e) {
- LOG.error("Error while getting DataTieringManager instance: {}",
e.getMessage());
+} else {
+ LOG.debug("Data tiering feature is not enabled. "
++ " The file: '{}' will be loaded if not already loaded", fileName);
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1583280569
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePreadReader.java:
##
@@ -40,8 +41,13 @@ public HFilePreadReader(ReaderContext context, HFileInfo
fileInfo, CacheConfig c
final MutableBoolean shouldCache = new MutableBoolean(true);
-// Initialize HFileInfo object with metadata for caching decisions
-fileInfo.initMetaAndIndex(this);
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
+ // Initialize HFileInfo object with metadata for caching decisions.
+ // Initialize the metadata only if the data-tiering is enabled.
+ // If not, the metadata will be initialized later.
+ fileInfo.initMetaAndIndex(this);
+}
Review Comment:
ok, I will remove the check here. The intention was to keep the old
behaviour when the feature is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28523) Use a single get call in REST multiget endpoint
[ https://issues.apache.org/jira/browse/HBASE-28523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-28523: --- Labels: beginner pull-request-available (was: beginner) > Use a single get call in REST multiget endpoint > --- > > Key: HBASE-28523 > URL: https://issues.apache.org/jira/browse/HBASE-28523 > Project: HBase > Issue Type: Improvement > Components: REST >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Labels: beginner, pull-request-available > > The REST multiget endpoint currently issues a separate HBase GET operation > for each key. > Use the method that accepts a list of keys instead. > That should be faster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica.
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
`TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica`. To
reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> Depending on where the
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
has a replica which is in transition, the unassign procedure in that step is
never created/rolled back and we are left with the parent region state in
splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
The region replica case is demonstrated in
`TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica`. To
reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region has
a replica which is in transition, the unassign procedure in that step is never
created/rolled back and we are left with the parent region state in splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> Depending on where the split procedure fails, SplitTableRegionProcedure
> rollback can leave the parent region's RegionStateNode in SPLITTING after
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Environment: (was: Tested on HBase Version 2.5.8 and latest master
branch )
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> Depending on where the split procedure fails, SplitTableRegionProcedure
> rollback can leave the parent region's RegionStateNode in SPLITTING after
> rollback is complete, when the parent region is still online on the assigned
> region server. This leaves active HMaster believing that the parent region is
> offline according to its RegionStates, and causes subsequent procedures that
> require the region to be online like merge/split/move to fail to start. One
> workaround is to restart active HMaster to reset the in memory record of
> region states.
> Two scenarios where this can happen:
> * If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
> has a replica which is in transition, the unassign procedure in that step is
> never created/rolled back and we are left with the parent region state in
> splitting.
> * If region quotas are enabled and a split is run for a region whose
> namespace is at its maximum region quota limit we will fail in
> SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
> with the parent region state in splitting
> To reproduce the region quota case in HBase shell:
> {code:java}
> > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> > 'UniformSplit'}
> > region_a =
> > region_b =
> > split region_a, 'x'
> # HMaster will report:
> pid=405, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
> master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
> Region split not possible for : as quota limits are exceeded ;
> SplitTableRegionProcedure table=test_ns:test_table, parent=...
> > merge_region region_a, region_b
> ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
> OPEN; state=SPLITTING
> > stop_master # trigger hmaster failover
> > merge_region region_a, region_b # merge now succeeds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Affects Version/s: 2.5.8
3.0.0-beta-2
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.5.8, 3.0.0-beta-2
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> Depending on where the split procedure fails, SplitTableRegionProcedure
> rollback can leave the parent region's RegionStateNode in SPLITTING after
> rollback is complete, when the parent region is still online on the assigned
> region server. This leaves active HMaster believing that the parent region is
> offline according to its RegionStates, and causes subsequent procedures that
> require the region to be online like merge/split/move to fail to start. One
> workaround is to restart active HMaster to reset the in memory record of
> region states.
> Two scenarios where this can happen:
> * If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region
> has a replica which is in transition, the unassign procedure in that step is
> never created/rolled back and we are left with the parent region state in
> splitting.
> * If region quotas are enabled and a split is run for a region whose
> namespace is at its maximum region quota limit we will fail in
> SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
> with the parent region state in splitting
> To reproduce the region quota case in HBase shell:
> {code:java}
> > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> > 'UniformSplit'}
> > region_a =
> > region_b =
> > split region_a, 'x'
> # HMaster will report:
> pid=405, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
> master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
> Region split not possible for : as quota limits are exceeded ;
> SplitTableRegionProcedure table=test_ns:test_table, parent=...
> > merge_region region_a, region_b
> ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
> OPEN; state=SPLITTING
> > stop_master # trigger hmaster failover
> > merge_region region_a, region_b # merge now succeeds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start. One
workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this can happen:
* If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region has
a replica which is in transition, the unassign procedure in that step is never
created/rolled back and we are left with the parent region state in splitting.
* If region quotas are enabled and a split is run for a region whose namespace
is at its maximum region quota limit we will fail in
SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left
with the parent region state in splitting
To reproduce the region quota case in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start.
Workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this happens:
*
When a SplitTableRegionProcedure is run for a region whose namespace is at its
maximum region quota limit, the split procedure will fail and rollback, and
Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
state. Hmaster will then refuse to start any subsequent merge/split/move
procedures for that region because it believes the region is not OPEN, until it
is restarted and the in memory record of region states is reset.
In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the parent
region's RegionStateNode state is set to SPLITTING, and the transition is not
written to the meta table. In the next step SPLIT_TABLE_REGION_PRE_OPERATION
the region quota check is done, QuotaExceededException is thrown and the
procedure ends in ROLLEDBACK state without reverting the RegionStateNode back
to OPEN state. Hmaster is left believing the region is in a SPLITTING state
according to its in memory RegionStates, while the region is still online on
the assigned region server and according to meta.
To reproduce in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL:
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
Depending on where the split procedure fails, SplitTableRegionProcedure
rollback can leave the parent region's RegionStateNode in SPLITTING after
rollback is complete, when the parent region is still online on the assigned
region server. This leaves active HMaster believing that the parent region is
offline according to its RegionStates, and causes subsequent procedures that
require the region to be online like merge/split/move to fail to start.
Workaround is to restart active HMaster to reset the in memory record of region
states.
Two scenarios where this happens:
*
When a SplitTableRegionProcedure is run for a region whose namespace is at its
maximum region quota limit, the split procedure will fail and rollback, and
Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
state. Hmaster will then refuse to start any subsequent merge/split/move
procedures for that region because it believes the region is not OPEN, until it
is restarted and the in memory record of region states is reset.
In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the parent
region's RegionStateNode state is set to SPLITTING, and the transition is not
written to the meta table. In the next step SPLIT_TABLE_REGION_PRE_OPERATION
the region quota check is done, QuotaExceededException is thrown and the
procedure ends in ROLLEDBACK state without reverting the RegionStateNode back
to OPEN state. Hmaster is left believing the region is in a SPLITTING state
according to its in memory RegionStates, while the region is still online on
the assigned region server and according to meta.
To reproduce in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
SplitTableRegionProcedure
When a SplitTableRegionProcedure is run for a region whose namespace is at its
maximum region quota limit, the split procedure will fail and rollback, and
Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
state. Hmaster will then refuse to start any subsequent merge/split/move
procedures for that region because it believes the region is not OPEN, until it
is restarted and the in memory record of region states is reset.
In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the parent
region's RegionStateNode state is set to SPLITTING, and the transition is not
written to the meta table. In the next step SPLIT_TABLE_REGION_PRE_OPERATION
the region quota check is done, QuotaExceededException is thrown and the
procedure ends in ROLLEDBACK state without reverting the RegionStateNode back
to OPEN state. Hmaster is left believing the region is in a SPLITTING state
according to its in memory RegionStates, while the region is still online on
the assigned region server and according to meta.
To reproduce in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project:
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Description:
SplitTableRegionProcedure
When a SplitTableRegionProcedure is run for a region whose namespace is at its
maximum region quota limit, the split procedure will fail and rollback, and
Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
state. Hmaster will then refuse to start any subsequent merge/split/move
procedures for that region because it believes the region is not OPEN, until it
is restarted and the in memory record of region states is reset.
In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the parent
region's RegionStateNode state is set to SPLITTING, and the transition is not
written to the meta table. In the next step SPLIT_TABLE_REGION_PRE_OPERATION
the region quota check is done, QuotaExceededException is thrown and the
procedure ends in ROLLEDBACK state without reverting the RegionStateNode back
to OPEN state. Hmaster is left believing the region is in a SPLITTING state
according to its in memory RegionStates, while the region is still online on
the assigned region server and according to meta.
To reproduce in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
was:
When a SplitTableRegionProcedure is run for a region whose namespace is at its
maximum region quota limit, the split procedure will fail and rollback, and
Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
state. Hmaster will then refuse to start any subsequent merge/split/move
procedures for that region because it believes the region is not OPEN, until it
is restarted and the in memory record of region states is reset.
In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the parent
region's RegionStateNode state is set to SPLITTING, and the transition is not
written to the meta table. In the next step SPLIT_TABLE_REGION_PRE_OPERATION
the region quota check is done, QuotaExceededException is thrown and the
procedure ends in ROLLEDBACK state without reverting the RegionStateNode back
to OPEN state. Hmaster is left believing the region is in a SPLITTING state
according to its in memory RegionStates, while the region is still online on
the assigned region server and according to meta.
To reproduce in HBase shell:
{code:java}
> create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> 'UniformSplit'}
> region_a =
> region_b =
> split region_a, 'x'
# HMaster will report:
pid=405, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for : as quota limits are exceeded ;
SplitTableRegionProcedure table=test_ns:test_table, parent=...
> merge_region region_a, region_b
ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
OPEN; state=SPLITTING
> stop_master # trigger hmaster failover
> merge_region region_a, region_b # merge now succeeds {code}
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Environment: Tested on HBase Version 2.5.8 and latest master branch
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> SplitTableRegionProcedure
> When a SplitTableRegionProcedure is run for a region whose namespace is at
> its maximum region quota limit, the split procedure will fail and rollback,
> and Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
> state. Hmaster will then refuse to start any
[jira] [Updated] (HBASE-28533) Split procedure rollback can leave parent region state in SPLITTING after completion
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Roudnitsky updated HBASE-28533:
--
Summary: Split procedure rollback can leave parent region state in
SPLITTING after completion (was: Region split failure due to region quota
limit leaves Hmaster's in memory state for the region in SPLITTING after
procedure rollback)
> Split procedure rollback can leave parent region state in SPLITTING after
> completion
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Environment: Tested on HBase Version 2.5.8 and latest master branch
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> When a SplitTableRegionProcedure is run for a region whose namespace is at
> its maximum region quota limit, the split procedure will fail and rollback,
> and Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
> state. Hmaster will then refuse to start any subsequent merge/split/move
> procedures for that region because it believes the region is not OPEN, until
> it is restarted and the in memory record of region states is reset.
> In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the
> parent region's RegionStateNode state is set to SPLITTING, and the transition
> is not written to the meta table. In the next step
> SPLIT_TABLE_REGION_PRE_OPERATION the region quota check is done,
> QuotaExceededException is thrown and the procedure ends in ROLLEDBACK state
> without reverting the RegionStateNode back to OPEN state. Hmaster is left
> believing the region is in a SPLITTING state according to its in memory
> RegionStates, while the region is still online on the assigned region server
> and according to meta.
> To reproduce in HBase shell:
> {code:java}
> > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> > 'UniformSplit'}
> > region_a =
> > region_b =
> > split region_a, 'x'
> # HMaster will report:
> pid=405, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
> master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
> Region split not possible for : as quota limits are exceeded ;
> SplitTableRegionProcedure table=test_ns:test_table, parent=...
> > merge_region region_a, region_b
> ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
> OPEN; state=SPLITTING
> > stop_master # trigger hmaster failover
> > merge_region region_a, region_b # merge now succeeds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28533) Region split failure due to region quota limit leaves Hmaster's in memory state for the region in SPLITTING after procedure rollback
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841993#comment-17841993
]
Daniel Roudnitsky commented on HBASE-28533:
---
Another case where the parent region state is not reverted to OPEN after
rollback is demonstrated in
TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica , if
we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region has a
replica which is in transition, the unassign procedure in that step is never
created/rolled back and we are left with the parent region state in splitting.
> Region split failure due to region quota limit leaves Hmaster's in memory
> state for the region in SPLITTING after procedure rollback
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Environment: Tested on HBase Version 2.5.8 and latest master branch
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> When a SplitTableRegionProcedure is run for a region whose namespace is at
> its maximum region quota limit, the split procedure will fail and rollback,
> and Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
> state. Hmaster will then refuse to start any subsequent merge/split/move
> procedures for that region because it believes the region is not OPEN, until
> it is restarted and the in memory record of region states is reset.
> In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the
> parent region's RegionStateNode state is set to SPLITTING, and the transition
> is not written to the meta table. In the next step
> SPLIT_TABLE_REGION_PRE_OPERATION the region quota check is done,
> QuotaExceededException is thrown and the procedure ends in ROLLEDBACK state
> without reverting the RegionStateNode back to OPEN state. Hmaster is left
> believing the region is in a SPLITTING state according to its in memory
> RegionStates, while the region is still online on the assigned region server
> and according to meta.
> To reproduce in HBase shell:
> {code:java}
> > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> > 'UniformSplit'}
> > region_a =
> > region_b =
> > split region_a, 'x'
> # HMaster will report:
> pid=405, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
> master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
> Region split not possible for : as quota limits are exceeded ;
> SplitTableRegionProcedure table=test_ns:test_table, parent=...
> > merge_region region_a, region_b
> ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
> OPEN; state=SPLITTING
> > stop_master # trigger hmaster failover
> > merge_region region_a, region_b # merge now succeeds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HBASE-28533) Region split failure due to region quota limit leaves Hmaster's in memory state for the region in SPLITTING after procedure rollback
[
https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-28533 started by Daniel Roudnitsky.
-
> Region split failure due to region quota limit leaves Hmaster's in memory
> state for the region in SPLITTING after procedure rollback
>
>
> Key: HBASE-28533
> URL: https://issues.apache.org/jira/browse/HBASE-28533
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Environment: Tested on HBase Version 2.5.8 and latest master branch
>Reporter: Daniel Roudnitsky
>Assignee: Daniel Roudnitsky
>Priority: Major
>
> When a SplitTableRegionProcedure is run for a region whose namespace is at
> its maximum region quota limit, the split procedure will fail and rollback,
> and Hmaster's in memory RegionStateNode for the region is left in a SPLITTING
> state. Hmaster will then refuse to start any subsequent merge/split/move
> procedures for that region because it believes the region is not OPEN, until
> it is restarted and the in memory record of region states is reset.
> In the first step of the split procedure SPLIT_TABLE_REGION_PREPARE the
> parent region's RegionStateNode state is set to SPLITTING, and the transition
> is not written to the meta table. In the next step
> SPLIT_TABLE_REGION_PRE_OPERATION the region quota check is done,
> QuotaExceededException is thrown and the procedure ends in ROLLEDBACK state
> without reverting the RegionStateNode back to OPEN state. Hmaster is left
> believing the region is in a SPLITTING state according to its in memory
> RegionStates, while the region is still online on the assigned region server
> and according to meta.
> To reproduce in HBase shell:
> {code:java}
> > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2}
> > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO =>
> > 'UniformSplit'}
> > region_a =
> > region_b =
> > split region_a, 'x'
> # HMaster will report:
> pid=405, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via
> master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException:
> Region split not possible for : as quota limits are exceeded ;
> SplitTableRegionProcedure table=test_ns:test_table, parent=...
> > merge_region region_a, region_b
> ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException:
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: is not
> OPEN; state=SPLITTING
> > stop_master # trigger hmaster failover
> > merge_region region_a, region_b # merge now succeeds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28557) Upgrade jline to version 3.x.
[ https://issues.apache.org/jira/browse/HBASE-28557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841987#comment-17841987 ] Abhradeep Kundu commented on HBASE-28557: - I think we first need to update zookeeper which supports jline3, WIP here -> https://issues.apache.org/jira/browse/ZOOKEEPER-3938 After that only we can update jline into hbase along with zookeeper version > Upgrade jline to version 3.x. > - > > Key: HBASE-28557 > URL: https://issues.apache.org/jira/browse/HBASE-28557 > Project: HBase > Issue Type: Improvement >Reporter: Abhradeep Kundu >Assignee: Abhradeep Kundu >Priority: Major > > Need to upgrade jline version to 3.x as 2.x is EOL -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28558 Fix constructors for sub classes of Connection [hbase]
Apache-HBase commented on PR #5861: URL: https://github.com/apache/hbase/pull/5861#issuecomment-2082577342 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 34s | master passed | | +1 :green_heart: | compile | 0m 46s | master passed | | +1 :green_heart: | shadedjars | 6m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 6m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 42s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 15m 3s | hbase-mapreduce in the patch passed. | | | | 41m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5861 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f5cacce8adf7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/testReport/ | | Max. process+thread count | 2927 (vs. ulimit of 3) | | modules | C: hbase-client hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28558 Fix constructors for sub classes of Connection [hbase]
Apache-HBase commented on PR #5861: URL: https://github.com/apache/hbase/pull/5861#issuecomment-2082574109 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 55s | master passed | | +1 :green_heart: | compile | 0m 41s | master passed | | +1 :green_heart: | shadedjars | 5m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 36s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 17m 32s | hbase-mapreduce in the patch passed. | | | | 40m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5861 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fc561ff8b7bd 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/testReport/ | | Max. process+thread count | 3252 (vs. ulimit of 3) | | modules | C: hbase-client hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28558 Fix constructors for sub classes of Connection [hbase]
Apache-HBase commented on PR #5861: URL: https://github.com/apache/hbase/pull/5861#issuecomment-2082567765 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 0m 38s | master passed | | +1 :green_heart: | shadedjars | 5m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 30s | the patch passed | | +1 :green_heart: | compile | 0m 38s | the patch passed | | +1 :green_heart: | javac | 0m 38s | the patch passed | | +1 :green_heart: | shadedjars | 5m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 29s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 14m 47s | hbase-mapreduce in the patch passed. | | | | 36m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5861 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3f4bee2155e0 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/testReport/ | | Max. process+thread count | 2845 (vs. ulimit of 3) | | modules | C: hbase-client hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28558 Fix constructors for sub classes of Connection [hbase]
Apache-HBase commented on PR #5861: URL: https://github.com/apache/hbase/pull/5861#issuecomment-2082560469 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 1m 24s | master passed | | +1 :green_heart: | checkstyle | 0m 33s | master passed | | +1 :green_heart: | spotless | 0m 57s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 26s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | compile | 1m 37s | the patch passed | | -0 :warning: | javac | 0m 43s | hbase-mapreduce generated 2 new + 197 unchanged - 2 fixed = 199 total (was 199) | | +1 :green_heart: | checkstyle | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 6m 31s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 53s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 32m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5861 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 03527c4b484e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4230c42b40 | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-mapreduce.txt | | Max. process+thread count | 77 (vs. ulimit of 3) | | modules | C: hbase-client hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5861/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28555 ThriftConnection does not need ConnectionRegistry [hbase]
Apache9 closed pull request #5860: HBASE-28555 ThriftConnection does not need ConnectionRegistry URL: https://github.com/apache/hbase/pull/5860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28558) Fix constructors for sub classes of Connection
[ https://issues.apache.org/jira/browse/HBASE-28558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-28558: --- Labels: pull-request-available (was: ) > Fix constructors for sub classes of Connection > -- > > Key: HBASE-28558 > URL: https://issues.apache.org/jira/browse/HBASE-28558 > Project: HBase > Issue Type: Bug > Components: Client, test >Reporter: Duo Zhang >Priority: Major > Labels: pull-request-available > > We still have some testing classes which implement Connection but do not have > constructors with ConnectionRegistry as parameter. > This lead to some test failures on branch-2.x, but we'd better also fix them > on master as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28558) Fix constructors for sub classes of Connection
[ https://issues.apache.org/jira/browse/HBASE-28558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-28558: -- Summary: Fix constructors for sub classes of Connection (was: Fix constructors for sub class of Connection) > Fix constructors for sub classes of Connection > -- > > Key: HBASE-28558 > URL: https://issues.apache.org/jira/browse/HBASE-28558 > Project: HBase > Issue Type: Bug > Components: Client, test >Reporter: Duo Zhang >Priority: Major > > We still have some testing classes which implement Connection but do not have > constructors with ConnectionRegistry as parameter. > This lead to some test failures on branch-2.x, but we'd better also fix them > on master as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
wchevreuil commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1582860755
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:
##
@@ -534,7 +534,9 @@ public HRegionServer(final Configuration conf) throws
IOException {
regionServerAccounting = new RegionServerAccounting(conf);
blockCache = BlockCacheFactory.createBlockCache(conf);
- DataTieringManager.instantiate(onlineRegions);
+ if (DataTieringManager.isDataTieringFeatureEnabled(conf)) {
+DataTieringManager.instantiate(conf, onlineRegions);
+ }
Review Comment:
No need for this if check, since you do it internally on
`DataTieringManager.instantiate`. If you want to keep the check inside
`DataTieringManager.instantiate`, then no need for this check here.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -2195,17 +2194,19 @@ public Optional
blockFitsIntoTheCache(HFileBlock block) {
@Override
public Optional shouldCacheFile(HFileInfo hFileInfo, Configuration
conf) {
String fileName = hFileInfo.getHFileContext().getHFileName();
-try {
- DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
if (!dataTieringManager.isHotData(hFileInfo, conf)) {
LOG.debug("Data tiering is enabled for file: '{}' and it is not hot
data", fileName);
return Optional.of(false);
+ } else {
+LOG.debug("Data tiering is enabled for file: '{}' and it is hot data",
fileName);
}
-} catch (IllegalStateException e) {
- LOG.error("Error while getting DataTieringManager instance: {}",
e.getMessage());
+} else {
+ LOG.debug("Data tiering feature is not enabled. "
++ " The file: '{}' will be loaded if not already loaded", fileName);
Review Comment:
Nit: no need for this extra else block, just do the logging.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -2195,17 +2194,19 @@ public Optional
blockFitsIntoTheCache(HFileBlock block) {
@Override
public Optional shouldCacheFile(HFileInfo hFileInfo, Configuration
conf) {
String fileName = hFileInfo.getHFileContext().getHFileName();
-try {
- DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
if (!dataTieringManager.isHotData(hFileInfo, conf)) {
LOG.debug("Data tiering is enabled for file: '{}' and it is not hot
data", fileName);
return Optional.of(false);
+ } else {
+LOG.debug("Data tiering is enabled for file: '{}' and it is hot data",
fileName);
}
-} catch (IllegalStateException e) {
- LOG.error("Error while getting DataTieringManager instance: {}",
e.getMessage());
+} else {
+ LOG.debug("Data tiering feature is not enabled. "
++ " The file: '{}' will be loaded if not already loaded", fileName);
}
-
-// if we don't have the file in fullyCachedFiles, we should cache it
+// if we don't have the file in fullyCachedFiles, we should cache it.
Review Comment:
Nit: avoid useless line changes in the PR.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePreadReader.java:
##
@@ -40,8 +41,13 @@ public HFilePreadReader(ReaderContext context, HFileInfo
fileInfo, CacheConfig c
final MutableBoolean shouldCache = new MutableBoolean(true);
-// Initialize HFileInfo object with metadata for caching decisions
-fileInfo.initMetaAndIndex(this);
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
+ // Initialize HFileInfo object with metadata for caching decisions.
+ // Initialize the metadata only if the data-tiering is enabled.
+ // If not, the metadata will be initialized later.
+ fileInfo.initMetaAndIndex(this);
+}
Review Comment:
Let us please avoid spaghetti code and keep components cohesive. We shall
not pollute the reader with data tiering logic. As @vinayakphegde mentioned,
initiMetaAndIndex already knows it should perform init logic only once in the
flow, so this call here won't add extra execution.
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -263,47 +264,56 @@ public void testPickColdDataFiles() {
@Test
public void testBlockEvictions() throws Exception {
long capacitySize = 40 * 1024;
-int writeThreads = 3;
-int writerQLen = 64;
+int writeThreads = 1;
+int writerQLen = 4;
int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
-// Setup: Create a bucket cache
[jira] [Created] (HBASE-28558) Fix constructors for sub class of Connection
Duo Zhang created HBASE-28558: - Summary: Fix constructors for sub class of Connection Key: HBASE-28558 URL: https://issues.apache.org/jira/browse/HBASE-28558 Project: HBase Issue Type: Bug Components: Client, test Reporter: Duo Zhang We still have some testing classes which implement Connection but do not have constructors with ConnectionRegistry as parameter. This lead to some test failures on branch-2.x, but we'd better also fix them on master as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1582729929
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePreadReader.java:
##
@@ -40,8 +41,13 @@ public HFilePreadReader(ReaderContext context, HFileInfo
fileInfo, CacheConfig c
final MutableBoolean shouldCache = new MutableBoolean(true);
-// Initialize HFileInfo object with metadata for caching decisions
-fileInfo.initMetaAndIndex(this);
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
+ // Initialize HFileInfo object with metadata for caching decisions.
+ // Initialize the metadata only if the data-tiering is enabled.
+ // If not, the metadata will be initialized later.
+ fileInfo.initMetaAndIndex(this);
+}
Review Comment:
This check is to avoid any changes to the old functionality. We had observed
that this line does introduce some performance regression leading to a unit
test failure. We keep the old behaviour to initialise the metadata at the
original location when data-tiering is not enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[
https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-28405.
---
Resolution: Fixed
Changed split to splitRegionAsync. Pushed the addendum to all branch-2.x.
> Region open procedure silently returns without notifying the parent proc
>
>
> Key: HBASE-28405
> URL: https://issues.apache.org/jira/browse/HBASE-28405
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Aman Poonia
>Assignee: Aman Poonia
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> *We had a scenario in production where a merge operation had failed as below*
> _2024-02-11 10:53:57,715 ERROR [PEWorker-31]
> assignment.MergeTableRegionsProcedure - Error trying to merge
> [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in
> table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_
> _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING,
> location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_
> _at
> org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_
> _at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_
> _at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_
> _at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_
> _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_
> _at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_
> *Now when we do rollback of failed merge operation we see a issue where
> region is in state opened until the RS holding it stopped.*
> Rollback create a TRSP as below
> _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor -
> Stored [pid=26674602,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=table1,
> region=a92008b76ccae47d55c590930b837036, ASSIGN]_
> *and rollback finished successfully*
> _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor -
> Rolled back pid=26673594, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.HBaseIOException via
> master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent
> region state=MERGING, location=rs-229,60020,1707587658182, table=table1,
> region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up;
> MergeTableRegionsProcedure table=table1,
> regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b],
> force=false exec-time=1.4820 sec_
> *We create a procedure to open the region a92008b76ccae47d55c590930b837036.
> Intrestingly we didnt close the region as creation of procedure to close
> regions had thrown exception and not execution of procedure. When we run TRSP
> it sends a OpenRegionProcedure which is handled by AssignRegionHandler. This
> handlers on execution suggests that region is already online*
> Sequence of events are as follow
> _2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore -
> pid=26674602 updating hbase:meta row=a92008b76ccae47d55c590930b837036,
> regionState=OPENING, regionLocation=rs-210,60020,1707596461539_
> _2024-02-11 10:53:58,920 INFO [PEWorker-58] procedure2.ProcedureExecutor -
> Initialized subprocedures=[\\{pid=26675798, ppid=26674602, state=RUNNABLE;
> OpenRegionProcedure a92008b76ccae47d55c590930b837036,
> server=rs-210,60020,1707596461539}]_
> _2024-02-11 10:53:59,074 WARN [REGION-regionserver/rs-210:60020-10]
> handler.AssignRegionHandler - Received OPEN for
> table1,r1,1685436252488.a92008b76ccae47d55c590930b837036. which is already
> online_
--
This message was sent by Atlassian Jira
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1582729929
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePreadReader.java:
##
@@ -40,8 +41,13 @@ public HFilePreadReader(ReaderContext context, HFileInfo
fileInfo, CacheConfig c
final MutableBoolean shouldCache = new MutableBoolean(true);
-// Initialize HFileInfo object with metadata for caching decisions
-fileInfo.initMetaAndIndex(this);
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
+ // Initialize HFileInfo object with metadata for caching decisions.
+ // Initialize the metadata only if the data-tiering is enabled.
+ // If not, the metadata will be initialized later.
+ fileInfo.initMetaAndIndex(this);
+}
Review Comment:
This check is to avoid the old functionality as is. We had observed that
this line does introduce some performance regression leading to a unit test
failure. We keep the old behaviour to initialise the metadata at the original
location when data-tiering is not enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1582727630
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:
##
@@ -534,7 +534,9 @@ public HRegionServer(final Configuration conf) throws
IOException {
regionServerAccounting = new RegionServerAccounting(conf);
blockCache = BlockCacheFactory.createBlockCache(conf);
- DataTieringManager.instantiate(onlineRegions);
+ if (DataTieringManager.isDataTieringFeatureEnabled(conf)) {
+DataTieringManager.instantiate(conf, onlineRegions);
+ }
Review Comment:
Yes, we will require server restart since we are initialising the
DataTieringManager during server restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
jhungund commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1582726351
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -61,25 +63,25 @@ private DataTieringManager(Map
onlineRegions) {
* Initializes the DataTieringManager instance with the provided map of
online regions.
* @param onlineRegions A map containing online regions.
*/
- public static synchronized void instantiate(Map
onlineRegions) {
-if (instance == null) {
- instance = new DataTieringManager(onlineRegions);
- LOG.info("DataTieringManager instantiated successfully.");
+ public static synchronized void instantiate(Configuration conf,
Review Comment:
This is an additional check to avoid an unintentional use if someone makes a
call to this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28557) Upgrade jline to version 3.x.
[ https://issues.apache.org/jira/browse/HBASE-28557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhradeep Kundu updated HBASE-28557: Description: Need to upgrade jline version to 3.x as 2.x is EOL > Upgrade jline to version 3.x. > - > > Key: HBASE-28557 > URL: https://issues.apache.org/jira/browse/HBASE-28557 > Project: HBase > Issue Type: Improvement >Reporter: Abhradeep Kundu >Assignee: Abhradeep Kundu >Priority: Major > > Need to upgrade jline version to 3.x as 2.x is EOL -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-28557) Upgrade jline to version 3.x.
[ https://issues.apache.org/jira/browse/HBASE-28557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhradeep Kundu reassigned HBASE-28557: --- Assignee: Abhradeep Kundu > Upgrade jline to version 3.x. > - > > Key: HBASE-28557 > URL: https://issues.apache.org/jira/browse/HBASE-28557 > Project: HBase > Issue Type: Improvement >Reporter: Abhradeep Kundu >Assignee: Abhradeep Kundu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28557) Upgrade jline to version 3.x.
Abhradeep Kundu created HBASE-28557: --- Summary: Upgrade jline to version 3.x. Key: HBASE-28557 URL: https://issues.apache.org/jira/browse/HBASE-28557 Project: HBase Issue Type: Improvement Reporter: Abhradeep Kundu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-28405) Region open procedure silently returns without notifying the parent proc
[ https://issues.apache.org/jira/browse/HBASE-28405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-28405: --- We need to apply an addendum to branch-2.x. We fixed a problem when rolling back MergeTableRegionsProcedure, and then TestNamespaceAuditor.testRegionMerge started to test what is really want to test, that's why we need to change the exception type from DoNotRetryRegionException to DoNotRetryIOException. Before this PR, we will get a region state error exception, which is not what we want to test. But the problem for branch-2 is that, the behavior for the split method is a bit different. The spit procedure will fail with quota exceeded, but the method will return normally so the assertion will fail. We need to find a way on how to get the exception for branch-2.x. > Region open procedure silently returns without notifying the parent proc > > > Key: HBASE-28405 > URL: https://issues.apache.org/jira/browse/HBASE-28405 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Affects Versions: 2.4.17, 2.5.8 >Reporter: Aman Poonia >Assignee: Aman Poonia >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9 > > > *We had a scenario in production where a merge operation had failed as below* > _2024-02-11 10:53:57,715 ERROR [PEWorker-31] > assignment.MergeTableRegionsProcedure - Error trying to merge > [a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b] in > table1 (in state=MERGE_TABLE_REGIONS_CLOSE_REGIONS)_ > _org.apache.hadoop.hbase.HBaseIOException: The parent region state=MERGING, > location=rs-229,60020,1707587658182, table=table1, > region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up_ > _at > org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120)_ > _at > org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.createUnassignProcedures(MergeTableRegionsProcedure.java:648)_ > _at > org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:205)_ > _at > org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:79)_ > _at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)_ > _at > org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)_ > _at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)_ > _at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)_ > _at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)_ > _at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1964)_ > _at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)_ > _at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1991)_ > *Now when we do rollback of failed merge operation we see a issue where > region is in state opened until the RS holding it stopped.* > Rollback create a TRSP as below > _2024-02-11 10:53:57,719 DEBUG [PEWorker-31] procedure2.ProcedureExecutor - > Stored [pid=26674602, > state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; > TransitRegionStateProcedure table=table1, > region=a92008b76ccae47d55c590930b837036, ASSIGN]_ > *and rollback finished successfully* > _2024-02-11 10:53:57,721 INFO [PEWorker-31] procedure2.ProcedureExecutor - > Rolled back pid=26673594, state=ROLLEDBACK, > exception=org.apache.hadoop.hbase.HBaseIOException via > master-merge-regions:org.apache.hadoop.hbase.HBaseIOException: The parent > region state=MERGING, location=rs-229,60020,1707587658182, table=table1, > region=f56752ae9f30fad9de5a80a8ba578e4b is currently in transition, give up; > MergeTableRegionsProcedure table=table1, > regions=[a92008b76ccae47d55c590930b837036, f56752ae9f30fad9de5a80a8ba578e4b], > force=false exec-time=1.4820 sec_ > *We create a procedure to open the region a92008b76ccae47d55c590930b837036. > Intrestingly we didnt close the region as creation of procedure to close > regions had thrown exception and not execution of procedure. When we run TRSP > it sends a OpenRegionProcedure which is handled by AssignRegionHandler. This > handlers on execution suggests that region is already online* > Sequence of events are as follow > _2024-02-11 10:53:58,919 INFO [PEWorker-58] assignment.RegionStateStore - > pid=26674602 updating hbase:meta
Re: [PR] HBASE-28535: Add a region-server wide key to enable data-tiering. [hbase]
vinayakphegde commented on code in PR #5856:
URL: https://github.com/apache/hbase/pull/5856#discussion_r1582666798
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePreadReader.java:
##
@@ -40,8 +41,13 @@ public HFilePreadReader(ReaderContext context, HFileInfo
fileInfo, CacheConfig c
final MutableBoolean shouldCache = new MutableBoolean(true);
-// Initialize HFileInfo object with metadata for caching decisions
-fileInfo.initMetaAndIndex(this);
+DataTieringManager dataTieringManager = DataTieringManager.getInstance();
+if (dataTieringManager != null) {
+ // Initialize HFileInfo object with metadata for caching decisions.
+ // Initialize the metadata only if the data-tiering is enabled.
+ // If not, the metadata will be initialized later.
+ fileInfo.initMetaAndIndex(this);
+}
Review Comment:
I don't think this part is needed. We decided not to use Data Tiering logic
here and restrict it to Bucket Cache, which I will take care of in the
refactoring JIRA. Additionally, if the concern is to avoid executing
initMetaAndIndex twice, then I have modified initMetaAndIndex to only execute
once.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -45,6 +45,8 @@
@InterfaceAudience.Private
public class DataTieringManager {
private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATA_TIERING_ENABLED_KEY =
"hbase.hstore.datatiering.enable";
Review Comment:
Since it is a cluster-wide (or region server-level) configuration, I think
the format should be something like `hbase.regionserver.*`, although I'm not
entirely sure. Additionally, could we use a better name for the variable to
indicate that this is a cluster-wide configuration? like,
`GLOBAL_DATA_TIERING_ENABLED_KEY`?
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:
##
@@ -534,7 +534,9 @@ public HRegionServer(final Configuration conf) throws
IOException {
regionServerAccounting = new RegionServerAccounting(conf);
blockCache = BlockCacheFactory.createBlockCache(conf);
- DataTieringManager.instantiate(onlineRegions);
+ if (DataTieringManager.isDataTieringFeatureEnabled(conf)) {
+DataTieringManager.instantiate(conf, onlineRegions);
+ }
Review Comment:
Since we are not instantiating the DataTieringManager itself when global
configuration is not enabled, do we need to restart every time we change that
global parameter?
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -61,25 +63,25 @@ private DataTieringManager(Map
onlineRegions) {
* Initializes the DataTieringManager instance with the provided map of
online regions.
* @param onlineRegions A map containing online regions.
*/
- public static synchronized void instantiate(Map
onlineRegions) {
-if (instance == null) {
- instance = new DataTieringManager(onlineRegions);
- LOG.info("DataTieringManager instantiated successfully.");
+ public static synchronized void instantiate(Configuration conf,
Review Comment:
Why do we need this Configuration parameter? We only enter here when
isDataTieringFeatureEnabled(conf) is true, so why do we need to check it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28555 ThriftConnection does not need ConnectionRegistry [hbase]
Apache-HBase commented on PR #5860: URL: https://github.com/apache/hbase/pull/5860#issuecomment-2082107326 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 51s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | branch-2 passed | | +1 :green_heart: | compile | 0m 23s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 22s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | shadedjars | 5m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 32s | hbase-client in the patch passed. | | | | 28m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5860 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 533890d05ef4 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d6beb6194e | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/testReport/ | | Max. process+thread count | 373 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28555 ThriftConnection does not need ConnectionRegistry [hbase]
Apache-HBase commented on PR #5860: URL: https://github.com/apache/hbase/pull/5860#issuecomment-2082103754 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 20s | branch-2 passed | | +1 :green_heart: | compile | 0m 18s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 17s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | shadedjars | 5m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 0s | hbase-client in the patch passed. | | | | 25m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5860 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f41d8667e72b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d6beb6194e | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/testReport/ | | Max. process+thread count | 359 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28555 ThriftConnection does not need ConnectionRegistry [hbase]
Apache-HBase commented on PR #5860: URL: https://github.com/apache/hbase/pull/5860#issuecomment-2082098007 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | branch-2 passed | | +1 :green_heart: | compile | 0m 49s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 20s | branch-2 passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 50s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 5s | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 22m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5860 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 040282025feb 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d6beb6194e | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5860/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28556) Reduce memory copying in Rest server when serializing CellModel to Protobuf
[ https://issues.apache.org/jira/browse/HBASE-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated HBASE-28556: Summary: Reduce memory copying in Rest server when serializing CellModel to Protobuf (was: Reduce memory copying in Rest server when converting CellModel to Protobuf) > Reduce memory copying in Rest server when serializing CellModel to Protobuf > --- > > Key: HBASE-28556 > URL: https://issues.apache.org/jira/browse/HBASE-28556 > Project: HBase > Issue Type: Improvement > Components: REST >Reporter: Istvan Toth >Priority: Major > > The REST server does a lot of unneccessary coping, which could be avoided at > least for protobuf encoding. > - It uses ByteStringer to handle ByteBuffer backed Cells. However, it uses > the client API, so it sjpuld never encounter ByteBuffer backed cells. > - It clones everything from the cells (sometimes multiple times) before > serializing to protbuf. > We could mimic the structure in Cell, with array, offset and length for each > field, and use the appropriate protobuf setters to avoid the extra copies. > There may or may not be a way to do the same for JSON and XML via jax-rs, I > don't know the frameworks well enough to tell, but if not, we could just do > the copying in the getters for them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28556) Reduce memory copying in Rest server when converting CellModel to Protobuf
Istvan Toth created HBASE-28556: --- Summary: Reduce memory copying in Rest server when converting CellModel to Protobuf Key: HBASE-28556 URL: https://issues.apache.org/jira/browse/HBASE-28556 Project: HBase Issue Type: Improvement Components: REST Reporter: Istvan Toth The REST server does a lot of unneccessary coping, which could be avoided at least for protobuf encoding. - It uses ByteStringer to handle ByteBuffer backed Cells. However, it uses the client API, so it sjpuld never encounter ByteBuffer backed cells. - It clones everything from the cells (sometimes multiple times) before serializing to protbuf. We could mimic the structure in Cell, with array, offset and length for each field, and use the appropriate protobuf setters to avoid the extra copies. There may or may not be a way to do the same for JSON and XML via jax-rs, I don't know the frameworks well enough to tell, but if not, we could just do the copying in the getters for them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28536 Fix `Disable Stripe Compaction` run error in document [hbase]
zhaomoran commented on PR #5836: URL: https://github.com/apache/hbase/pull/5836#issuecomment-2081946460 @Apache9 This pr is ready. Help me deal with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
