Re: [PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
Apache-HBase commented on PR #5559: URL: https://github.com/apache/hbase/pull/5559#issuecomment-1842255503 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 0m 37s | master passed | | +1 :green_heart: | shadedjars | 5m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 19s | the patch passed | | +1 :green_heart: | compile | 0m 36s | the patch passed | | +1 :green_heart: | javac | 0m 36s | the patch passed | | +1 :green_heart: | shadedjars | 5m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 226m 9s | hbase-server in the patch failed. | | | | 247m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5559 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d2735a4f3686 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/testReport/ | | Max. process+thread count | 4640 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
Apache-HBase commented on PR #5559: URL: https://github.com/apache/hbase/pull/5559#issuecomment-1842227549 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 53s | master passed | | +1 :green_heart: | compile | 0m 43s | master passed | | +1 :green_heart: | shadedjars | 5m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | shadedjars | 5m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 213m 20s | hbase-server in the patch passed. | | | | 235m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5559 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 355011e1968a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/testReport/ | | Max. process+thread count | 4672 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28209: Create a jmx metrics to expose the oldWALs directory size [hbase]
Apache-HBase commented on PR #5556: URL: https://github.com/apache/hbase/pull/5556#issuecomment-1842208969 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 53s | branch-2 passed | | +1 :green_heart: | compile | 5m 7s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 9s | branch-2 passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 10m 2s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 4m 58s | the patch passed | | +1 :green_heart: | javac | 4m 58s | the patch passed | | +1 :green_heart: | checkstyle | 1m 7s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 13m 50s | Patch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 10m 49s | the patch passed | ||| _ Other Tests _ | | -1 :x: | asflicense | 0m 51s | The patch generated 1 ASF License warnings. | | | | 58m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5556/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5556 | | JIRA Issue | HBASE-28209 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml | | uname | Linux 1a979c97b3d3 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / f792149347 | | Default Java | Eclipse Adoptium-11.0.17+8 | | asflicense | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5556/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 181 (vs. ulimit of 3) | | modules | C: hbase-common hbase-hadoop-compat hbase-hadoop2-compat hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5556/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28238 rpcservice should perform some important admin operation to priority ADMIN_QOS [hbase]
Apache-HBase commented on PR #5558: URL: https://github.com/apache/hbase/pull/5558#issuecomment-1842207382 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 21s | master passed | | +1 :green_heart: | compile | 0m 41s | master passed | | +1 :green_heart: | shadedjars | 4m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 21s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | shadedjars | 4m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 230m 27s | hbase-server in the patch passed. | | | | 252m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5558 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e1a99b435cbd 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/testReport/ | | Max. process+thread count | 4325 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28238 rpcservice should perform some important admin operation to priority ADMIN_QOS [hbase]
Apache-HBase commented on PR #5558: URL: https://github.com/apache/hbase/pull/5558#issuecomment-1842202161 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 56s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | shadedjars | 4m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 223m 59s | hbase-server in the patch passed. | | | | 246m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5558 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 353da2974974 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/testReport/ | | Max. process+thread count | 4577 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
Apache-HBase commented on PR #5559: URL: https://github.com/apache/hbase/pull/5559#issuecomment-1842044346 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 40s | master passed | | +1 :green_heart: | compile | 2m 32s | master passed | | +1 :green_heart: | checkstyle | 0m 33s | master passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | the patch passed | | +1 :green_heart: | compile | 2m 32s | the patch passed | | -0 :warning: | javac | 2m 32s | hbase-server generated 1 new + 194 unchanged - 1 fixed = 195 total (was 195) | | +1 :green_heart: | checkstyle | 0m 34s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 56s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 32m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5559 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 453f2a6254b1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | Max. process+thread count | 82 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28238 rpcservice should perform some important admin operation to priority ADMIN_QOS [hbase]
Apache-HBase commented on PR #5558: URL: https://github.com/apache/hbase/pull/5558#issuecomment-1842025607 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 32s | master passed | | +1 :green_heart: | compile | 2m 20s | master passed | | +1 :green_heart: | checkstyle | 0m 33s | master passed | | +1 :green_heart: | spotless | 0m 39s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 41s | the patch passed | | +1 :green_heart: | compile | 2m 28s | the patch passed | | +1 :green_heart: | javac | 2m 28s | the patch passed | | +1 :green_heart: | checkstyle | 0m 34s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 26s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 32m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5558 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 0adbfe2c03a3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28241 Fixed the check for an ongoing Snapshot before proceeding with the merge region operation. [hbase]
Apache9 commented on code in PR #5560:
URL: https://github.com/apache/hbase/pull/5560#discussion_r1416608198
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java:
##
@@ -442,7 +442,7 @@ protected ProcedureMetrics
getProcedureMetrics(MasterProcedureEnv env) {
private boolean prepareMergeRegion(final MasterProcedureEnv env) throws
IOException {
// Fail if we are taking snapshot for the given table
TableName tn = regionsToMerge[0].getTable();
-if (env.getMasterServices().getSnapshotManager().isTakingSnapshot(tn)) {
+if
(env.getMasterServices().getSnapshotManager().isTableTakingAnySnapshot(tn)) {
Review Comment:
Let's change the title of the issue and then fix them both.
And also better include the UT provided by @frostruan ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28241 Fixed the check for an ongoing Snapshot before proceeding with the merge region operation. [hbase]
frostruan commented on code in PR #5560:
URL: https://github.com/apache/hbase/pull/5560#discussion_r1416595305
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java:
##
@@ -442,7 +442,7 @@ protected ProcedureMetrics
getProcedureMetrics(MasterProcedureEnv env) {
private boolean prepareMergeRegion(final MasterProcedureEnv env) throws
IOException {
// Fail if we are taking snapshot for the given table
TableName tn = regionsToMerge[0].getTable();
-if (env.getMasterServices().getSnapshotManager().isTakingSnapshot(tn)) {
+if
(env.getMasterServices().getSnapshotManager().isTableTakingAnySnapshot(tn)) {
Review Comment:
+1
SplitTableRegionProcedure has the same problem. Maybe we can fix it also ?
Or File a new issue ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28238) rpcservice should perform some important admin operation to priority ADMIN_QOS
[
https://issues.apache.org/jira/browse/HBASE-28238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
selina.yan updated HBASE-28238:
---
Description:
some important admin operations should be admin priority,like
compactionSwitch,rollWALWriter,updateConfiguration.
these operations called by admin should be considered a higher priority
operation and given to the priority queue, but is currently considered to have
the same priority as read or write requests. If the handle is occupied, the
operations cannot be execute successful. For example, in our scenario, some
abnormal phoenix sql is killed by controlling the amount of data that scan can
filter with dynamic parameters. Abnormal sql often results in a full handle. In
this case, the configuration cannot be dynamically modified to implement
emergency traffic limiting.
{code:java}
@Override
@QosPriority(priority = HConstants.ADMIN_QOS)
public UpdateConfigurationResponse updateConfiguration(
RpcController controller, UpdateConfigurationRequest request)
throws ServiceException {{code}
was:
The updateConfiguration operation called by admin should be considered a higher
priority operation and given to the priority queue, but is currently considered
to have the same priority as read and write requests. If the handle is
occupied, the configuration cannot be updated. In fact, in our scenario, some
abnormal phoenix sql is killed by controlling the amount of data that scan can
filter with dynamic parameters. Abnormal sql often results in a full handle. In
this case, the configuration cannot be dynamically modified to implement
emergency traffic limiting.
{code:java}
@Override
@QosPriority(priority = HConstants.ADMIN_QOS)
public UpdateConfigurationResponse updateConfiguration(
RpcController controller, UpdateConfigurationRequest request)
throws ServiceException {{code}
Summary: rpcservice should perform some important admin operation to
priority ADMIN_QOS (was: rsrpcservice should perform the updateConfiguration
operation to priority @QosPriority(priority = HConstants.ADMIN_QOS))
> rpcservice should perform some important admin operation to priority ADMIN_QOS
> --
>
> Key: HBASE-28238
> URL: https://issues.apache.org/jira/browse/HBASE-28238
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.5.0
>Reporter: selina.yan
>Assignee: selina.yan
>Priority: Minor
>
> some important admin operations should be admin priority,like
> compactionSwitch,rollWALWriter,updateConfiguration.
> these operations called by admin should be considered a higher priority
> operation and given to the priority queue, but is currently considered to
> have the same priority as read or write requests. If the handle is occupied,
> the operations cannot be execute successful. For example, in our scenario,
> some abnormal phoenix sql is killed by controlling the amount of data that
> scan can filter with dynamic parameters. Abnormal sql often results in a full
> handle. In this case, the configuration cannot be dynamically modified to
> implement emergency traffic limiting.
> {code:java}
> @Override
> @QosPriority(priority = HConstants.ADMIN_QOS)
> public UpdateConfigurationResponse updateConfiguration(
> RpcController controller, UpdateConfigurationRequest request)
> throws ServiceException {{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793477#comment-17793477
]
Haiping lv commented on HBASE-28241:
Hi [~frostruan] Please help review the code.
Thank you very much.
> The snapshot operation encountered an NPE and failed.
> -
>
> Key: HBASE-28241
> URL: https://issues.apache.org/jira/browse/HBASE-28241
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Haiping lv
>Assignee: Haiping lv
>Priority: Major
>
> Executing a merge region before the snapshot operation is completed will
> result in an NPE error for the snapshot operation and cause it to fail
> Triggering logic
> # Execute the snapshot command.
> # Merge regions before the snapshot is completed.
> # After the merge is completed, the two parent regions will be cleaned up.
> # An NPE will be reported when the snapshot operation is performed on that
> parent region. The log is as follow:
> {code:java}
> 2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714,
> state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
> c47539f1d6032ba1a037f5279c22baa0
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 [hbase]
Apache-HBase commented on PR #5561: URL: https://github.com/apache/hbase/pull/5561#issuecomment-1841793860 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 19s | master passed | | +1 :green_heart: | compile | 0m 42s | master passed | | +1 :green_heart: | shadedjars | 4m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 0m 43s | the patch passed | | +1 :green_heart: | javac | 0m 43s | the patch passed | | +1 :green_heart: | shadedjars | 4m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 237m 0s | hbase-server in the patch failed. | | | | 258m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5561 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f79662dda915 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/testReport/ | | Max. process+thread count | 4889 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 [hbase]
Apache-HBase commented on PR #5561: URL: https://github.com/apache/hbase/pull/5561#issuecomment-184164 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 4m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 42s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | shadedjars | 4m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 220m 48s | hbase-server in the patch passed. | | | | 243m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5561 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 44332dbc22ab 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/testReport/ | | Max. process+thread count | 4819 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27532) Add block bytes scanned metrics
[ https://issues.apache.org/jira/browse/HBASE-27532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-27532: -- Fix Version/s: 2.6.0 3.0.0-beta-1 Release Note: Adds blockBytesScannedCount metrics for the regionserver, table, and operations in JMX. Adds a countOfBlockBytesScanned to ScanMetrics. Resolution: Fixed Status: Resolved (was: Patch Available) Apparently, I forgot to close this. It was merged to master/branch-3/branch-2 > Add block bytes scanned metrics > --- > > Key: HBASE-27532 > URL: https://issues.apache.org/jira/browse/HBASE-27532 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: patch-available > Fix For: 2.6.0, 3.0.0-beta-1 > > > Initially in HBASE-14978, but more recently improved in HBASE-27558 and > HBASE-27570, we now track the amount of block bytes scanned by read requests. > Random read use-cases (which might read 1 small row from a 64kb block) and > heavily filtered scans (which might scan many blocks in order to return a > needle in a haystack row) can unknowingly cause RegionServers to do more work > than a user expects. Now that we have block IO accounting, we should expose > this to the user in the form of jmx metrics and ScanMetrics. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28215) Region reopen procedure should support some sort of throttling
[
https://issues.apache.org/jira/browse/HBASE-28215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793417#comment-17793417
]
Hudson commented on HBASE-28215:
Results for branch branch-3
[build #97 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Region reopen procedure should support some sort of throttling
> --
>
> Key: HBASE-28215
> URL: https://issues.apache.org/jira/browse/HBASE-28215
> Project: HBase
> Issue Type: Improvement
> Components: master, proc-v2
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
> Fix For: 2.6.0, 3.0.0-beta-1
>
>
> The mass reopening of regions caused by a table descriptor modification can
> be quite disruptive. For latency/error sensitive workloads, like our user
> facing traffic, we need to be very careful about when we modify table
> descriptors, and it can be virtually impossible to do it painlessly for busy
> tables.
> It would be nice if we supported configurable batching/throttling of
> reopenings so that the amplitude of any disruption can be kept relatively
> small.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28199) Phase I: Suspend TRSP and SCP when updating meta
[
https://issues.apache.org/jira/browse/HBASE-28199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793415#comment-17793415
]
Hudson commented on HBASE-28199:
Results for branch branch-3
[build #97 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Phase I: Suspend TRSP and SCP when updating meta
>
>
> Key: HBASE-28199
> URL: https://issues.apache.org/jira/browse/HBASE-28199
> Project: HBase
> Issue Type: Sub-task
> Components: master, proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-beta-1
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28209) Create a jmx metrics to expose the oldWALs directory size
[
https://issues.apache.org/jira/browse/HBASE-28209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793416#comment-17793416
]
Hudson commented on HBASE-28209:
Results for branch branch-3
[build #97 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/97/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Create a jmx metrics to expose the oldWALs directory size
> -
>
> Key: HBASE-28209
> URL: https://issues.apache.org/jira/browse/HBASE-28209
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6, 4.0.0-alpha-1
>Reporter: Vinayak Hegde
>Assignee: Vinayak Hegde
>Priority: Major
> Fix For: 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> Create a jmx metrics that can return the size of the old WALs in bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 [hbase]
Apache-HBase commented on PR #5561: URL: https://github.com/apache/hbase/pull/5561#issuecomment-1841505148 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 40s | master passed | | +1 :green_heart: | compile | 2m 27s | master passed | | +1 :green_heart: | checkstyle | 0m 34s | master passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 2m 24s | the patch passed | | +1 :green_heart: | javac | 2m 24s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 3 new + 0 unchanged - 1 fixed = 3 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 15s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 31m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5561 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux ab603117c156 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28215) Region reopen procedure should support some sort of throttling
[
https://issues.apache.org/jira/browse/HBASE-28215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793407#comment-17793407
]
Hudson commented on HBASE-28215:
Results for branch master
[build #958 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Region reopen procedure should support some sort of throttling
> --
>
> Key: HBASE-28215
> URL: https://issues.apache.org/jira/browse/HBASE-28215
> Project: HBase
> Issue Type: Improvement
> Components: master, proc-v2
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
> Fix For: 2.6.0, 3.0.0-beta-1
>
>
> The mass reopening of regions caused by a table descriptor modification can
> be quite disruptive. For latency/error sensitive workloads, like our user
> facing traffic, we need to be very careful about when we modify table
> descriptors, and it can be virtually impossible to do it painlessly for busy
> tables.
> It would be nice if we supported configurable batching/throttling of
> reopenings so that the amplitude of any disruption can be kept relatively
> small.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28199) Phase I: Suspend TRSP and SCP when updating meta
[
https://issues.apache.org/jira/browse/HBASE-28199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793405#comment-17793405
]
Hudson commented on HBASE-28199:
Results for branch master
[build #958 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Phase I: Suspend TRSP and SCP when updating meta
>
>
> Key: HBASE-28199
> URL: https://issues.apache.org/jira/browse/HBASE-28199
> Project: HBase
> Issue Type: Sub-task
> Components: master, proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-beta-1
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28209) Create a jmx metrics to expose the oldWALs directory size
[
https://issues.apache.org/jira/browse/HBASE-28209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793406#comment-17793406
]
Hudson commented on HBASE-28209:
Results for branch master
[build #958 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/958/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Create a jmx metrics to expose the oldWALs directory size
> -
>
> Key: HBASE-28209
> URL: https://issues.apache.org/jira/browse/HBASE-28209
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6, 4.0.0-alpha-1
>Reporter: Vinayak Hegde
>Assignee: Vinayak Hegde
>Priority: Major
> Fix For: 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> Create a jmx metrics that can return the size of the old WALs in bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 [hbase]
Apache-HBase commented on PR #5561: URL: https://github.com/apache/hbase/pull/5561#issuecomment-1841437764 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 55s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 4m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | -1 :x: | shadedjars | 3m 47s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 13m 24s | hbase-server in the patch failed. | | | | 32m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5561 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4a064a44f920 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/testReport/ | | Max. process+thread count | 1631 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 [hbase]
Apache-HBase commented on PR #5561: URL: https://github.com/apache/hbase/pull/5561#issuecomment-1841437591 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 50s | master passed | | +1 :green_heart: | compile | 2m 28s | master passed | | +1 :green_heart: | checkstyle | 0m 34s | master passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 31s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 41s | the patch passed | | +1 :green_heart: | compile | 2m 29s | the patch passed | | +1 :green_heart: | javac | 2m 29s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 7 new + 0 unchanged - 1 fixed = 7 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 28s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | -1 :x: | spotless | 0m 21s | patch has 34 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | asflicense | 0m 13s | The patch generated 1 ASF License warnings. | | | | 32m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5561 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux c1a2f41f0783 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-general-check/output/patch-spotless.txt | | asflicense | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 [hbase]
Apache-HBase commented on PR #5561: URL: https://github.com/apache/hbase/pull/5561#issuecomment-1841431107 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 42s | master passed | | +1 :green_heart: | compile | 0m 43s | master passed | | +1 :green_heart: | shadedjars | 4m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | -1 :x: | shadedjars | 3m 43s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 11m 6s | hbase-server in the patch failed. | | | | 29m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5561 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ae77aecf3449 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Temurin-1.8.0_352-b08 | | shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/testReport/ | | Max. process+thread count | 1745 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine StoreEngine and related interfaces have evolved organically and the current state is maybe not ideal. If we take the above approach, to refactor the default compactor interface, perhaps these called out compactors and engines can be refactored to take a cleaner approach (imho). If we look at the pattern that other recent refactors have taken there might be cause to consolidate common logic into an abstract class named appropriately and have the store engines inherit from that. This could be follow up work. Not asking for this work to be performed for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine StoreEngine and related interfaces have evolved organically and the current state is maybe not ideal. If we take the above approach, to refactor the default compactor to become a multi file compactor that only compacts to a single file, these called out compactors and engines can possibly be refactored to inherit from `DefaultStoreEngine` to take a cleaner approach (imho), although that would be follow up work. It would be worth taking a look at that. Later. Of course I am not suggesting that be done for this PR. Or if we look at the pattern that other recent refactors have taken there might be cause to consolidate common logic into an abstract class named appropriately and have the store engines inherit from that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine StoreEngine and related interfaces have evolved organically and the current state is maybe not ideal. If we take the above approach, to refactor the default compactor to become a multi file compactor that only compacts to a single file, these called out compactors and engines can possibly be refactored to inherit from `DefaultStoreEngine` to take a cleaner approach (imho), although that would be follow up work. It would be worth taking a look at that. Later. Of course I am not suggesting that be done for this PR. Or if we look at the pattern that other recent refactors have taken there might be cause to consolidate common logic into an `AbstractStoreEngine` and have the store engines inherit from that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine StoreEngine and related interfaces have evolved organically and the current state is maybe not ideal. If we take the above approach, to refactor the default compactor to become a multi file compactor that only compacts to a single file, these called out compactors and engines can possibly be refactored to inherit from `DefaultStoreEngine` to take a cleaner approach (imho), although that would be follow up work. It would be worth taking a look at that. Later. Of course I am not suggesting that be done for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416119851 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEnginehttps://docs.google.com/document/d/1Ea42tEBh2X2fCq0_tXSe1BgEqBz58oswJULEbA8-MfI/edit?disco=AAABAgKl--o > HAS_LATEST_VERSION handling could be introduced into the default store engine and compatibility is assured given how you handle HFiles that lack this metadata. Also older versions that don't know about and ignore HAS_LATEST_VERSION will also function correctly because all HFiles will be examined as before. > It seems that we can incrementally upgrade or downgrade from a store engine that understands HAS_LATEST_VERSION and one that does not, unless I am missing something, which is certainly possible. Is my understanding correct? If so I am wondering if we really need a new StoreEngine implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine ... and if we take the above approach, to refactor the default compactor to become a multi file compactor that only compacts to a single file, these called out compactors and engines can possibly be refactored to inherit from `DefaultStoreEngine` to take a cleaner approach (imho), although that would be follow up work. It would be worth taking a look at that. Later. Of course I am not suggesting that be done for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine ... and if we take the above approach, to refactor the default compactor to become a multi file compactor that only compacts to a single file, these called out compactors can possibly be refactored to inherit from `DefaultStoreEngine` to take a cleaner approach (imho), although that would be follow up work. I am not suggesting that be done for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine ... and if we take the above approach, to refactor the default compactor to become a multi file compactor that only compacts to a single file, these called out compactors can be refactored to inherit from `DefaultStoreEngine`, although that would be follow up work. I am not suggesting that be done for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416126845 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine DualFileCompactor is a multi file writer as DateTierCompactor and StripeCompactor. Both DateTierStoreEngine and StripeStoreEngine do not inherit from DefaultStoreEngine ... and if we take the above approach, these can be refactored to inherit from DefaultStoreEngine, although that would be follow up work. I am not suggesting that be done for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416119851 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEnginehttps://docs.google.com/document/d/1Ea42tEBh2X2fCq0_tXSe1BgEqBz58oswJULEbA8-MfI/edit?disco=AAABAgKl--o > HAS_LATEST_VERSION handling could be introduced into the default store engine and compatibility is assured given how you handle HFiles that lack this metadata. Also older versions that don't know about and ignore HAS_LATEST_VERSION will also function correctly because all HFiles will be examined as before. > It seems that we can incrementally upgrade or downgrade from a store engine that understands HAS_LATEST_VERSION and one that does not, unless I am missing something, which is certainly possible. Is my understanding correct? If so I am wondering if we really need a new StoreEngine implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
apurtell commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1416119851 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine
[jira] [Updated] (HBASE-28206) [JDK17] JVM crashes intermittently on aarch64
[ https://issues.apache.org/jira/browse/HBASE-28206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28206: -- Assignee: Bryan Beaudreault Status: Patch Available (was: Open) PR available > [JDK17] JVM crashes intermittently on aarch64 > - > > Key: HBASE-28206 > URL: https://issues.apache.org/jira/browse/HBASE-28206 > Project: HBase > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > > The crash is related to the C1 JIT compiler. It's crashing while trying to > compile RegionServerMetricsWrapperRunnable. See > [https://github.com/adoptium/adoptium-support/issues/951] for a bit more > details (no resolution yet). > I'm going to experiment with breaking up the run() method a bit to see if it > can play nicer with the compiler. The recommendation from the adoptium folks > was to try disabling the C1 compiler, which I'd rather not do. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28206) [JDK17] JVM crashes intermittently on aarch64
[ https://issues.apache.org/jira/browse/HBASE-28206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28206: -- Labels: patch-available (was: ) > [JDK17] JVM crashes intermittently on aarch64 > - > > Key: HBASE-28206 > URL: https://issues.apache.org/jira/browse/HBASE-28206 > Project: HBase > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: patch-available > > The crash is related to the C1 JIT compiler. It's crashing while trying to > compile RegionServerMetricsWrapperRunnable. See > [https://github.com/adoptium/adoptium-support/issues/951] for a bit more > details (no resolution yet). > I'm going to experiment with breaking up the run() method a bit to see if it > can play nicer with the compiler. The recommendation from the adoptium folks > was to try disabling the C1 compiler, which I'd rather not do. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 [hbase]
bbeaudreault opened a new pull request, #5561: URL: https://github.com/apache/hbase/pull/5561 We routinely see JVM crashes originating from our RegionServerMetricsWrapperRunnable in jdk17 on aarch64. Since that method is so large, it's hard to know exactly where or what the problem is. There is an [issue submitted](https://github.com/adoptium/adoptium-support/issues/951), but no indication of whether or when a fix will come. The core dump indicates its a problem with On Stack Replacement (OSR). Reading up on that, it can kick in when long methods are suddenly deemed hot and attempt to be optimized in place. Taking a stab in the dark based on that, I saw two potential issues in RegionServerMetricsWrapperRunnable: 1. The method is exceptionally long, especially when considering the loops 2. There are over 50 volatile variables updated in the parent class This PR attempts to solve both issues: 1. Most of the volatile variables are replaced by a single volatile RegionMetricAggregate. 2. I tried to break down the giant method into organized units: a. regionserver-level calculations are left in the run() method b. the main `aggregate()` method of RegionMetricAggregate loops and collects region-level metrics c. for each region, an `aggregateStores()` method is called which loops and collects store-level metrics. This may not be perfect, but I think it's an improvement for three reasons: 1. I've been running it internally for a couple weeks and have not seen a crash (previously multiple per day) 2. It attempts to organize the code a little bit so its easier to manage. 3. It's a net-reduction in production code In order to ensure this refactor did not break any metrics, I wrote an exhaustive unit test. It validates each of the aggregated getters, and succeeds against both the old and new implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28241 Fixed the check for an ongoing Snapshot before proceeding with the merge region operation. [hbase]
Apache-HBase commented on PR #5560: URL: https://github.com/apache/hbase/pull/5560#issuecomment-1841326321 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | master passed | | +1 :green_heart: | compile | 0m 36s | master passed | | +1 :green_heart: | shadedjars | 5m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 24s | the patch passed | | +1 :green_heart: | compile | 0m 36s | the patch passed | | +1 :green_heart: | javac | 0m 36s | the patch passed | | +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 234m 58s | hbase-server in the patch failed. | | | | 256m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5560 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1c7ed8b45b1c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/testReport/ | | Max. process+thread count | 4567 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
Apache-HBase commented on PR #5559: URL: https://github.com/apache/hbase/pull/5559#issuecomment-1841305122 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 0m 43s | master passed | | +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 261m 47s | hbase-server in the patch failed. | | | | 287m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5559 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 024f938cc9d0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/testReport/ | | Max. process+thread count | 4590 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28241 Fixed the check for an ongoing Snapshot before proceeding with the merge region operation. [hbase]
Apache-HBase commented on PR #5560: URL: https://github.com/apache/hbase/pull/5560#issuecomment-1841299286 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 59s | master passed | | +1 :green_heart: | compile | 0m 43s | master passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 40s | the patch passed | | +1 :green_heart: | compile | 0m 43s | the patch passed | | +1 :green_heart: | javac | 0m 43s | the patch passed | | +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 216m 25s | hbase-server in the patch passed. | | | | 238m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5560 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 25010ab7fb2e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/testReport/ | | Max. process+thread count | 4767 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
Apache-HBase commented on PR #5559: URL: https://github.com/apache/hbase/pull/5559#issuecomment-1841264048 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 54s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 4m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 6m 24s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 235m 36s | hbase-server in the patch failed. | | | | 261m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5559 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 71cf4981e22a 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/testReport/ | | Max. process+thread count | 4606 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27795: Define RPC API for cache cleaning [hbase]
wchevreuil commented on code in PR #5492:
URL: https://github.com/apache/hbase/pull/5492#discussion_r1415985529
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -2144,4 +2146,21 @@ public Optional getBlockSize(BlockCacheKey key)
{
}
}
+
+ public Optional>
+uncacheStaleBlocks(RegionAvailabilityChecker regionAvailabilityChecker) {
+Map evictedFilesWithStaleBlocks = new HashMap<>();
Review Comment:
Copying my previous concerns shared on the other PR:
> Hum, I think we should make this method async. And we shouldn't block the
RPC call until its finished. We should dedicate a thread pool with a max size
of one.
I'm worried now this might not be a lightweight operation, if the related
RPC is synchronous (as implemented here), we might eventually timeout and
client might submit another call, eventually exhausting RPC handlers.
Simply making the RPC async isn't enough, we should also make sure we don't
have more than one background thread running this.`
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java:
##
@@ -439,6 +441,16 @@ public Optional>>
getFullyCachedFiles() {
return this.l2Cache.getFullyCachedFiles();
}
+ @Override
+ public Optional>
+uncacheStaleBlocks(RegionAvailabilityChecker regionAvailabilityChecker) {
+Map uncachedStaleBlocksMap =
+
l1Cache.uncacheStaleBlocks(regionAvailabilityChecker).orElseGet(HashMap::new);
+l2Cache.uncacheStaleBlocks(regionAvailabilityChecker).ifPresent(
+ map2 -> map2.forEach((key, value) -> uncachedStaleBlocksMap.merge(key,
value, Integer::sum)));
Review Comment:
Why merge? We should just do putAll, no?
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java:
##
@@ -245,4 +246,15 @@ default Optional getBlockSize(BlockCacheKey key) {
default Optional>> getFullyCachedFiles() {
return Optional.empty();
}
+
+ /**
+ * Clean Cache by evicting the blocks of files belonging to regions that are
no longer served by
+ * the RegionServer.
+ * @param regionAvailabilityChecker RegionAvailabilityChecker
+ * @return A map of filename and number of blocks evicted.
+ */
+ default Optional>
+uncacheStaleBlocks(RegionAvailabilityChecker regionAvailabilityChecker) {
Review Comment:
I think @Apache9 meant to use the `RegionServerServices` interface in the
method signature, not creating a new interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28241 Fixed the check for an ongoing Snapshot before proceeding with the merge region operation. [hbase]
Apache-HBase commented on PR #5560: URL: https://github.com/apache/hbase/pull/5560#issuecomment-1840874206 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 58s | master passed | | +1 :green_heart: | compile | 2m 31s | master passed | | +1 :green_heart: | checkstyle | 0m 37s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 2m 28s | the patch passed | | +1 :green_heart: | javac | 2m 28s | the patch passed | | +1 :green_heart: | checkstyle | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 33s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 32m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5560 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 9240941ad116 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5560/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
Apache-HBase commented on PR #5559: URL: https://github.com/apache/hbase/pull/5559#issuecomment-1840795450 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 3s | master passed | | +1 :green_heart: | compile | 2m 33s | master passed | | +1 :green_heart: | checkstyle | 0m 36s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 42s | the patch passed | | +1 :green_heart: | compile | 2m 31s | the patch passed | | -0 :warning: | javac | 2m 31s | hbase-server generated 1 new + 194 unchanged - 1 fixed = 195 total (was 195) | | +1 :green_heart: | checkstyle | 0m 33s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 7s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | -1 :x: | spotbugs | 1m 35s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 33m 25s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Condition.await() not in loop in org.apache.hadoop.hbase.master.assignment.RegionStateNodeLock$1.await() At RegionStateNodeLock.java:org.apache.hadoop.hbase.master.assignment.RegionStateNodeLock$1.await() At RegionStateNodeLock.java:[line 154] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5559 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 29d529ed6c76 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5559/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793278#comment-17793278
]
ruanhui commented on HBASE-28241:
-
(y)(y)(y)
> The snapshot operation encountered an NPE and failed.
> -
>
> Key: HBASE-28241
> URL: https://issues.apache.org/jira/browse/HBASE-28241
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Haiping lv
>Assignee: Haiping lv
>Priority: Major
>
> Executing a merge region before the snapshot operation is completed will
> result in an NPE error for the snapshot operation and cause it to fail
> Triggering logic
> # Execute the snapshot command.
> # Merge regions before the snapshot is completed.
> # After the merge is completed, the two parent regions will be cleaned up.
> # An NPE will be reported when the snapshot operation is performed on that
> parent region. The log is as follow:
> {code:java}
> 2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714,
> state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
> c47539f1d6032ba1a037f5279c22baa0
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793273#comment-17793273
]
Haiping lv commented on HBASE-28241:
Hi, [~frostruan] HBASE-28226 addressed the NPE issue in FlushRegionProcedure
by simply checking for null and logging the occurrence.
> The snapshot operation encountered an NPE and failed.
> -
>
> Key: HBASE-28241
> URL: https://issues.apache.org/jira/browse/HBASE-28241
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Haiping lv
>Assignee: Haiping lv
>Priority: Major
>
> Executing a merge region before the snapshot operation is completed will
> result in an NPE error for the snapshot operation and cause it to fail
> Triggering logic
> # Execute the snapshot command.
> # Merge regions before the snapshot is completed.
> # After the merge is completed, the two parent regions will be cleaned up.
> # An NPE will be reported when the snapshot operation is performed on that
> parent region. The log is as follow:
> {code:java}
> 2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714,
> state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
> c47539f1d6032ba1a037f5279c22baa0
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
Apache9 opened a new pull request, #5559: URL: https://github.com/apache/hbase/pull/5559 …on node lock immediately -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793245#comment-17793245
]
ruanhui commented on HBASE-28241:
-
Hi [~luoen] Thanks for the nice discovery.
If you don't mind, you can submit a patch and I can help reivewing.
Thanks.
> The snapshot operation encountered an NPE and failed.
> -
>
> Key: HBASE-28241
> URL: https://issues.apache.org/jira/browse/HBASE-28241
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Haiping lv
>Assignee: Haiping lv
>Priority: Major
>
> Executing a merge region before the snapshot operation is completed will
> result in an NPE error for the snapshot operation and cause it to fail
> Triggering logic
> # Execute the snapshot command.
> # Merge regions before the snapshot is completed.
> # After the merge is completed, the two parent regions will be cleaned up.
> # An NPE will be reported when the snapshot operation is performed on that
> parent region. The log is as follow:
> {code:java}
> 2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714,
> state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
> c47539f1d6032ba1a037f5279c22baa0
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793243#comment-17793243
]
ruanhui commented on HBASE-28241:
-
I wrote an UT and reproduced the problem, here is the UT code:
{code:java}
@Test
public void testMergingRegionWhileTakingSnapshot() throws Exception {
// take snapshot
ProcedureExecutor procExec =
master.getMasterProcedureExecutor();
// long snapshotProcId = procExec.submitProcedure(
// new DelaySnapshotProcedure(procExec.getEnvironment(), snapshot));
long snapshotProcId = master.getSnapshotManager().takeSnapshot(snapshotProto,
HConstants.NO_NONCE, HConstants.NO_NONCE);
// merge region
List regions =
master.getAssignmentManager().getTableRegions(TABLE_NAME, true)
.stream().sorted(RegionInfo.COMPARATOR).collect(Collectors.toList());
MergeTableRegionsProcedure mergeProc = new
MergeTableRegionsProcedure(procExec.getEnvironment(),
new RegionInfo[] { regions.get(0), regions.get(1) }, false);
long mergeProcId = procExec.submitProcedure(mergeProc);
// wait procedure finish
ProcedureTestingUtility.waitProcedure(master.getMasterProcedureExecutor(),
mergeProcId);
ProcedureTestingUtility.waitProcedure(master.getMasterProcedureExecutor(),
snapshotProcId);
}
// to make the problem easier to reproduce, this class will wait 10 seconds
before scheduling
// snapshot region procedure to make sure that we have merge the region and
remove the parent regionNode
public static final class DelaySnapshotProcedure extends SnapshotProcedure {
public DelaySnapshotProcedure() {
}
public DelaySnapshotProcedure(final MasterProcedureEnv env,
final SnapshotProtos.SnapshotDescription snapshot) {
super(env, snapshot);
}
@Override
protected Flow executeFromState(MasterProcedureEnv env,
MasterProcedureProtos.SnapshotState state)
throws ProcedureSuspendedException, ProcedureYieldException,
InterruptedException {
Flow flow = super.executeFromState(env, state);
if (state == SNAPSHOT_SNAPSHOT_ONLINE_REGIONS) {
TimeUnit.SECONDS.sleep(10);
}
return flow;
}
}{code}
and got the following exception
{code:java}
2023-12-05T19:29:08,270 ERROR [PEWorker-3 {}]
procedure2.ProcedureExecutor(1784): CODE-BUG: Uncaught runtime exception:
pid=33, ppid=25, state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
18f01fba77da8a1e1ad836abf49a62a2
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
~[classes/:?]
at
org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:941)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1765)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1443)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:77)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:2091)
~[classes/:?]
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2118)
~[classes/:?] {code}
After correct the method, I got the expected failed merge procedure.
{code:java}
2023-12-05T19:44:58,207 ERROR [PEWorker-4 {}]
assignment.MergeTableRegionsProcedure(249): Error trying to merge
[6421d004d9e61c8c30e49b3af4b923de, 7ef0155567c0060bad76b0c82990f010] in
SPTestTable (in state=MERGE_TABLE_REGIONS_PREPARE)
org.apache.hadoop.hbase.exceptions.MergeRegionException: Skip merging regions
[6421d004d9e61c8c30e49b3af4b923de, 7ef0155567c0060bad76b0c82990f010], because
we are snapshotting SPTestTable
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.prepareMergeRegion(MergeTableRegionsProcedure.java:447)
~[classes/:?]
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:193)
~[classes/:?]
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:78)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:941)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1765)
~[classes/:?]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProc
Re: [PR] HBASE-28238 rpcservice should perform the updateConfiguration operation to priority ADMIN_QOS [hbase]
Apache-HBase commented on PR #5558: URL: https://github.com/apache/hbase/pull/5558#issuecomment-1840531038 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 54s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 4m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 244m 20s | hbase-server in the patch passed. | | | | 266m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5558 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 27db36d658e2 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/1/testReport/ | | Max. process+thread count | 4849 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28238 rpcservice should perform the updateConfiguration operation to priority ADMIN_QOS [hbase]
Apache-HBase commented on PR #5558: URL: https://github.com/apache/hbase/pull/5558#issuecomment-1840515877 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 5m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 29s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 234m 35s | hbase-server in the patch failed. | | | | 258m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5558 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e6e67a249c12 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 25e9228e2c | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/1/testReport/ | | Max. process+thread count | 4346 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5558/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793127#comment-17793127
]
ruanhui edited comment on HBASE-28241 at 12/5/23 9:35 AM:
--
In the SnapshotManager, we have two methods
{code:java}
public boolean isTakingSnapshot(final TableName tableName) {
return isTakingSnapshot(tableName, /* checkProcedure= */false);
}
public boolean isTableTakingAnySnapshot(final TableName tableName) {
return isTakingSnapshot(tableName, /* checkProcedure= */ true);
} {code}
Currently the SplitTableRegionProcedure and MergeTableRegionProcedure calls the
first method because we used to think that spliting/merging region procedure
and snapshoting region procedure were not mutually exclusive. But actually it
do causes some problems. We should call the second one ? Let me design an UT to
verify it.
was (Author: frostruan):
In the SnapshotManager, we have two methods
{code:java}
public boolean isTakingSnapshot(final TableName tableName) {
return isTakingSnapshot(tableName, /* checkProcedure= */false);
}
public boolean isTableTakingAnySnapshot(final TableName tableName) {
return isTakingSnapshot(tableName, /* checkProcedure= */ true);
} {code}
Currently the SplitTableRegionProcedure and MergeTableRegionProcedure calls the
first method because we used to think that spliting/merging region procedure
and snapshoting region procedure were not mutually exclusive. But actually it
do causes some problems. We should calls the second one ? Let me design an UT
to verify it.
> The snapshot operation encountered an NPE and failed.
> -
>
> Key: HBASE-28241
> URL: https://issues.apache.org/jira/browse/HBASE-28241
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Haiping lv
>Assignee: Haiping lv
>Priority: Major
>
> Executing a merge region before the snapshot operation is completed will
> result in an NPE error for the snapshot operation and cause it to fail
> Triggering logic
> # Execute the snapshot command.
> # Merge regions before the snapshot is completed.
> # After the merge is completed, the two parent regions will be cleaned up.
> # An NPE will be reported when the snapshot operation is performed on that
> parent region. The log is as follow:
> {code:java}
> 2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714,
> state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
> c47539f1d6032ba1a037f5279c22baa0
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793200#comment-17793200
]
ruanhui commented on HBASE-28241:
-
Agree.
And the FlushRegionProcedure have the same problem. If region list changed, the
regionNode we get is null. But it's not a big deal for flush operation, I think
we can just log this event and ignore it ?
> The snapshot operation encountered an NPE and failed.
> -
>
> Key: HBASE-28241
> URL: https://issues.apache.org/jira/browse/HBASE-28241
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Haiping lv
>Assignee: Haiping lv
>Priority: Major
>
> Executing a merge region before the snapshot operation is completed will
> result in an NPE error for the snapshot operation and cause it to fail
> Triggering logic
> # Execute the snapshot command.
> # Merge regions before the snapshot is completed.
> # After the merge is completed, the two parent regions will be cleaned up.
> # An NPE will be reported when the snapshot operation is performed on that
> parent region. The log is as follow:
> {code:java}
> 2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714,
> state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
> c47539f1d6032ba1a037f5279c22baa0
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28241) The snapshot operation encountered an NPE and failed.
[
https://issues.apache.org/jira/browse/HBASE-28241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793196#comment-17793196
]
Duo Zhang commented on HBASE-28241:
---
I think we should call the second one. No matter we use zk coordinated
procedure or proc-v2 based procedure, the snapshot operation needs to make sure
that the region list does not change.
> The snapshot operation encountered an NPE and failed.
> -
>
> Key: HBASE-28241
> URL: https://issues.apache.org/jira/browse/HBASE-28241
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Haiping lv
>Assignee: Haiping lv
>Priority: Major
>
> Executing a merge region before the snapshot operation is completed will
> result in an NPE error for the snapshot operation and cause it to fail
> Triggering logic
> # Execute the snapshot command.
> # Merge regions before the snapshot is completed.
> # After the merge is completed, the two parent regions will be cleaned up.
> # An NPE will be reported when the snapshot operation is performed on that
> parent region. The log is as follow:
> {code:java}
> 2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714,
> state=RUNNABLE, hasLock=true; SnapshotRegionProcedure
> c47539f1d6032ba1a037f5279c22baa0
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58)
> ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988)
> ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
