Re: [PR] HBASE-28294 Support to skip Kerberos authentication for metric endpoints [hbase]
Apache-HBase commented on PR #5606: URL: https://github.com/apache/hbase/pull/5606#issuecomment-1879576307 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 10s | master passed | | +1 :green_heart: | compile | 5m 5s | master passed | | +1 :green_heart: | checkstyle | 1m 11s | master passed | | +0 :ok: | refguide | 2m 42s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 8m 12s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 5m 5s | the patch passed | | +1 :green_heart: | javac | 5m 5s | the patch passed | | -0 :warning: | checkstyle | 1m 7s | root: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 2m 7s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 10m 43s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | -1 :x: | spotless | 0m 18s | patch has 26 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 8m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 60m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5606/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5606 | | Optional Tests | dupname asflicense javac refguide spotless xml spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 360304f0d262 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 119885415c | | Default Java | Eclipse Adoptium-11.0.17+8 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5606/1/yetus-general-check/output/branch-site/book.html | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5606/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5606/1/yetus-general-check/output/patch-site/book.html | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5606/1/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 176 (vs. ulimit of 3) | | modules | C: hbase-common hbase-http . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5606/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[
https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803763#comment-17803763
]
Hudson commented on HBASE-28259:
Results for branch branch-2
[build #955 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/955/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/955/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/955/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/955/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/955/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
> --
>
> Key: HBASE-28259
> URL: https://issues.apache.org/jira/browse/HBASE-28259
> Project: HBase
> Issue Type: Bug
> Components: java
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 4.0.0-alpha-1, 2.5.7, 2.7.0
>Reporter: Moran
>Assignee: Moran
>Priority: Trivial
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> hbase shell
> 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess
> control requires open access to the JDK IO subsystem
> Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens
> java.base/java.io=ALL-UNNAMED' to enable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28294) Support to skip Kerberos authentication for metric endpoints
YUBI LEE created HBASE-28294: Summary: Support to skip Kerberos authentication for metric endpoints Key: HBASE-28294 URL: https://issues.apache.org/jira/browse/HBASE-28294 Project: HBase Issue Type: New Feature Components: metrics, UI Affects Versions: 2.5.5 Reporter: YUBI LEE Make HBase support to skip Kerberos authentication for metric endpoints. (e.g. /jvm, /prometheus, /metrics) Since HBase uses KerberoAuthenticationHandler.java, whitelist configuration can be used like [HADOOP-16527|https://issues.apache.org/jira/browse/HADOOP-16527]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[
https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803744#comment-17803744
]
Hudson commented on HBASE-28259:
Results for branch branch-2.4
[build #675 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/675/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/675/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/675/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/675/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/675/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
> --
>
> Key: HBASE-28259
> URL: https://issues.apache.org/jira/browse/HBASE-28259
> Project: HBase
> Issue Type: Bug
> Components: java
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 4.0.0-alpha-1, 2.5.7, 2.7.0
>Reporter: Moran
>Assignee: Moran
>Priority: Trivial
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> hbase shell
> 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess
> control requires open access to the JDK IO subsystem
> Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens
> java.base/java.io=ALL-UNNAMED' to enable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[
https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803742#comment-17803742
]
Hudson commented on HBASE-28259:
Results for branch branch-2.6
[build #22 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/22/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/22/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/22/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/22/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/22/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
> --
>
> Key: HBASE-28259
> URL: https://issues.apache.org/jira/browse/HBASE-28259
> Project: HBase
> Issue Type: Bug
> Components: java
>Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 4.0.0-alpha-1, 2.5.7, 2.7.0
>Reporter: Moran
>Assignee: Moran
>Priority: Trivial
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> hbase shell
> 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess
> control requires open access to the JDK IO subsystem
> Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens
> java.base/java.io=ALL-UNNAMED' to enable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
virajjasani commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1443550627
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileWriter.java:
##
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static
org.apache.hadoop.hbase.regionserver.HStoreFile.HAS_LIVE_VERSIONS_KEY;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Separates the provided cells into two files, one file for the live cells
and the other for the
+ * rest of the cells (historical cells). The live cells includes the live put
cells, delete all and
+ * version delete markers that are not masked by other delete all markers.
+ */
+@InterfaceAudience.Private
+public class DualFileWriter extends AbstractMultiFileWriter {
+
+ private final CellComparator comparator;
+ private StoreFileWriter liveVersionWriter;
+ private StoreFileWriter historicalVersionWriter;
+
+ private final List writers;
+ // The last cell of the current row
+ private Cell lastCell;
+ // The first (latest) delete family marker of the current row
+ private Cell deleteFamily;
+ // The list of delete family version markers of the current row
+ private List deleteFamilyVersionList = new ArrayList<>();
+ // The first (latest) delete column marker of the current column
+ private Cell deleteColumn;
+ // The list of delete column version markers of the current column
+ private List deleteColumnVersionList = new ArrayList<>();
+ // The live put cell count for the current column
+ private int livePutCellCount;
+ private final boolean dualWriterEnabled;
+ private final int maxVersions;
+ private final boolean newVersionBehavior;
+
+ public DualFileWriter(CellComparator comparator, int maxVersions, boolean
dualWriterEnabled,
+boolean newVersionBehavior) {
+this.comparator = comparator;
+this.maxVersions = maxVersions;
+this.dualWriterEnabled = dualWriterEnabled;
+this.newVersionBehavior = newVersionBehavior;
+writers = new ArrayList<>(2);
+initRowState();
+ }
+
+ private void initRowState() {
+deleteFamily = null;
+deleteFamilyVersionList.clear();
+lastCell = null;
+ }
+
+ private void initColumnState() {
+livePutCellCount = 0;
+deleteColumn = null;
+deleteColumnVersionList.clear();
+
+ }
+
+ private void addLiveVersion(Cell cell) throws IOException {
+if (liveVersionWriter == null) {
+ liveVersionWriter = writerFactory.createWriter();
+ writers.add(liveVersionWriter);
+}
+liveVersionWriter.append(cell);
+ }
+
+ private void addHistoricalVersion(Cell cell) throws IOException {
+if (historicalVersionWriter == null) {
+ historicalVersionWriter = writerFactory.createWriter();
+ writers.add(historicalVersionWriter);
+}
+historicalVersionWriter.append(cell);
+ }
+
+ private boolean isDeletedByDeleteFamily(Cell cell) {
+return deleteFamily != null && (deleteFamily.getTimestamp() >
cell.getTimestamp()
+ || (deleteFamily.getTimestamp() == cell.getTimestamp()
+&& (!newVersionBehavior || cell.getSequenceId() <
deleteFamily.getSequenceId(;
+ }
+
+ private boolean isDeletedByDeleteFamilyVersion(Cell cell) {
+for (Cell deleteFamilyVersion : deleteFamilyVersionList) {
+ if (
+deleteFamilyVersion.getTimestamp() == cell.getTimestamp()
+ && (!newVersionBehavior || cell.getSequenceId() <
deleteFamilyVersion.getSequenceId())
+ ) return true;
+}
+return false;
+ }
+
+ private boolean isDeletedByDeleteColumn(Cell cell) {
+return deleteColumn != null && (deleteColumn.getTimestamp() >
cell.getTimestamp()
+ || (deleteColumn.getTimestamp() == cell.getTimestamp()
+&& (!newVersionBehavior || cell.getSequenceId() <
deleteColumn.getSequenceId(;
+ }
[jira] [Commented] (HBASE-28293) Add metric for GetClusterStatus request count.
[ https://issues.apache.org/jira/browse/HBASE-28293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803725#comment-17803725 ] Bryan Beaudreault commented on HBASE-28293: --- Yep that sounds good > Add metric for GetClusterStatus request count. > -- > > Key: HBASE-28293 > URL: https://issues.apache.org/jira/browse/HBASE-28293 > Project: HBase > Issue Type: Bug >Reporter: Rushabh Shah >Priority: Major > > We have been bitten multiple times by GetClusterStatus request overwhelming > HMaster's memory usage. It would be good to add a metric for the total > GetClusterStatus requests count. > In almost all of our production incidents involving GetClusterStatus request, > HMaster were running out of memory with many clients call this RPC in > parallel and the response size is very big. > In hbase2 we have > [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] > which can reduce the size of the response. > It would be nice to add another metric to indicate if the response size of > GetClusterStatus is greater than some threshold (like 5MB) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HBASE-28293) Add metric for GetClusterStatus request count.
[ https://issues.apache.org/jira/browse/HBASE-28293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803724#comment-17803724 ] Viraj Jasani edited comment on HBASE-28293 at 1/5/24 11:18 PM: --- +1, maybe for this Jira we can focus on getClusterStatus as it is heavy one, and in follow-up jiras, we can extend this for other RPCs served by master. was (Author: vjasani): +1 > Add metric for GetClusterStatus request count. > -- > > Key: HBASE-28293 > URL: https://issues.apache.org/jira/browse/HBASE-28293 > Project: HBase > Issue Type: Bug >Reporter: Rushabh Shah >Priority: Major > > We have been bitten multiple times by GetClusterStatus request overwhelming > HMaster's memory usage. It would be good to add a metric for the total > GetClusterStatus requests count. > In almost all of our production incidents involving GetClusterStatus request, > HMaster were running out of memory with many clients call this RPC in > parallel and the response size is very big. > In hbase2 we have > [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] > which can reduce the size of the response. > It would be nice to add another metric to indicate if the response size of > GetClusterStatus is greater than some threshold (like 5MB) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28293) Add metric for GetClusterStatus request count.
[ https://issues.apache.org/jira/browse/HBASE-28293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803724#comment-17803724 ] Viraj Jasani commented on HBASE-28293: -- +1 > Add metric for GetClusterStatus request count. > -- > > Key: HBASE-28293 > URL: https://issues.apache.org/jira/browse/HBASE-28293 > Project: HBase > Issue Type: Bug >Reporter: Rushabh Shah >Priority: Major > > We have been bitten multiple times by GetClusterStatus request overwhelming > HMaster's memory usage. It would be good to add a metric for the total > GetClusterStatus requests count. > In almost all of our production incidents involving GetClusterStatus request, > HMaster were running out of memory with many clients call this RPC in > parallel and the response size is very big. > In hbase2 we have > [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] > which can reduce the size of the response. > It would be nice to add another metric to indicate if the response size of > GetClusterStatus is greater than some threshold (like 5MB) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28293) Add metric for GetClusterStatus request count.
[ https://issues.apache.org/jira/browse/HBASE-28293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803722#comment-17803722 ] Bryan Beaudreault commented on HBASE-28293: --- Maybe response time histogram as well? Would be nice to have better coverage of these 3 metric types for all the rpc types overall (similar to regionserver) > Add metric for GetClusterStatus request count. > -- > > Key: HBASE-28293 > URL: https://issues.apache.org/jira/browse/HBASE-28293 > Project: HBase > Issue Type: Bug >Reporter: Rushabh Shah >Priority: Major > > We have been bitten multiple times by GetClusterStatus request overwhelming > HMaster's memory usage. It would be good to add a metric for the total > GetClusterStatus requests count. > In almost all of our production incidents involving GetClusterStatus request, > HMaster were running out of memory with many clients call this RPC in > parallel and the response size is very big. > In hbase2 we have > [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] > which can reduce the size of the response. > It would be nice to add another metric to indicate if the response size of > GetClusterStatus is greater than some threshold (like 5MB) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28293) Add metric for GetClusterStatus request count.
[ https://issues.apache.org/jira/browse/HBASE-28293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803718#comment-17803718 ] Viraj Jasani commented on HBASE-28293: -- We can have two metrics: response size and request count > Add metric for GetClusterStatus request count. > -- > > Key: HBASE-28293 > URL: https://issues.apache.org/jira/browse/HBASE-28293 > Project: HBase > Issue Type: Bug >Reporter: Rushabh Shah >Priority: Major > > We have been bitten multiple times by GetClusterStatus request overwhelming > HMaster's memory usage. It would be good to add a metric for the total > GetClusterStatus requests count. > In almost all of our production incidents involving GetClusterStatus request, > HMaster were running out of memory with many clients call this RPC in > parallel and the response size is very big. > In hbase2 we have > [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] > which can reduce the size of the response. > It would be nice to add another metric to indicate if the response size of > GetClusterStatus is greater than some threshold (like 5MB) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28293) Add metric for GetClusterStatus request count.
Rushabh Shah created HBASE-28293: Summary: Add metric for GetClusterStatus request count. Key: HBASE-28293 URL: https://issues.apache.org/jira/browse/HBASE-28293 Project: HBase Issue Type: Bug Reporter: Rushabh Shah We have been bitten multiple times by GetClusterStatus request overwhelming HMaster's memory usage. It would be good to add a metric for the total GetClusterStatus requests count. In almost all of our production incidents involving GetClusterStatus request, HMaster were running out of memory with many clients call this RPC in parallel and the response size is very big. In hbase2 we have [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] which can reduce the size of the response. It would be nice to add another metric to indicate if the response size of GetClusterStatus is greater than some threshold (like 5MB) -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28271 Infinite waiting on lock acquisition by snapshot can result in unresponsive master [hbase]
virajjasani commented on code in PR #5603:
URL: https://github.com/apache/hbase/pull/5603#discussion_r1443273449
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java:
##
@@ -129,6 +132,8 @@ public TakeSnapshotHandler(SnapshotDescription snapshot,
final MasterServices ma
"Taking " + snapshot.getType() + " snapshot on table: " + snapshotTable,
false, true);
this.snapshotManifest =
SnapshotManifest.create(conf, rootFs, workingDir, snapshot, monitor,
status);
+this.lockAcquireTimeoutMs =
+ conf.getLong(HBASE_SNAPSHOT_MASTER_LOCK_ACQUIRE_TIMEOUT, 5 * 60 * 1000L);
Review Comment:
Sounds good, will make this change.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java:
##
@@ -147,12 +152,16 @@ private TableDescriptor loadTableDescriptor() throws
IOException {
public TakeSnapshotHandler prepare() throws Exception {
super.prepare();
// after this, you should ensure to release this lock in case of exceptions
-this.tableLock.acquire();
-try {
- this.htd = loadTableDescriptor(); // check that .tableinfo is present
-} catch (Exception e) {
- this.tableLock.release();
- throw e;
+if (this.tableLock.tryAcquire(this.lockAcquireTimeoutMs)) {
+ try {
+this.htd = loadTableDescriptor(); // check that .tableinfo is present
+ } catch (Exception e) {
+this.tableLock.release();
+throw e;
+ }
+} else {
+ LOG.error("Master lock could not be acquired in {} ms",
lockAcquireTimeoutMs);
+ throw new IOException("Master lock could not be acquired");
Review Comment:
Actually, regardless of the Exception type here, it does eventually throw
`SnapshotCreationException`:
```
try {
handler.prepare();
this.executorService.submit(handler);
this.snapshotHandlers.put(TableName.valueOf(snapshot.getTable()),
handler);
} catch (Exception e) {
// cleanup the working directory by trying to delete it from the fs.
Path workingDir =
SnapshotDescriptionUtils.getWorkingSnapshotDir(snapshot, rootDir,
master.getConfiguration());
FileSystem workingDirFs =
workingDir.getFileSystem(master.getConfiguration());
try {
if (!workingDirFs.delete(workingDir, true)) {
LOG.error("Couldn't delete working directory (" + workingDir + "
for snapshot:"
+ ClientSnapshotDescriptionUtils.toString(snapshot));
}
} catch (IOException e1) {
LOG.error("Couldn't delete working directory (" + workingDir + " for
snapshot:"
+ ClientSnapshotDescriptionUtils.toString(snapshot));
}
// fail the snapshot
throw new SnapshotCreationException("Could not build snapshot
handler", e,
ProtobufUtil.createSnapshotDesc(snapshot));
}
```
but sure i can still make it throw DoNotRetryIOException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28271 Infinite waiting on lock acquisition by snapshot can result in unresponsive master [hbase]
virajjasani commented on code in PR #5603:
URL: https://github.com/apache/hbase/pull/5603#discussion_r1443273668
##
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSnapshotProcedureWithLockTimeout.java:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.procedure;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.SnapshotDescription;
+import org.apache.hadoop.hbase.client.SnapshotType;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler;
+import org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility;
+import org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils;
+import org.apache.hadoop.hbase.testclassification.MasterTests;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.RegionSplitter;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos;
+
+@Category({ MasterTests.class, MediumTests.class })
+public class TestSnapshotProcedureWithLockTimeout {
Review Comment:
The purpose is to test with master lock timeout value small enough that we
can ensure snapshot will not be created. Let me add some comments to specify
this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-28279) Bump up jetty-server, jetty-http to 9.4.53.v20231009
[ https://issues.apache.org/jira/browse/HBASE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla resolved HBASE-28279. - Resolution: Fixed Committed to master. Thanks for review [~bbeaudreault], [~nihaljain.cs] > Bump up jetty-server, jetty-http to 9.4.53.v20231009 > > > Key: HBASE-28279 > URL: https://issues.apache.org/jira/browse/HBASE-28279 > Project: HBase > Issue Type: Bug > Components: thirdparty >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla >Priority: Major > Fix For: thirdparty-4.1.6 > > > Bump up jetty-server to 9.4.53.v20231009 to avoid CVE CVE-2023-36478 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28279 [hbase-thirdparty] Bump up jetty-server, jetty-http to 9.4.53.v20231009 [hbase-thirdparty]
chrajeshbabu merged PR #107: URL: https://github.com/apache/hbase-thirdparty/pull/107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28279 Bump up jetty-server, jetty-http to 9.4.53.v20231009 [hbase-thirdparty]
Apache-HBase commented on PR #107: URL: https://github.com/apache/hbase-thirdparty/pull/107#issuecomment-1879078196 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 53s | master passed | | +1 :green_heart: | compile | 0m 12s | master passed | | +1 :green_heart: | javadoc | 0m 6s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 34s | the patch passed | | +1 :green_heart: | compile | 0m 11s | the patch passed | | +1 :green_heart: | javac | 0m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 0s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 32s | root in the patch passed. | | +1 :green_heart: | asflicense | 0m 6s | The patch does not generate ASF License warnings. | | | | 3m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/5/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/107 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux d4a7a8897c0d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 2462b19 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/5/testReport/ | | Max. process+thread count | 403 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/5/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28279 Bump up jetty-server, jetty-http to 9.4.53.v20231009 [hbase-thirdparty]
chrajeshbabu commented on PR #107: URL: https://github.com/apache/hbase-thirdparty/pull/107#issuecomment-1879057570 Build passing locally. Some how the pipelines not running. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1879034340 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 4m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 4m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 229m 25s | hbase-server in the patch passed. | | | | 252m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 18cedc568ea3 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 119885415c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/testReport/ | | Max. process+thread count | 5143 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1879022218 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | master passed | | +1 :green_heart: | compile | 0m 35s | master passed | | +1 :green_heart: | shadedjars | 5m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 19s | the patch passed | | +1 :green_heart: | compile | 0m 36s | the patch passed | | +1 :green_heart: | javac | 0m 36s | the patch passed | | +1 :green_heart: | shadedjars | 5m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 220m 35s | hbase-server in the patch passed. | | | | 242m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d4891d8bbb51 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 119885415c | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/testReport/ | | Max. process+thread count | 4812 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-1879017714 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 21s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | shadedjars | 4m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | | -0 :warning: | patch | 5m 41s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 4m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 230m 18s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 39s | hbase-mapreduce in the patch passed. | | | | 268m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 44af8257e7f7 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 119885415c | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/testReport/ | | Max. process+thread count | 5414 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-1879002974 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 7s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | | -0 :warning: | patch | 6m 19s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 5m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 218m 44s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 6s | hbase-mapreduce in the patch passed. | | | | 258m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 050ccab316ed 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 119885415c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/testReport/ | | Max. process+thread count | 4684 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28279 Bump up jetty-server, jetty-http to 9.4.53.v20231009 [hbase-thirdparty]
Apache-HBase commented on PR #107: URL: https://github.com/apache/hbase-thirdparty/pull/107#issuecomment-1878975868 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 0m 17s | root in master failed. | | -1 :x: | compile | 0m 18s | root in master failed. | | -1 :x: | javadoc | 0m 18s | root in master failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 17s | root in the patch failed. | | -1 :x: | compile | 0m 18s | root in the patch failed. | | -1 :x: | javac | 0m 18s | root in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | -1 :x: | javadoc | 0m 17s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 17s | root in the patch failed. | | +0 :ok: | asflicense | 0m 17s | ASF License check generated no output? | | | | 3m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/107 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 75bccc067c02 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 2462b19 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/branch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/branch-compile-root.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/branch-javadoc-root.txt | | mvninstall | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/patch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/patch-compile-root.txt | | javac | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/patch-compile-root.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/patch-javadoc-root.txt | | unit | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/artifact/yetus-precommit-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/testReport/ | | Max. process+thread count | 27 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/4/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28279 Bump up jetty-server, jetty-http to 9.4.53.v20231009 [hbase-thirdparty]
Apache-HBase commented on PR #107: URL: https://github.com/apache/hbase-thirdparty/pull/107#issuecomment-1878940842 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 0m 17s | root in master failed. | | -1 :x: | compile | 0m 17s | root in master failed. | | -1 :x: | javadoc | 0m 17s | root in master failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 18s | root in the patch failed. | | -1 :x: | compile | 0m 17s | root in the patch failed. | | -1 :x: | javac | 0m 17s | root in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | -1 :x: | javadoc | 0m 18s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 17s | root in the patch failed. | | +0 :ok: | asflicense | 0m 18s | ASF License check generated no output? | | | | 3m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/107 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 6af621ff92c6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 2462b19 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/branch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/branch-compile-root.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/branch-javadoc-root.txt | | mvninstall | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/patch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/patch-compile-root.txt | | javac | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/patch-compile-root.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/patch-javadoc-root.txt | | unit | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/artifact/yetus-precommit-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/testReport/ | | Max. process+thread count | 9 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/2/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28279 Bump up jetty-server, jetty-http to 9.4.53.v20231009 [hbase-thirdparty]
Apache-HBase commented on PR #107: URL: https://github.com/apache/hbase-thirdparty/pull/107#issuecomment-1878919900 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 56s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 0m 17s | root in master failed. | | -1 :x: | compile | 0m 17s | root in master failed. | | -1 :x: | javadoc | 0m 17s | root in master failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 18s | root in the patch failed. | | -1 :x: | compile | 0m 17s | root in the patch failed. | | -1 :x: | javac | 0m 17s | root in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | -1 :x: | javadoc | 0m 17s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 18s | root in the patch failed. | | +0 :ok: | asflicense | 0m 17s | ASF License check generated no output? | | | | 4m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/107 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 0fc8172202f7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 2462b19 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/branch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/branch-compile-root.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/branch-javadoc-root.txt | | mvninstall | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/patch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/patch-compile-root.txt | | javac | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/patch-compile-root.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/patch-javadoc-root.txt | | unit | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/artifact/yetus-precommit-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/testReport/ | | Max. process+thread count | 11 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-Thirdparty-PreCommit/job/PR-107/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28279) Bump up jetty-server, jetty-http to 9.4.53.v20231009
[ https://issues.apache.org/jira/browse/HBASE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla updated HBASE-28279: Fix Version/s: thirdparty-4.1.6 > Bump up jetty-server, jetty-http to 9.4.53.v20231009 > > > Key: HBASE-28279 > URL: https://issues.apache.org/jira/browse/HBASE-28279 > Project: HBase > Issue Type: Bug > Components: thirdparty >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla >Priority: Major > Fix For: thirdparty-4.1.6 > > > Bump up jetty-server to 9.4.53.v20231009 to avoid CVE CVE-2023-36478 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-28279) Bump up jetty-server, jetty-http to 9.4.53.v20231009
[ https://issues.apache.org/jira/browse/HBASE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-28279 started by Rajeshbabu Chintaguntla. --- > Bump up jetty-server, jetty-http to 9.4.53.v20231009 > > > Key: HBASE-28279 > URL: https://issues.apache.org/jira/browse/HBASE-28279 > Project: HBase > Issue Type: Bug > Components: thirdparty >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla >Priority: Major > > Bump up jetty-server to 9.4.53.v20231009 to avoid CVE CVE-2023-36478 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
charlesconnell commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1878813606 If there is interest in backporting this to 2.5, I can make that branch once this is approved. It's not a super straightforward backport. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1878708672 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 54s | master passed | | +1 :green_heart: | compile | 2m 28s | master passed | | +1 :green_heart: | checkstyle | 0m 34s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 31s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 2m 27s | the patch passed | | +1 :green_heart: | javac | 2m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 35s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 19s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 31m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 634ef056d855 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 119885415c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 77 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28291) [hbase-thirdparty] Upgrade Netty to 4.1.104.Final
[ https://issues.apache.org/jira/browse/HBASE-28291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28291: -- Fix Version/s: thirdparty-4.1.6 Resolution: Fixed Status: Resolved (was: Patch Available) > [hbase-thirdparty] Upgrade Netty to 4.1.104.Final > - > > Key: HBASE-28291 > URL: https://issues.apache.org/jira/browse/HBASE-28291 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: patch-available, third-party > Fix For: thirdparty-4.1.6 > > > There is a CVE: > [https://github.com/netty/netty/security/advisories/GHSA-xpw8-rcwv-8f8p.] It > does not affect us, but we can clear it anyway. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28291 [hbase-thirdparty] Upgrade Netty to 4.1.104.Final [hbase-thirdparty]
bbeaudreault merged PR #106: URL: https://github.com/apache/hbase-thirdparty/pull/106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28287 Fix the inaccurate expired time of MOB HFiles [hbase]
Apache9 commented on code in PR #5599:
URL: https://github.com/apache/hbase/pull/5599#discussion_r1442890449
##
hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestExpiredMobFileCleaner.java:
##
@@ -140,7 +140,8 @@ public void testCleaner() throws Exception {
assertEquals("Before cleanup without delay 1", 1, firstFiles.length);
String firstFile = firstFiles[0].getPath().getName();
-ts = EnvironmentEdgeManager.currentTime() - 1 * secondsOfDay() * 1000; //
1 day before
+ts = (long) (EnvironmentEdgeManager.currentTime() - 1.5 * secondsOfDay() *
1000); // 1.5 day
Review Comment:
Please fix this minor style issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28287 Fix the inaccurate expired time of MOB HFiles [hbase]
Apache9 commented on PR #5599: URL: https://github.com/apache/hbase/pull/5599#issuecomment-1878666181 OK. Got it. Thanks for the explaination. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-1878666005 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 3s | master passed | | +1 :green_heart: | compile | 2m 54s | master passed | | +1 :green_heart: | checkstyle | 0m 43s | master passed | | +1 :green_heart: | spotless | 0m 39s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 52s | master passed | | -0 :warning: | patch | 0m 37s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 3m 33s | the patch passed | | +1 :green_heart: | javac | 3m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 2 new + 42 unchanged - 0 fixed = 44 total (was 42) | | -0 :warning: | checkstyle | 0m 12s | hbase-mapreduce: The patch generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 7s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 39s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 37m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 1a0c5b583486 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 119885415c | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/artifact/yetus-general-check/output/diff-checkstyle-hbase-mapreduce.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/15/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
charlesconnell commented on code in PR #5593:
URL: https://github.com/apache/hbase/pull/5593#discussion_r1442866122
##
hbase-server/src/main/java/org/apache/hadoop/hbase/HBaseServerBase.java:
##
@@ -617,15 +616,45 @@ public ConfigurationManager getConfigurationManager() {
public void updateConfiguration() {
LOG.info("Reloading the configuration from disk.");
// Reload the configuration from disk.
+preUpdateConfiguration();
conf.reloadConfiguration();
configurationManager.notifyAllObservers(conf);
+postUpdateConfiguration();
+ }
+
+ private void preUpdateConfiguration() {
+try {
+ CoprocessorHost coprocessorHost = getCoprocessorHost();
+ if (coprocessorHost instanceof RegionServerCoprocessorHost) {
+((RegionServerCoprocessorHost)
coprocessorHost).preUpdateConfiguration(conf);
+ } else if (coprocessorHost instanceof MasterCoprocessorHost) {
+((MasterCoprocessorHost) coprocessorHost).preUpdateConfiguration(conf);
+ }
+} catch (IOException e) {
+ LOG.error("Error while calling coprocessor preUpdateConfiguration()", e);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
charlesconnell commented on code in PR #5593:
URL: https://github.com/apache/hbase/pull/5593#discussion_r1442863315
##
hbase-server/src/main/java/org/apache/hadoop/hbase/HBaseServerBase.java:
##
@@ -614,11 +616,41 @@ public ConfigurationManager getConfigurationManager() {
/**
* Reload the configuration from disk.
*/
- public void updateConfiguration() {
+ public void updateConfiguration() throws IOException {
Review Comment:
I gave this a try and it seems to work just fine without user info. Note
that if this method is called by the signal handler, the AccessChecker is not
used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28287 Fix the inaccurate expired time of MOB HFiles [hbase]
liuwenjing17 commented on PR #5599:
URL: https://github.com/apache/hbase/pull/5599#issuecomment-1878527381
> > > I still do not fully understand the problem here...
> > > If we do not set millis to zero, it will only affect the life time of
a MOB file for less than 1 second, how could it make the MOB file expire 2
hours earlier?
> >
> >
> > Because in org.apache.hadoop.hbase.mob.MobUtils, the creation time of
mob files is obtained by parsing their names from fileName using the statement
(Date fileDate = parseDate(MobFileName.getDateFromName(fileName));). For
instance, data created on 20240105, their timestamps will be parsed as
170438400 (2024-01-05 00:00:00). In this way, when the master expired mob
thread starts, it may affect the life time of a MOB file for less than 1 day.
>
> Then the problem is we should use a timestamp instead of '20240105' in the
mob file name? I still do not understand why setting MILLISECOND to 0 can solve
the problem...
Here is an example:
1. Assume the Time-To-Live (TTL) for the mob data is set to 1 day.
2. We write mob data at 18:33 on 01/04/2023, and the data is flushed to a
mob file named 20230104.
3. The mob expiration thread starts within 1 day, at 10:45 on 01/05/2023.
4. When checking ts, the standard expired timestamp is calculated as
(currentTS - 1day parsed by Calendar) : 1704297600720 (2024-01-04 00:00:00)(3
random digits when only set to SECOND level)
The mob file's ts, parsed from its name, is : 170429760 (2024-01-04
00:00:00)
5. if (fileDate.getTime() < expireDate.getTime()) {/* expired */}
if statement is true, indicating that the mob file has expired, and it
will be cleaned. **These 3 random digits cause the mob files to be cleaned
earlier than expected.**
6. But if we set to MILLISECOND level, the expireDate.getTime() will be
170429760, and if statement will be false. In this case, the mob file will
be retained as intended.
And here is the link to jira:
https://issues.apache.org/jira/browse/HBASE-28287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[ https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-28259: - Fix Version/s: 2.6.0 2.4.18 2.5.8 > Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags > -- > > Key: HBASE-28259 > URL: https://issues.apache.org/jira/browse/HBASE-28259 > Project: HBase > Issue Type: Bug > Components: java >Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 4.0.0-alpha-1, 2.5.7, 2.7.0 >Reporter: Moran >Assignee: Moran >Priority: Trivial > Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2 > > > hbase shell > 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess > control requires open access to the JDK IO subsystem > Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens > java.base/java.io=ALL-UNNAMED' to enable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[ https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-28259: - Affects Version/s: 2.7.0 > Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags > -- > > Key: HBASE-28259 > URL: https://issues.apache.org/jira/browse/HBASE-28259 > Project: HBase > Issue Type: Bug > Components: java >Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 4.0.0-alpha-1, 2.5.7, 2.7.0 >Reporter: Moran >Assignee: Moran >Priority: Trivial > Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2 > > > hbase shell > 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess > control requires open access to the JDK IO subsystem > Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens > java.base/java.io=ALL-UNNAMED' to enable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[ https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-28259: - Affects Version/s: 2.5.7 2.4.17 3.0.0-alpha-4 2.6.0 4.0.0-alpha-1 > Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags > -- > > Key: HBASE-28259 > URL: https://issues.apache.org/jira/browse/HBASE-28259 > Project: HBase > Issue Type: Bug > Components: java >Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 4.0.0-alpha-1, 2.5.7 >Reporter: Moran >Assignee: Moran >Priority: Trivial > > hbase shell > 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess > control requires open access to the JDK IO subsystem > Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens > java.base/java.io=ALL-UNNAMED' to enable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[ https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-28259: - Fix Version/s: 4.0.0-alpha-1 2.7.0 3.0.0-beta-2 > Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags > -- > > Key: HBASE-28259 > URL: https://issues.apache.org/jira/browse/HBASE-28259 > Project: HBase > Issue Type: Bug > Components: java >Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 4.0.0-alpha-1, 2.5.7 >Reporter: Moran >Assignee: Moran >Priority: Trivial > Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2 > > > hbase shell > 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess > control requires open access to the JDK IO subsystem > Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens > java.base/java.io=ALL-UNNAMED' to enable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28259) Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags
[ https://issues.apache.org/jira/browse/HBASE-28259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-28259. -- Resolution: Fixed Merged to master, branch-3, branch-2, branch-2.6, branch-2.5 and branch-2.4. Thanks for the contribution, [~mrzhao] ! > Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags > -- > > Key: HBASE-28259 > URL: https://issues.apache.org/jira/browse/HBASE-28259 > Project: HBase > Issue Type: Bug > Components: java >Reporter: Moran >Assignee: Moran >Priority: Trivial > > hbase shell > 2023-12-13T23:49:50.846+08:00 [main] WARN FilenoUtil : Native subprocess > control requires open access to the JDK IO subsystem > Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens > java.base/java.io=ALL-UNNAMED' to enable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28271 Infinite waiting on lock acquisition by snapshot can result in unresponsive master [hbase]
frostruan commented on PR #5603: URL: https://github.com/apache/hbase/pull/5603#issuecomment-1878473043 Overall, LGTM, only a few details that I think we can discuss a little more. Thanks for the great work ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28271 Infinite waiting on lock acquisition by snapshot can result in unresponsive master [hbase]
frostruan commented on code in PR #5603:
URL: https://github.com/apache/hbase/pull/5603#discussion_r1442729082
##
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSnapshotProcedureWithLockTimeout.java:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.procedure;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.SnapshotDescription;
+import org.apache.hadoop.hbase.client.SnapshotType;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler;
+import org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility;
+import org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils;
+import org.apache.hadoop.hbase.testclassification.MasterTests;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.RegionSplitter;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos;
+
+@Category({ MasterTests.class, MediumTests.class })
+public class TestSnapshotProcedureWithLockTimeout {
Review Comment:
I am a little confused about that scenario this unit test is for. Would you
mind explaining a little bit ? Thanks. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28292 Make Delay prefetch property to be dynamically configured [hbase]
Apache-HBase commented on PR #5605: URL: https://github.com/apache/hbase/pull/5605#issuecomment-1878426688 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 33s | master passed | | +1 :green_heart: | compile | 0m 40s | master passed | | +1 :green_heart: | shadedjars | 4m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | shadedjars | 4m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 234m 26s | hbase-server in the patch failed. | | | | 256m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5605/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5605 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 79cbea8fc45b 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a545581ca | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5605/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5605/1/testReport/ | | Max. process+thread count | 4627 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5605/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28271 Infinite waiting on lock acquisition by snapshot can result in unresponsive master [hbase]
frostruan commented on code in PR #5603:
URL: https://github.com/apache/hbase/pull/5603#discussion_r1442698886
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java:
##
@@ -147,12 +152,16 @@ private TableDescriptor loadTableDescriptor() throws
IOException {
public TakeSnapshotHandler prepare() throws Exception {
super.prepare();
// after this, you should ensure to release this lock in case of exceptions
-this.tableLock.acquire();
-try {
- this.htd = loadTableDescriptor(); // check that .tableinfo is present
-} catch (Exception e) {
- this.tableLock.release();
- throw e;
+if (this.tableLock.tryAcquire(this.lockAcquireTimeoutMs)) {
+ try {
+this.htd = loadTableDescriptor(); // check that .tableinfo is present
+ } catch (Exception e) {
+this.tableLock.release();
+throw e;
+ }
+} else {
+ LOG.error("Master lock could not be acquired in {} ms",
lockAcquireTimeoutMs);
+ throw new IOException("Master lock could not be acquired");
Review Comment:
I don't think retrying makes much sense in this scenario, so I would prefer
to throw a DoNotRetryIOException like HBaseSnapshotException or
SnapshotCreationException. What do you think ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28292 Make Delay prefetch property to be dynamically configured [hbase]
Apache-HBase commented on PR #5605: URL: https://github.com/apache/hbase/pull/5605#issuecomment-1878424696 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 1s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 4m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 4m 51s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 230m 48s | hbase-server in the patch passed. | | | | 254m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5605/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5605 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 108dd21c32ec 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a545581ca | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5605/1/testReport/ | | Max. process+thread count | 4685 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5605/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1442694151
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileWriter.java:
##
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static
org.apache.hadoop.hbase.regionserver.HStoreFile.HAS_LIVE_VERSIONS_KEY;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Separates the provided cells into two files, one file for the live cells
and the other for the
+ * rest of the cells (historical cells). The live cells includes the live put
cells, delete all and
+ * version delete markers that are not masked by other delete all markers.
+ */
+@InterfaceAudience.Private
+public class DualFileWriter extends AbstractMultiFileWriter {
+
+ private final CellComparator comparator;
+ private StoreFileWriter liveVersionWriter;
+ private StoreFileWriter historicalVersionWriter;
+
+ private final List writers;
+ // The last cell of the current row
+ private Cell lastCell;
+ // The first (latest) delete family marker of the current row
+ private Cell deleteFamily;
+ // The list of delete family version markers of the current row
+ private List deleteFamilyVersionList = new ArrayList<>();
+ // The first (latest) delete column marker of the current column
+ private Cell deleteColumn;
+ // The list of delete column version markers of the current column
+ private List deleteColumnVersionList = new ArrayList<>();
+ // The live put cell count for the current column
+ private int livePutCellCount;
+ private final boolean dualWriterEnabled;
+ private final int maxVersions;
+ private final boolean newVersionBehavior;
+
+ public DualFileWriter(CellComparator comparator, int maxVersions, boolean
dualWriterEnabled,
+boolean newVersionBehavior) {
+this.comparator = comparator;
+this.maxVersions = maxVersions;
+this.dualWriterEnabled = dualWriterEnabled;
+this.newVersionBehavior = newVersionBehavior;
+writers = new ArrayList<>(2);
+initRowState();
+ }
+
+ private void initRowState() {
+deleteFamily = null;
+deleteFamilyVersionList.clear();
+lastCell = null;
+ }
+
+ private void initColumnState() {
+livePutCellCount = 0;
+deleteColumn = null;
+deleteColumnVersionList.clear();
+
+ }
+
+ private void addLiveVersion(Cell cell) throws IOException {
+if (liveVersionWriter == null) {
+ liveVersionWriter = writerFactory.createWriter();
+ writers.add(liveVersionWriter);
+}
+liveVersionWriter.append(cell);
+ }
+
+ private void addHistoricalVersion(Cell cell) throws IOException {
+if (historicalVersionWriter == null) {
+ historicalVersionWriter = writerFactory.createWriter();
+ writers.add(historicalVersionWriter);
+}
+historicalVersionWriter.append(cell);
+ }
+
+ private boolean isDeletedByDeleteFamily(Cell cell) {
+return deleteFamily != null && (deleteFamily.getTimestamp() >
cell.getTimestamp()
+ || (deleteFamily.getTimestamp() == cell.getTimestamp()
+&& (!newVersionBehavior || cell.getSequenceId() <
deleteFamily.getSequenceId(;
+ }
+
+ private boolean isDeletedByDeleteFamilyVersion(Cell cell) {
+for (Cell deleteFamilyVersion : deleteFamilyVersionList) {
+ if (
+deleteFamilyVersion.getTimestamp() == cell.getTimestamp()
+ && (!newVersionBehavior || cell.getSequenceId() <
deleteFamilyVersion.getSequenceId())
+ ) return true;
+}
+return false;
+ }
+
+ private boolean isDeletedByDeleteColumn(Cell cell) {
+return deleteColumn != null && (deleteColumn.getTimestamp() >
cell.getTimestamp()
+ || (deleteColumn.getTimestamp() == cell.getTimestamp()
+&& (!newVersionBehavior || cell.getSequenceId() <
deleteColumn.getSequenceId(;
+ }
+
Re: [PR] HBASE-28271 Infinite waiting on lock acquisition by snapshot can result in unresponsive master [hbase]
frostruan commented on code in PR #5603: URL: https://github.com/apache/hbase/pull/5603#discussion_r1442693980 ## hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java: ## @@ -129,6 +132,8 @@ public TakeSnapshotHandler(SnapshotDescription snapshot, final MasterServices ma "Taking " + snapshot.getType() + " snapshot on table: " + snapshotTable, false, true); this.snapshotManifest = SnapshotManifest.create(conf, rootFs, workingDir, snapshot, monitor, status); +this.lockAcquireTimeoutMs = + conf.getLong(HBASE_SNAPSHOT_MASTER_LOCK_ACQUIRE_TIMEOUT, 5 * 60 * 1000L); Review Comment: Would 5 minutes be a little long ? How about 60 seconds ? As I mentioned before, the client will time out after 60 seconds by default, so there seems to be no difference between waiting 60 seconds and waiting 5 minutes? What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28287 Fix the inaccurate expired time of MOB HFiles [hbase]
Apache9 commented on PR #5599: URL: https://github.com/apache/hbase/pull/5599#issuecomment-1878414176 > > I still do not fully understand the problem here... > > If we do not set millis to zero, it will only affect the life time of a MOB file for less than 1 second, how could it make the MOB file expire 2 hours earlier? > > Because in org.apache.hadoop.hbase.mob.MobUtils, the creation time of mob files is obtained by parsing their names from fileName using the statement (Date fileDate = parseDate(MobFileName.getDateFromName(fileName));). For instance, data created on 20240105, their timestamps will be parsed as 170438400 (2024-01-05 00:00:00). In this way, when the master expired mob thread starts, it may affect the life time of a MOB file for less than 1 day. Then the problem is we should use a timestamp instead of '20240105' to the mob file name? I still do not understand why setting MILLISECOND to 0 can solve the problem... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1442683796
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java:
##
@@ -124,7 +124,7 @@ public static List
getScannersForStoreFiles(Collection
getScannersForStoreFiles(Collection files,
boolean cacheBlocks, boolean usePread, boolean isCompaction, boolean
canUseDrop,
-ScanQueryMatcher matcher, long readPt) throws IOException {
+ScanQueryMatcher matcher, long readPt, boolean onlyLatestVersion) throws
IOException {
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1442652833
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStoreFile.java:
##
@@ -138,6 +140,12 @@ public class HStoreFile implements StoreFile {
// Indicates if the file got compacted
private volatile boolean compactedAway = false;
+ // Indicate if the file contains live cell versions for a given column
+ // in a row. MemStore flushes generate files with all cell versions. However,
+ // compactions can generate two files, one with the live cell versions and
the other
+ // with the remaining (historical) cell versions.
+ private volatile boolean hasLiveVersions = true;
Review Comment:
Even when `hbase.hstore.defaultengine.enable.dualfilewriter` is enabled, the
files generated by memstore flushes do not include the HAS_LIVE_VERSIONS HFile
metadata key since memstore does not use compaction writers. However, these
files will include live version cells. When the HAS_LIVE_VERSIONS HFile
metadata key is not included in the HFile, we still want to return true for
HStoreFile#hasLiveVersion(). That is why the default is true. I will add more
comments for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28259 Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags [hbase]
wchevreuil merged PR #5581: URL: https://github.com/apache/hbase/pull/5581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1442621969
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileWriter.java:
##
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static
org.apache.hadoop.hbase.regionserver.HStoreFile.HAS_LIVE_VERSIONS_KEY;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Separates the provided cells into two files, one file for the live cells
and the other for the
+ * rest of the cells (historical cells). The live cells includes the live put
cells, delete all and
+ * version delete markers that are not masked by other delete all markers.
+ */
+@InterfaceAudience.Private
+public class DualFileWriter extends AbstractMultiFileWriter {
+
+ private final CellComparator comparator;
+ private StoreFileWriter liveVersionWriter;
+ private StoreFileWriter historicalVersionWriter;
+
+ private final List writers;
+ // The last cell of the current row
+ private Cell lastCell;
+ // The first (latest) delete family marker of the current row
+ private Cell deleteFamily;
+ // The list of delete family version markers of the current row
+ private List deleteFamilyVersionList = new ArrayList<>();
+ // The first (latest) delete column marker of the current column
+ private Cell deleteColumn;
+ // The list of delete column version markers of the current column
+ private List deleteColumnVersionList = new ArrayList<>();
+ // The live put cell count for the current column
+ private int livePutCellCount;
+ private final boolean dualWriterEnabled;
+ private final int maxVersions;
+ private final boolean newVersionBehavior;
+
+ public DualFileWriter(CellComparator comparator, int maxVersions, boolean
dualWriterEnabled,
+boolean newVersionBehavior) {
+this.comparator = comparator;
+this.maxVersions = maxVersions;
+this.dualWriterEnabled = dualWriterEnabled;
+this.newVersionBehavior = newVersionBehavior;
+writers = new ArrayList<>(2);
+initRowState();
+ }
+
+ private void initRowState() {
+deleteFamily = null;
+deleteFamilyVersionList.clear();
+lastCell = null;
+ }
+
+ private void initColumnState() {
+livePutCellCount = 0;
+deleteColumn = null;
+deleteColumnVersionList.clear();
+
+ }
+
+ private void addLiveVersion(Cell cell) throws IOException {
+if (liveVersionWriter == null) {
+ liveVersionWriter = writerFactory.createWriter();
+ writers.add(liveVersionWriter);
+}
+liveVersionWriter.append(cell);
+ }
+
+ private void addHistoricalVersion(Cell cell) throws IOException {
+if (historicalVersionWriter == null) {
+ historicalVersionWriter = writerFactory.createWriter();
+ writers.add(historicalVersionWriter);
+}
+historicalVersionWriter.append(cell);
+ }
+
+ private boolean isDeletedByDeleteFamily(Cell cell) {
+return deleteFamily != null && (deleteFamily.getTimestamp() >
cell.getTimestamp()
+ || (deleteFamily.getTimestamp() == cell.getTimestamp()
+&& (!newVersionBehavior || cell.getSequenceId() <
deleteFamily.getSequenceId(;
+ }
+
+ private boolean isDeletedByDeleteFamilyVersion(Cell cell) {
+for (Cell deleteFamilyVersion : deleteFamilyVersionList) {
+ if (
+deleteFamilyVersion.getTimestamp() == cell.getTimestamp()
+ && (!newVersionBehavior || cell.getSequenceId() <
deleteFamilyVersion.getSequenceId())
+ ) return true;
+}
+return false;
+ }
+
+ private boolean isDeletedByDeleteColumn(Cell cell) {
+return deleteColumn != null && (deleteColumn.getTimestamp() >
cell.getTimestamp()
+ || (deleteColumn.getTimestamp() == cell.getTimestamp()
+&& (!newVersionBehavior || cell.getSequenceId() <
deleteColumn.getSequenceId(;
+ }
+
[jira] [Assigned] (HBASE-28292) Make Delay prefetch property to be dynamically configured
[ https://issues.apache.org/jira/browse/HBASE-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros reassigned HBASE-28292: --- Assignee: Abhishek Kothalikar > Make Delay prefetch property to be dynamically configured > - > > Key: HBASE-28292 > URL: https://issues.apache.org/jira/browse/HBASE-28292 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Kothalikar >Assignee: Abhishek Kothalikar >Priority: Major > > Rolling restart triggers region movement on the cluster while the > RegionServers are getting restarted. During this time, the temporary > RegionServers will start prefetching these files which are only hosted until > the source RegionServer is restarted. Hence, in this timing window, fetches > are executed on temporary region servers which takes few minutes. > Currrently, this behaviour is governed by hbase.hfile.prefetch.delay > Also, file-based block cache gives performance improvement on HBase clusters > using object storage. COD uses a large ephemeral disk for this purpose. > > Possible solution is, > > The number of prefetches on the cluster caused by these temporary region > movements during Rolling Upgrade (rolling restart) can be mininized by > introducing delay before the prefetch starts. > To achieve this, making the hbase.hfile.prefetch.delay dynamically > configurable. The interval will let the rolling restart complete. > Resultantly, prefetches on temporary region server can be minimized. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1442606527 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java: ## @@ -46,13 +45,16 @@ public class DefaultStoreEngine extends StoreEngine DEFAULT_STORE_FLUSHER_CLASS = DefaultStoreFlusher.class; private static final Class DEFAULT_COMPACTOR_CLASS = DefaultCompactor.class; - private static final Class DEFAULT_COMPACTION_POLICY_CLASS = + public static final Class DEFAULT_COMPACTION_POLICY_CLASS = ExploringCompactionPolicy.class; + public static final boolean DEFAULT_ENABLE_DUAL_FILE_WRITER = false; Review Comment: I did that couple commits back and will do it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
