[jira] [Commented] (HBASE-28055) Performance improvement for scan over several stores.
[ https://issues.apache.org/jira/browse/HBASE-28055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762283#comment-17762283 ] Tak-Lon (Stephen) Wu commented on HBASE-28055: -- merged into master , will do branch-2 tomorrow. > Performance improvement for scan over several stores. > -- > > Key: HBASE-28055 > URL: https://issues.apache.org/jira/browse/HBASE-28055 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-4, 2.5.5 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov >Priority: Major > > During the fix of HBASE-19863, an additional check for fake cells that > trigger reseek was added. It comes that this check produces unnecessary > reseeks because > matcher.compareKeyForNextColumn should be used only with indexed keys. Later > [~larsh] suggested doing a simple check for OLD_TIMESTAMP and it looks like a > better solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28038) Add TLS settings to ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-28038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762267#comment-17762267
]
Hudson commented on HBASE-28038:
Results for branch branch-2
[build #875 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add TLS settings to ZooKeeper client
>
>
> Key: HBASE-28038
> URL: https://issues.apache.org/jira/browse/HBASE-28038
> Project: HBase
> Issue Type: Improvement
> Components: Zookeeper
>Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.5
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Labels: ssl, tls, zookeeper
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> ZooKeeper supports TLS connection from its clients. Currently the only way to
> set up HBase for this is to add the following Java properties to the HBase
> process:
> {noformat}
>
> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
> -Dzookeeper.client.secure=true
> -Dzookeeper.ssl.keyStore.location=/path/to/keystore.jks
> -Dzookeeper.ssl.keyStore.password=password
> -Dzookeeper.ssl.trustStore.location=/path/to/truststore.jks
> -Dzookeeper.ssl.trustStore.password=password
> {noformat}
> KeyStore is only needed if ZooKeeper server wants client certificate to be
> provided.
> I'd like to add these options to hbase-site.xml in the following way:
> {noformat}
> hbase.zookeeper.property.clientCnxnSocket
> hbase.zookeeper.property.client.secure
> hbase.zookeeper.property.ssl.keyStore.location
> hbase.zookeeper.property.ssl.keyStore.password or
> hbase.zookeeper.property.ssl.keyStore.passwordPath
> ...{noformat}
> It will follow the way that we already do for ZooKeeper clientPort and quorum
> settings. ("hbase.zookeeper.property.clientPort", "hbase.zookeeper.quorum")
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28059) Use correct units in RegionLoad#getStoreUncompressedSizeMB()
[
https://issues.apache.org/jira/browse/HBASE-28059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762268#comment-17762268
]
Hudson commented on HBASE-28059:
Results for branch branch-2
[build #875 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/875/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use correct units in RegionLoad#getStoreUncompressedSizeMB()
>
>
> Key: HBASE-28059
> URL: https://issues.apache.org/jira/browse/HBASE-28059
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.5.5
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 2.5.6
>
>
> When I run a snippet of code like this:
> {code:java}
> Map regionLoadMap = admin
> .getClusterStatus()
> .getLoad(
> ServerName.parseServerName(
> "my-server.my-company.net,60020,1693513660506"
> )
> )
> .getRegionsLoad();
> for (byte[] startKey : regionLoadMap.keySet()) {
> RegionLoad regionLoad = regionLoadMap.get(startKey);
> LOG.info("Region {}: {}", Bytes.toStringBinary(startKey), regionLoad);
> } {code}
> I get logs like this:
> {noformat}
> Region , key>,1659484033280.2b89407a1223720344bed425bf3c29b0.: numberOfStores=1,
> numberOfStorefiles=3, storeRefCount=0, storefileUncompressedSizeMB=17998848,
> lastMajorCompactionTimestamp=1693211464712, storefileSizeMB=5895,
> compressionRatio=0.0003, memstoreSizeMB=1, readRequestsCount=118899553,
> writeRequestsCount=731192, rootIndexSizeKB=9, totalStaticIndexSizeKB=10413,
> totalStaticBloomSizeKB=6592, totalCompactingKVs=0, currentCompactedKVs=0,
> compactionProgressPct=NaN, completeSequenceId=78093096,
> dataLocality=1.0{noformat}
> The {{storefileUncompressedSizeMB}} is vastly larger than the
> {{{}storefileSizeMB{}}}, so much more that it's not believable. I checked the
> store files in question in this instance. Adding up the uncompressed size
> reported in the HFile trailers sums to 5895 MiB, exactly 1024 times less than
> the reported 17998848.
> The reason for the misreporting is that
> {{RegionLoad#getStoreUncompressedSizeMB()}} converts the value from a
> {{Size}} object wrong.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28059) Use correct units in RegionLoad#getStoreUncompressedSizeMB()
[
https://issues.apache.org/jira/browse/HBASE-28059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762252#comment-17762252
]
Hudson commented on HBASE-28059:
Results for branch branch-2.4
[build #612 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use correct units in RegionLoad#getStoreUncompressedSizeMB()
>
>
> Key: HBASE-28059
> URL: https://issues.apache.org/jira/browse/HBASE-28059
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.5.5
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 2.5.6
>
>
> When I run a snippet of code like this:
> {code:java}
> Map regionLoadMap = admin
> .getClusterStatus()
> .getLoad(
> ServerName.parseServerName(
> "my-server.my-company.net,60020,1693513660506"
> )
> )
> .getRegionsLoad();
> for (byte[] startKey : regionLoadMap.keySet()) {
> RegionLoad regionLoad = regionLoadMap.get(startKey);
> LOG.info("Region {}: {}", Bytes.toStringBinary(startKey), regionLoad);
> } {code}
> I get logs like this:
> {noformat}
> Region , key>,1659484033280.2b89407a1223720344bed425bf3c29b0.: numberOfStores=1,
> numberOfStorefiles=3, storeRefCount=0, storefileUncompressedSizeMB=17998848,
> lastMajorCompactionTimestamp=1693211464712, storefileSizeMB=5895,
> compressionRatio=0.0003, memstoreSizeMB=1, readRequestsCount=118899553,
> writeRequestsCount=731192, rootIndexSizeKB=9, totalStaticIndexSizeKB=10413,
> totalStaticBloomSizeKB=6592, totalCompactingKVs=0, currentCompactedKVs=0,
> compactionProgressPct=NaN, completeSequenceId=78093096,
> dataLocality=1.0{noformat}
> The {{storefileUncompressedSizeMB}} is vastly larger than the
> {{{}storefileSizeMB{}}}, so much more that it's not believable. I checked the
> store files in question in this instance. Adding up the uncompressed size
> reported in the HFile trailers sums to 5895 MiB, exactly 1024 times less than
> the reported 17998848.
> The reason for the misreporting is that
> {{RegionLoad#getStoreUncompressedSizeMB()}} converts the value from a
> {{Size}} object wrong.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28038) Add TLS settings to ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-28038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762251#comment-17762251
]
Hudson commented on HBASE-28038:
Results for branch branch-2.4
[build #612 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/612/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add TLS settings to ZooKeeper client
>
>
> Key: HBASE-28038
> URL: https://issues.apache.org/jira/browse/HBASE-28038
> Project: HBase
> Issue Type: Improvement
> Components: Zookeeper
>Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.5
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Labels: ssl, tls, zookeeper
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> ZooKeeper supports TLS connection from its clients. Currently the only way to
> set up HBase for this is to add the following Java properties to the HBase
> process:
> {noformat}
>
> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
> -Dzookeeper.client.secure=true
> -Dzookeeper.ssl.keyStore.location=/path/to/keystore.jks
> -Dzookeeper.ssl.keyStore.password=password
> -Dzookeeper.ssl.trustStore.location=/path/to/truststore.jks
> -Dzookeeper.ssl.trustStore.password=password
> {noformat}
> KeyStore is only needed if ZooKeeper server wants client certificate to be
> provided.
> I'd like to add these options to hbase-site.xml in the following way:
> {noformat}
> hbase.zookeeper.property.clientCnxnSocket
> hbase.zookeeper.property.client.secure
> hbase.zookeeper.property.ssl.keyStore.location
> hbase.zookeeper.property.ssl.keyStore.password or
> hbase.zookeeper.property.ssl.keyStore.passwordPath
> ...{noformat}
> It will follow the way that we already do for ZooKeeper clientPort and quorum
> settings. ("hbase.zookeeper.property.clientPort", "hbase.zookeeper.quorum")
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28064) Implement truncate_region command to truncate region directly from FS
Ankit Singhal created HBASE-28064: - Summary: Implement truncate_region command to truncate region directly from FS Key: HBASE-28064 URL: https://issues.apache.org/jira/browse/HBASE-28064 Project: HBase Issue Type: New Feature Reporter: Ankit Singhal One of our users has brought up a use-case where they need to truncate a region to delete data within a specific range. There are two scenarios to consider: * In the first scenario, the region boundaries involve a time range defined through pre-splitting, and user is looking to efficiently clean old date data. If HBase can directly truncate the region from the file system and then the user can merge the empty region with adjacent regions to effectively eliminate it which will be more optimized compared to deleting the data using Delete API. * In another case, if the HFile for that region becomes corrupted for some reason, user want to get rid of the HFile and reload the entire region to avoid consistency issues and ensure performance. we can do this when the table is offline/disabled to avoid the consideration of race conditions involving Region In Transition (RITs), region re-opening, and merge/split scenarios, as taking the region offline is necessary regardless -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase-connectors] dependabot[bot] opened a new pull request, #124: Bump org.apache.kafka:kafka-clients from 2.0.0 to 2.6.3 in /kafka/hbase-kafka-proxy
dependabot[bot] opened a new pull request, #124: URL: https://github.com/apache/hbase-connectors/pull/124 Bumps org.apache.kafka:kafka-clients from 2.0.0 to 2.6.3. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/hbase-connectors/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28059) Use correct units in RegionLoad#getStoreUncompressedSizeMB()
[
https://issues.apache.org/jira/browse/HBASE-28059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762198#comment-17762198
]
Hudson commented on HBASE-28059:
Results for branch branch-2.5
[build #397 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use correct units in RegionLoad#getStoreUncompressedSizeMB()
>
>
> Key: HBASE-28059
> URL: https://issues.apache.org/jira/browse/HBASE-28059
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.5.5
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 2.5.6
>
>
> When I run a snippet of code like this:
> {code:java}
> Map regionLoadMap = admin
> .getClusterStatus()
> .getLoad(
> ServerName.parseServerName(
> "my-server.my-company.net,60020,1693513660506"
> )
> )
> .getRegionsLoad();
> for (byte[] startKey : regionLoadMap.keySet()) {
> RegionLoad regionLoad = regionLoadMap.get(startKey);
> LOG.info("Region {}: {}", Bytes.toStringBinary(startKey), regionLoad);
> } {code}
> I get logs like this:
> {noformat}
> Region , key>,1659484033280.2b89407a1223720344bed425bf3c29b0.: numberOfStores=1,
> numberOfStorefiles=3, storeRefCount=0, storefileUncompressedSizeMB=17998848,
> lastMajorCompactionTimestamp=1693211464712, storefileSizeMB=5895,
> compressionRatio=0.0003, memstoreSizeMB=1, readRequestsCount=118899553,
> writeRequestsCount=731192, rootIndexSizeKB=9, totalStaticIndexSizeKB=10413,
> totalStaticBloomSizeKB=6592, totalCompactingKVs=0, currentCompactedKVs=0,
> compactionProgressPct=NaN, completeSequenceId=78093096,
> dataLocality=1.0{noformat}
> The {{storefileUncompressedSizeMB}} is vastly larger than the
> {{{}storefileSizeMB{}}}, so much more that it's not believable. I checked the
> store files in question in this instance. Adding up the uncompressed size
> reported in the HFile trailers sums to 5895 MiB, exactly 1024 times less than
> the reported 17998848.
> The reason for the misreporting is that
> {{RegionLoad#getStoreUncompressedSizeMB()}} converts the value from a
> {{Size}} object wrong.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28038) Add TLS settings to ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-28038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762197#comment-17762197
]
Hudson commented on HBASE-28038:
Results for branch branch-2.5
[build #397 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/397/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add TLS settings to ZooKeeper client
>
>
> Key: HBASE-28038
> URL: https://issues.apache.org/jira/browse/HBASE-28038
> Project: HBase
> Issue Type: Improvement
> Components: Zookeeper
>Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.5
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Labels: ssl, tls, zookeeper
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> ZooKeeper supports TLS connection from its clients. Currently the only way to
> set up HBase for this is to add the following Java properties to the HBase
> process:
> {noformat}
>
> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
> -Dzookeeper.client.secure=true
> -Dzookeeper.ssl.keyStore.location=/path/to/keystore.jks
> -Dzookeeper.ssl.keyStore.password=password
> -Dzookeeper.ssl.trustStore.location=/path/to/truststore.jks
> -Dzookeeper.ssl.trustStore.password=password
> {noformat}
> KeyStore is only needed if ZooKeeper server wants client certificate to be
> provided.
> I'd like to add these options to hbase-site.xml in the following way:
> {noformat}
> hbase.zookeeper.property.clientCnxnSocket
> hbase.zookeeper.property.client.secure
> hbase.zookeeper.property.ssl.keyStore.location
> hbase.zookeeper.property.ssl.keyStore.password or
> hbase.zookeeper.property.ssl.keyStore.passwordPath
> ...{noformat}
> It will follow the way that we already do for ZooKeeper clientPort and quorum
> settings. ("hbase.zookeeper.property.clientPort", "hbase.zookeeper.quorum")
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] anmolnar commented on a diff in pull request #5383: HBASE-28063. Added docs for ZooKeeper TLS connection
anmolnar commented on code in PR #5383: URL: https://github.com/apache/hbase/pull/5383#discussion_r1316101143 ## src/main/asciidoc/_chapters/zookeeper.adoc: ## @@ -441,7 +441,70 @@ This would avoid the need for a separate Hadoop jar that fixes link:https://issu Elimination of `kerberos.removeHostFromPrincipal` and`kerberos.removeRealmFromPrincipal` +== TLS connection to ZooKeeper +Apache ZooKeeper also supports SSL/TLS client connections to encrypt the data in transmission. This is particularly +useful when the ZooKeeper ensemble is running on a host different from HBase and data has to be sent +over the wire. + +=== Java system properties + +The ZooKeeper client supports the following Java system properties to set up TLS connection: + +[source,bourne] + +zookeeper.client.secure=true +zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty +zookeeper.ssl.keyStore.location="/path/to/your/keystore" +zookeeper.ssl.keyStore.password="keystore_password" +zookeeper.ssl.trustStore.location="/path/to/your/truststore" +zookeeper.ssl.trustStore.password="truststore_password" + + +Setting up KeyStore is optional and only required if ZooKeeper server requests for client certificate. + +Find more detailed information in the link:https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide[ZooKeeper SSL User Guide]. + +[WARNING] +These're standard Java properties which should be set in the HBase command line and are effective in +the entire Java process. All ZooKeeper clients running in the same process will pick them up including +co-processors. + +[NOTE] +Since ZooKeeper version 3.8 the following two properties are useful to store the +keystore and truststore passwords in protected text files rather than exposing them in the command line. + +[source,bourne] + +zookeeper.ssl.keyStore.passwordPath=/path/to/secure/file +zookeeper.ssl.trustStore.passwordPath=/path/to/secure/file + + +=== HBase configuration + +Since HBase version 2.4.18 ZooKeeper client TLS settings are also available in _hbase-site.xml_ via Review Comment: Makes sense to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5383: HBASE-28063. Added docs for ZooKeeper TLS connection
Apache-HBase commented on PR #5383: URL: https://github.com/apache/hbase/pull/5383#issuecomment-1706757512 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | master passed | | +0 :ok: | refguide | 2m 54s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 2m 14s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 17m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5383/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5383 | | Optional Tests | dupname asflicense refguide spotless | | uname | Linux adf3556f7ac6 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 198385aa7b | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5383/1/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5383/1/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5383/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-operator-tools] Apache-HBase commented on pull request #135: HBASE-27961 Running assigns/unassigns command with large number of files/regions throws CallTimeoutException
Apache-HBase commented on PR #135: URL: https://github.com/apache/hbase-operator-tools/pull/135#issuecomment-1706755935 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +0 :ok: | spotbugs | 0m 0s | spotbugs executables are not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 51s | master passed | | +1 :green_heart: | compile | 0m 10s | master passed | | +1 :green_heart: | checkstyle | 0m 8s | master passed | | +1 :green_heart: | spotless | 0m 7s | branch has no errors when running spotless:check. | | +1 :green_heart: | javadoc | 0m 7s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 11s | the patch passed | | +1 :green_heart: | compile | 0m 10s | the patch passed | | +1 :green_heart: | javac | 0m 10s | the patch passed | | +1 :green_heart: | checkstyle | 0m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 4s | patch has no errors when running spotless:check. | | +1 :green_heart: | javadoc | 0m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 4m 38s | hbase-hbck2 in the patch passed. | | +1 :green_heart: | asflicense | 0m 6s | The patch does not generate ASF License warnings. | | | | 8m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-135/3/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-operator-tools/pull/135 | | Optional Tests | dupname asflicense spotless markdownlint javac javadoc unit spotbugs findbugs checkstyle compile | | uname | Linux 1abc46019ed1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/jenkins/hbase-personality.sh | | git revision | master / 0207601 | | Default Java | Oracle Corporation-1.8.0_342-b07 | | Test Results | https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-135/3/testReport/ | | Max. process+thread count | 1274 (vs. ulimit of 5000) | | modules | C: hbase-hbck2 U: hbase-hbck2 | | Console output | https://ci-hbase.apache.org/job/HBase-Operator-Tools-PreCommit/job/PR-135/3/console | | versions | git=2.30.2 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] NihalJain commented on a diff in pull request #5383: HBASE-28063. Added docs for ZooKeeper TLS connection
NihalJain commented on code in PR #5383: URL: https://github.com/apache/hbase/pull/5383#discussion_r1316001563 ## src/main/asciidoc/_chapters/zookeeper.adoc: ## @@ -441,7 +441,70 @@ This would avoid the need for a separate Hadoop jar that fixes link:https://issu Elimination of `kerberos.removeHostFromPrincipal` and`kerberos.removeRealmFromPrincipal` +== TLS connection to ZooKeeper +Apache ZooKeeper also supports SSL/TLS client connections to encrypt the data in transmission. This is particularly +useful when the ZooKeeper ensemble is running on a host different from HBase and data has to be sent +over the wire. + +=== Java system properties + +The ZooKeeper client supports the following Java system properties to set up TLS connection: + +[source,bourne] + +zookeeper.client.secure=true +zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty +zookeeper.ssl.keyStore.location="/path/to/your/keystore" +zookeeper.ssl.keyStore.password="keystore_password" +zookeeper.ssl.trustStore.location="/path/to/your/truststore" +zookeeper.ssl.trustStore.password="truststore_password" + + +Setting up KeyStore is optional and only required if ZooKeeper server requests for client certificate. + +Find more detailed information in the link:https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide[ZooKeeper SSL User Guide]. + +[WARNING] +These're standard Java properties which should be set in the HBase command line and are effective in +the entire Java process. All ZooKeeper clients running in the same process will pick them up including +co-processors. + +[NOTE] +Since ZooKeeper version 3.8 the following two properties are useful to store the +keystore and truststore passwords in protected text files rather than exposing them in the command line. + +[source,bourne] + +zookeeper.ssl.keyStore.passwordPath=/path/to/secure/file +zookeeper.ssl.trustStore.passwordPath=/path/to/secure/file + + +=== HBase configuration + +Since HBase version 2.4.18 ZooKeeper client TLS settings are also available in _hbase-site.xml_ via Review Comment: Is it worth mentioning the JIRA# so that users get a full list of versions where the fix landed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28060) Region server getting crashed during write - A fatal error has been detected by the Java Runtime Environment
[
https://issues.apache.org/jira/browse/HBASE-28060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762118#comment-17762118
]
Pankaj Kumar commented on HBASE-28060:
--
HBASE-26024 also have similar error.
> Region server getting crashed during write - A fatal error has been detected
> by the Java Runtime Environment
>
>
> Key: HBASE-28060
> URL: https://issues.apache.org/jira/browse/HBASE-28060
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.4.14
>Reporter: Pankaj Kumar
>Priority: Critical
>
> In our production environment, Region server JVM is getting crashed randomly
> with the following error logs.
> {code:java}
> Stack: [0xfffdd32d8000,0xfffdd34d8000], sp=0xfffdd34d6000, free
> space=2040k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jshort_disjoint_arraycopy
> J 98497 C2
> org.apache.hadoop.hbase.io.ByteBufferListOutputStream.write(Ljava/nio/ByteBuffer;II)V
> (53 bytes) @ 0x8cc9a154 [0x8cc9a040+0x114]
> C [ld-linux-aarch64.so.1+0x40018] _rtld_global+0x0
> [error occurred during error reporting (printing native stack), id 0xb]
> {code}
>
> We are using HBase 2.4.14.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #5383: HBASE-28063. Added docs for ZooKeeper TLS connection
Apache-HBase commented on PR #5383: URL: https://github.com/apache/hbase/pull/5383#issuecomment-1706731441 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 2m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5383/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5383 | | Optional Tests | | | uname | Linux 688df0ed2d26 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 198385aa7b | | Max. process+thread count | 34 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5383/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5383: HBASE-28063. Added docs for ZooKeeper TLS connection
Apache-HBase commented on PR #5383: URL: https://github.com/apache/hbase/pull/5383#issuecomment-1706729701 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5383/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5383 | | Optional Tests | | | uname | Linux 4ed46b59312f 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 198385aa7b | | Max. process+thread count | 39 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5383/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-operator-tools] NihalJain commented on pull request #135: HBASE-27961 Running assigns/unassigns command with large number of files/regions throws CallTimeoutException
NihalJain commented on PR #135: URL: https://github.com/apache/hbase-operator-tools/pull/135#issuecomment-1706728002 Rebased code and removed unrelated code format changes as they have already been fixed with https://github.com/apache/hbase-operator-tools/pull/137 @petersomogyi , @wchevreuil could you please review the changes? Also, why this is needed is captured in [JIRA](https://issues.apache.org/jira/browse/HBASE-27961) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28063) Add documentation to HBase book
[ https://issues.apache.org/jira/browse/HBASE-28063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andor Molnar updated HBASE-28063: - Status: Patch Available (was: Open) https://github.com/apache/hbase/pull/5383 > Add documentation to HBase book > --- > > Key: HBASE-28063 > URL: https://issues.apache.org/jira/browse/HBASE-28063 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Andor Molnar >Assignee: Andor Molnar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27976) [hbase-operator-tools] Add spotless for hbase-operator-tools
[ https://issues.apache.org/jira/browse/HBASE-27976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-27976: --- Component/s: community > [hbase-operator-tools] Add spotless for hbase-operator-tools > > > Key: HBASE-27976 > URL: https://issues.apache.org/jira/browse/HBASE-27976 > Project: HBase > Issue Type: Umbrella > Components: build, community, hbase-operator-tools >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: hbase-operator-tools-1.3.0 > > > HBase code repo has spotless plugin to check and fix spotless issues > seamlessly, making it easier for developers to fix issue in case the builds > fails due to code formatting. > The goal of this Jira is to integrate spotless with hbase-operator-tools. > * As a 1st step will try to add a plugin to run spotless check via maven > * Next will fix all spotless issues as part of same task or another (as > community suggests) > * Finally will integrate the same to pre-commit build to not let PRs wit > spotless issues get in. (Would need some support/direction on how to do this > as I am not much familiar with the Jenkins and related code.) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27976) [hbase-operator-tools] Add spotless for hbase-operator-tools
[ https://issues.apache.org/jira/browse/HBASE-27976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain resolved HBASE-27976. Fix Version/s: hbase-operator-tools-1.3.0 Release Note: Before creating a PR for hbase-operator-tools repo, developers can now run 'mvn spotless:apply' to fix code formatting issues . Resolution: Fixed All the sub-tasks are done, marking the Jira as resolved. > [hbase-operator-tools] Add spotless for hbase-operator-tools > > > Key: HBASE-27976 > URL: https://issues.apache.org/jira/browse/HBASE-27976 > Project: HBase > Issue Type: Umbrella > Components: build, hbase-operator-tools >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: hbase-operator-tools-1.3.0 > > > HBase code repo has spotless plugin to check and fix spotless issues > seamlessly, making it easier for developers to fix issue in case the builds > fails due to code formatting. > The goal of this Jira is to integrate spotless with hbase-operator-tools. > * As a 1st step will try to add a plugin to run spotless check via maven > * Next will fix all spotless issues as part of same task or another (as > community suggests) > * Finally will integrate the same to pre-commit build to not let PRs wit > spotless issues get in. (Would need some support/direction on how to do this > as I am not much familiar with the Jenkins and related code.) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27976) [hbase-operator-tools] Add spotless for hbase-operator-tools
[ https://issues.apache.org/jira/browse/HBASE-27976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-27976: --- Issue Type: Umbrella (was: Task) > [hbase-operator-tools] Add spotless for hbase-operator-tools > > > Key: HBASE-27976 > URL: https://issues.apache.org/jira/browse/HBASE-27976 > Project: HBase > Issue Type: Umbrella > Components: build, hbase-operator-tools >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > > HBase code repo has spotless plugin to check and fix spotless issues > seamlessly, making it easier for developers to fix issue in case the builds > fails due to code formatting. > The goal of this Jira is to integrate spotless with hbase-operator-tools. > * As a 1st step will try to add a plugin to run spotless check via maven > * Next will fix all spotless issues as part of same task or another (as > community suggests) > * Finally will integrate the same to pre-commit build to not let PRs wit > spotless issues get in. (Would need some support/direction on how to do this > as I am not much familiar with the Jenkins and related code.) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28034) Rewrite hbck2 documentation using GPT
[
https://issues.apache.org/jira/browse/HBASE-28034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-28034:
---
Description:
Just a thought, could we re-write the operator tools
[README.md|https://github.com/apache/hbase-operator-tools/blob/master/hbase-hbck2/README.md]
using GPT and make it better?
A sample paragraph re-written by ChatGPT is as follows:
Original:
{quote}
h3. Some General Principals
When making repair, make sure hbase:meta is consistent first before you go
about fixing any other issue type such as a filesystem deviance. Deviance in
the filesystem or problems with assign should be addressed after the hbase:meta
has been put in order. If hbase:meta is out of whack, the Master cannot make
proper placements when adopting orphan filesystem data or making region
assignments.
Other general principles to keep in mind include a Region can not be assigned
if it is in _CLOSING_ state (or the inverse, unassigned if in _OPENING_ state)
without first transitioning via {_}CLOSED{_}: Regions must always move from
{_}CLOSED{_}, to {_}OPENING{_}, to {_}OPEN{_}, and then to {_}CLOSING{_},
{_}CLOSED{_}.
When making repair, do fixup of a table-at-a-time.
Also, if a table is {_}DISABLED{_}, you cannot assign a Region. In the Master
logs, you will see that the Master will report that the assign has been skipped
because the table is {_}DISABLED{_}. You may want to assign a Region because it
is currently in the _OPENING_ state and you want it in the _CLOSED_ state so it
agrees with the table's _DISABLED_ state. In this situation, you may have to
temporarily set the table status to {_}ENABLED{_}, just so you can do the
assign, and then set it back again after the unassign. _HBCK2_ has facility to
allow you do this. See the _HBCK2_ usage output.
What follows is a mix of notes and prescription that comes of experience
running hbase-2.x so far. The root issues that brought on states described
below has been fixed in later versions of hbase so upgrade if you can so as to
avoid scenarios described.
{quote}
Re-written using ChatGPT:
{quote}*Some Basic Principles*
Ensure hbase:meta's consistency before attempting to fix any other type of
problem, such as filesystem deviations. Issues related to filesystem or
assignment should be addressed once hbase:meta is sorted out. If hbase:meta is
awry, the Master cannot accurately allocate orphan filesystem data or region
assignments.
Remember that a Region cannot be assigned if it is in a CLOSING state (or
conversely, unassigned if in an OPENING state) without going through a CLOSED
state first. Regions must always progress from CLOSED, to OPENING, to OPEN, to
CLOSING, and then back to CLOSED.
Make repairs table-by-table.
Additionally, if a table is DISABLED, you cannot assign a Region. The Master
logs will indicate that the assignment has been bypassed due to the table's
DISABLED status. You might want to assign a Region because it is currently in
the OPENING state, but you want it in the CLOSED state to match the DISABLED
state of the table. In such cases, you might need to briefly change the table
status to ENABLED to make the assignment, then switch it back after the
unassignment. HBCK2 provides a facility for this. Please refer to the HBCK2
usage output.
The following notes and instructions come from the experience of running
hbase-2.x so far. The underlying issues causing the states described below have
been resolved in later versions of hbase, so upgrading is recommended to avoid
these scenarios.
{quote}
Is this worth the effort? Or do others feel current doc is good and does not
need any refinement?
It may require some effort, as we may only start with first commit with
untouched document generated by ChatGPT, but then the draft would need to be
worked upon, based on some proofreading by the contributor and reviewers.
Curious to know how others feel.
Also, Apache has some guidelines around using of generative ai tools at
[https://www.apache.org/legal/generative-tooling.html]
was:
Just a thought, could we re-write the operator tools
[README.md|https://github.com/apache/hbase-operator-tools/blob/master/README.md]
using GPT and make it better?
A sample paragraph re-written by ChatGPT is as follows:
Original:
{quote}
h3. Some General Principals
When making repair, make sure hbase:meta is consistent first before you go
about fixing any other issue type such as a filesystem deviance. Deviance in
the filesystem or problems with assign should be addressed after the hbase:meta
has been put in order. If hbase:meta is out of whack, the Master cannot make
proper placements when adopting orphan filesystem data or making region
assignments.
Other general principles to keep in mind include a Region can not be assigned
if it is in _CLOSING_ state (or the inverse, unassigned if in _OPENING_ state)
without first transitioning
[jira] [Commented] (HBASE-26780) HFileBlock.verifyOnDiskSizeMatchesHeader throw IOException: Passed in onDiskSizeWithHeader= A != B
[
https://issues.apache.org/jira/browse/HBASE-26780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762098#comment-17762098
]
xingyi li commented on HBASE-26780:
---
Hi [~ndimiduk]
There is a full stack, maybe it's useful for us~
{code:java}
// code placeholder
ERROR: java.io.IOException: Could not seek StoreFileScanner[HFileScanner for
reader reader=hdfs://${nn-host}/home/hbase/mobdir/data/default/$fileName,
compression=snappy, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false,
cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false,
firstKey=Optional[:bdata/1621438257591/Put/seqid=0],
lastKey=Optional[:bdata/1621438257593/Put/seqid=0], avgKeyLen=125,
avgValueLen=510647, entries=66236, length=30035708837, cur=null] to key
:bdata/1621438257593/Put vlen=76/seqid=7730409/
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:246)
at org.apache.hadoop.hbase.mob.MobFile.readCell(MobFile.java:95)
at
org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:396)
at
org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:348)
at
org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:73)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6635)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6799)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6572)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6549)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6536)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2520)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:138)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Caused by: java.io.IOException: Passed in onDiskSizeWithHeader=524603 !=
1904338447, offset=28910565258, fileContext=[usesHBaseChecksum=true,
checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE,
includesMvcc=true, includesTags=true, compressAlgo=SNAPPY, compressTags=false,
cryptoContext=[cipher=NONE keyHash=NONE], name=$fileName]
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.verifyOnDiskSizeMatchesHeader(HFileBlock.java:1688)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1793)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1607)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1649)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:340)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:979)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:889)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:327)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:228)
{code}
> HFileBlock.verifyOnDiskSizeMatchesHeader throw IOException: Passed in
> onDiskSizeWithHeader= A != B
> --
>
> Key: HBASE-26780
> URL: https://issues.apache.org/jira/browse/HBASE-26780
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 2.2.2
>Reporter: yuzhang
>Priority: Major
> Attachments: IOException.png
>
>
> When I scan a region, HBase throw IOException: Passed in
> onDiskSizeWithHeader= A != B
> The HFile mentioned Error message can be access normally.
> it recover by command – move region. I guess that onDiskSizeWithHeader of
> HFileBlock has been changed. And RS get the correct BlockHeader Info after
> region reopened.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-27978) [hbase-operator-tools] Add spotless in hbase-operator-tools pre-commit check
[ https://issues.apache.org/jira/browse/HBASE-27978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-27978: -- Fix Version/s: hbase-operator-tools-1.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Merged to master. Thanks [~nihaljain.cs]! > [hbase-operator-tools] Add spotless in hbase-operator-tools pre-commit check > > > Key: HBASE-27978 > URL: https://issues.apache.org/jira/browse/HBASE-27978 > Project: HBase > Issue Type: Sub-task > Components: build, hbase-operator-tools >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: hbase-operator-tools-1.3.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase-operator-tools] petersomogyi merged pull request #136: HBASE-27978 Add spotless in hbase-operator-tools pre-commit check
petersomogyi merged PR #136: URL: https://github.com/apache/hbase-operator-tools/pull/136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-operator-tools] NihalJain commented on pull request #136: HBASE-27978 Add spotless in hbase-operator-tools pre-commit check
NihalJain commented on PR #136: URL: https://github.com/apache/hbase-operator-tools/pull/136#issuecomment-1706558506 Gentle ping @Apache9 @petersomogyi @wchevreuil Please let me know, if this is good to go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-28063) Add documentation to HBase book
Andor Molnar created HBASE-28063: Summary: Add documentation to HBase book Key: HBASE-28063 URL: https://issues.apache.org/jira/browse/HBASE-28063 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Andor Molnar Assignee: Andor Molnar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #5376: HBASE-27999 Implement cache prefetch aware load balancer
Apache-HBase commented on PR #5376: URL: https://github.com/apache/hbase/pull/5376#issuecomment-1706426716 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-27389 Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 16s | HBASE-27389 passed | | +1 :green_heart: | compile | 1m 9s | HBASE-27389 passed | | +1 :green_heart: | shadedjars | 4m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | HBASE-27389 passed | | -0 :warning: | patch | 5m 42s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 20s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 4m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 6s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 9m 21s | hbase-balancer in the patch passed. | | -1 :x: | unit | 234m 55s | hbase-server in the patch failed. | | | | 272m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5376 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 79b84e469419 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-27389 / 5e2cc6363b | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/testReport/ | | Max. process+thread count | 4454 (vs. ulimit of 3) | | modules | C: hbase-common hbase-balancer hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5376: HBASE-27999 Implement cache prefetch aware load balancer
Apache-HBase commented on PR #5376: URL: https://github.com/apache/hbase/pull/5376#issuecomment-1706419568 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 23s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-27389 Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 39s | HBASE-27389 passed | | +1 :green_heart: | compile | 1m 19s | HBASE-27389 passed | | +1 :green_heart: | shadedjars | 4m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | HBASE-27389 passed | | -0 :warning: | patch | 5m 54s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 4m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 18s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 9m 7s | hbase-balancer in the patch passed. | | -1 :x: | unit | 230m 19s | hbase-server in the patch failed. | | | | 267m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5376 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 82401e44080a 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-27389 / 5e2cc6363b | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/testReport/ | | Max. process+thread count | 4672 (vs. ulimit of 3) | | modules | C: hbase-common hbase-balancer hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-28059) Use correct units in RegionLoad#getStoreUncompressedSizeMB()
[
https://issues.apache.org/jira/browse/HBASE-28059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-28059.
---
Fix Version/s: 2.6.0
2.4.18
2.5.6
Resolution: Fixed
Meged to branch-2, branch-2.5, and branch-2.4. Thanks for the contribution,
[~charlesconnell]!
> Use correct units in RegionLoad#getStoreUncompressedSizeMB()
>
>
> Key: HBASE-28059
> URL: https://issues.apache.org/jira/browse/HBASE-28059
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.5.5
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 2.5.6
>
>
> When I run a snippet of code like this:
> {code:java}
> Map regionLoadMap = admin
> .getClusterStatus()
> .getLoad(
> ServerName.parseServerName(
> "my-server.my-company.net,60020,1693513660506"
> )
> )
> .getRegionsLoad();
> for (byte[] startKey : regionLoadMap.keySet()) {
> RegionLoad regionLoad = regionLoadMap.get(startKey);
> LOG.info("Region {}: {}", Bytes.toStringBinary(startKey), regionLoad);
> } {code}
> I get logs like this:
> {noformat}
> Region , key>,1659484033280.2b89407a1223720344bed425bf3c29b0.: numberOfStores=1,
> numberOfStorefiles=3, storeRefCount=0, storefileUncompressedSizeMB=17998848,
> lastMajorCompactionTimestamp=1693211464712, storefileSizeMB=5895,
> compressionRatio=0.0003, memstoreSizeMB=1, readRequestsCount=118899553,
> writeRequestsCount=731192, rootIndexSizeKB=9, totalStaticIndexSizeKB=10413,
> totalStaticBloomSizeKB=6592, totalCompactingKVs=0, currentCompactedKVs=0,
> compactionProgressPct=NaN, completeSequenceId=78093096,
> dataLocality=1.0{noformat}
> The {{storefileUncompressedSizeMB}} is vastly larger than the
> {{{}storefileSizeMB{}}}, so much more that it's not believable. I checked the
> store files in question in this instance. Adding up the uncompressed size
> reported in the HFile trailers sums to 5895 MiB, exactly 1024 times less than
> the reported 17998848.
> The reason for the misreporting is that
> {{RegionLoad#getStoreUncompressedSizeMB()}} converts the value from a
> {{Size}} object wrong.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] petersomogyi merged pull request #5382: HBASE-28059 Use correct units in RegionLoad#getStoreUncompressedSizeMB()
petersomogyi merged PR #5382: URL: https://github.com/apache/hbase/pull/5382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28038) Add TLS settings to ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-28038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-28038:
--
Fix Version/s: 2.6.0
2.4.18
2.5.6
3.0.0-beta-1
4.0.0-alpha-1
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to all active branches. Thanks [~andor]!
> Add TLS settings to ZooKeeper client
>
>
> Key: HBASE-28038
> URL: https://issues.apache.org/jira/browse/HBASE-28038
> Project: HBase
> Issue Type: Improvement
> Components: Zookeeper
>Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.5
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Labels: ssl, tls, zookeeper
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 4.0.0-alpha-1
>
>
> ZooKeeper supports TLS connection from its clients. Currently the only way to
> set up HBase for this is to add the following Java properties to the HBase
> process:
> {noformat}
>
> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
> -Dzookeeper.client.secure=true
> -Dzookeeper.ssl.keyStore.location=/path/to/keystore.jks
> -Dzookeeper.ssl.keyStore.password=password
> -Dzookeeper.ssl.trustStore.location=/path/to/truststore.jks
> -Dzookeeper.ssl.trustStore.password=password
> {noformat}
> KeyStore is only needed if ZooKeeper server wants client certificate to be
> provided.
> I'd like to add these options to hbase-site.xml in the following way:
> {noformat}
> hbase.zookeeper.property.clientCnxnSocket
> hbase.zookeeper.property.client.secure
> hbase.zookeeper.property.ssl.keyStore.location
> hbase.zookeeper.property.ssl.keyStore.password or
> hbase.zookeeper.property.ssl.keyStore.passwordPath
> ...{noformat}
> It will follow the way that we already do for ZooKeeper clientPort and quorum
> settings. ("hbase.zookeeper.property.clientPort", "hbase.zookeeper.quorum")
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] ndimiduk commented on a diff in pull request #5352: HBASE-26780 HFileBlock.verifyOnDiskSizeMatchesHeader throw IOException: Passed in onDiskSizeWithHeader= A != 33
ndimiduk commented on code in PR #5352: URL: https://github.com/apache/hbase/pull/5352#discussion_r1315650212 ## hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java: ## @@ -1729,7 +1729,31 @@ protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset, } onDiskSizeWithHeader = getOnDiskSizeWithHeader(headerBuf, checksumSupport); } - int preReadHeaderSize = headerBuf == null ? 0 : hdrSize; + int preReadHeaderSize = 0; + if ( +headerBuf == null + || headerBuf.getInt(Header.ON_DISK_SIZE_WITHOUT_HEADER_INDEX) != onDiskSizeWithHeader Review Comment: The value read from `onDiskSizeWithoutHeader` should never equal `onDiskSizeWithHeader`, right? > 1: onDiskSizeWithoutHeader: Compressed -- a.k.a 'on disk' -- block size, excluding header, but including tailing checksum bytes (4 bytes) While `onDiskSizeWithHeader` should be the entire serialized block: header + serialized data + checksums. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26780) HFileBlock.verifyOnDiskSizeMatchesHeader throw IOException: Passed in onDiskSizeWithHeader= A != B
[ https://issues.apache.org/jira/browse/HBASE-26780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762007#comment-17762007 ] Nick Dimiduk commented on HBASE-26780: -- Heya [~yyZhang]. We've been seeing a similar symptom on several of our clusters. Our current analysis is that this is caused by HDFS data block corruption and holes in HBase's ability to recognise and request repair of the block. The file in question later gets compacted away, so I assume that a different code path makes correct use of HDFS checksums to access it, but it's difficult to "catch in the act". Can you share a full stack trace from HBase? I'll see if I can get permission to the same. Thanks. [~cribbee] This is interesting -- we have not correlated with a newly started data node. Indeed, our clumsy solution is to take the data node offline and force HDFS to re-replicate the block. We are not using erasure encoding. [~xytss123] Are you certain that the underlying block is not corrupted? How to you check it before HDFS can automatically repair? When you read the file again, how do you force your HDFS client to read that block from the same datanode as when the missmatching data was encountered? FYI, we're running with a build of 2.5.2 + some back-ported patches. > HFileBlock.verifyOnDiskSizeMatchesHeader throw IOException: Passed in > onDiskSizeWithHeader= A != B > -- > > Key: HBASE-26780 > URL: https://issues.apache.org/jira/browse/HBASE-26780 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 2.2.2 >Reporter: yuzhang >Priority: Major > Attachments: IOException.png > > > When I scan a region, HBase throw IOException: Passed in > onDiskSizeWithHeader= A != B > The HFile mentioned Error message can be access normally. > it recover by command – move region. I guess that onDiskSizeWithHeader of > HFileBlock has been changed. And RS get the correct BlockHeader Info after > region reopened. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase] petersomogyi merged pull request #5370: HBASE-28038 Add TLS settings to ZooKeeper client
petersomogyi merged PR #5370: URL: https://github.com/apache/hbase/pull/5370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anmolnar commented on a diff in pull request #5370: HBASE-28038 Add TLS settings to ZooKeeper client
anmolnar commented on code in PR #5370:
URL: https://github.com/apache/hbase/pull/5370#discussion_r1315565341
##
hbase-common/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java:
##
@@ -330,4 +341,27 @@ public static String
getClientZKQuorumServersString(Configuration conf) {
final String[] serverHosts = StringUtils.getStrings(clientQuromServers);
return buildZKQuorumServerString(serverHosts, clientZkClientPort);
}
+
+ private static void setZooKeeperClientSystemProperties(String prefix,
Configuration conf) {
Review Comment:
@wchevreuil @Apache9 Sorry, I couldn't look here over the weekend. Since the
patch already has everything to submit I wouldn't make any change for now.
There's chance to have more ZK properties to add here which are not TLS
related, so hopefully not a big deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] anmolnar commented on a diff in pull request #5370: HBASE-28038 Add TLS settings to ZooKeeper client
anmolnar commented on code in PR #5370:
URL: https://github.com/apache/hbase/pull/5370#discussion_r1315565341
##
hbase-common/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java:
##
@@ -330,4 +341,27 @@ public static String
getClientZKQuorumServersString(Configuration conf) {
final String[] serverHosts = StringUtils.getStrings(clientQuromServers);
return buildZKQuorumServerString(serverHosts, clientZkClientPort);
}
+
+ private static void setZooKeeperClientSystemProperties(String prefix,
Configuration conf) {
Review Comment:
@wchevreuil @Apache9 Sorry, I couldn't look here over the weekend. Since the
patch already has everything to submit I wouldn't make any change it for now.
There's chance to have more ZK properties to add here which are not TLS
related, so hopefully not a big deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5379: HBASE-28055 Performance improvement for scan over several stores.
Apache-HBase commented on PR #5379: URL: https://github.com/apache/hbase/pull/5379#issuecomment-1706150887 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 58s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 5m 43s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 33s | the patch passed | | +1 :green_heart: | compile | 0m 35s | the patch passed | | +1 :green_heart: | javac | 0m 35s | the patch passed | | +1 :green_heart: | shadedjars | 4m 53s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 225m 25s | hbase-server in the patch passed. | | | | 249m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5379/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5379 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 519a685d4e0d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 121c8e17ec | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5379/3/testReport/ | | Max. process+thread count | 4596 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5379/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5379: HBASE-28055 Performance improvement for scan over several stores.
Apache-HBase commented on PR #5379: URL: https://github.com/apache/hbase/pull/5379#issuecomment-1706132333 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 3s | master passed | | +1 :green_heart: | compile | 0m 44s | master passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 211m 30s | hbase-server in the patch passed. | | | | 235m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5379/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5379 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 816dac36fbb2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 121c8e17ec | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5379/3/testReport/ | | Max. process+thread count | 4653 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5379/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5376: HBASE-27999 Implement cache prefetch aware load balancer
Apache-HBase commented on PR #5376: URL: https://github.com/apache/hbase/pull/5376#issuecomment-1706084640 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 53s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-27389 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 3s | HBASE-27389 passed | | +1 :green_heart: | compile | 3m 20s | HBASE-27389 passed | | +1 :green_heart: | checkstyle | 0m 53s | HBASE-27389 passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 20s | HBASE-27389 passed | | -0 :warning: | patch | 1m 44s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 42s | the patch passed | | +1 :green_heart: | compile | 3m 21s | the patch passed | | -0 :warning: | javac | 0m 18s | hbase-balancer generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) | | -0 :warning: | checkstyle | 0m 7s | hbase-balancer: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 25s | Patch does not cause any errors with Hadoop 3.2.4 3.3.5. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 41m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5376 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux f8b64df3ead8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-27389 / 5e2cc6363b | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | Max. process+thread count | 76 (vs. ulimit of 3) | | modules | C: hbase-common hbase-balancer hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5376/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
