[jira] [Work started] (HBASE-25993) Make excluded SSL cipher suites configurable for all Web UIs
[
https://issues.apache.org/jira/browse/HBASE-25993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25993 started by Mate Szalay-Beko.
> Make excluded SSL cipher suites configurable for all Web UIs
>
>
> Key: HBASE-25993
> URL: https://issues.apache.org/jira/browse/HBASE-25993
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> When starting a jetty http server, one can explicitly exclude certain
> (unsecure) SSL cipher suites. This can be especially important, when the
> HBase cluster needs to be compliant with security regulations (e.g. FIPS).
> Currently it is possible to set the excluded ciphers for the ThriftServer
> ("hbase.thrift.ssl.exclude.cipher.suites") or for the RestServer
> ("hbase.rest.ssl.exclude.cipher.suites"), but one can not configure it for
> the regular InfoServer started by e.g. the master or region servers.
> In this commit I want to introduce a new configuration
> "ssl.server.exclude.cipher.list" to configure the excluded cipher suites for
> the http server started by the InfoServer. This parameter has the same name
> and will work in the same way, as it was already implemented in hadoop (e.g.
> for hdfs/yarn). See: HADOOP-12668, HADOOP-14341
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25993) Make excluded SSL cipher suites configurable for all Web UIs
[
https://issues.apache.org/jira/browse/HBASE-25993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25993:
-
Summary: Make excluded SSL cipher suites configurable for all Web UIs
(was: Make excluded cipher suites configurable for all Web UIs)
> Make excluded SSL cipher suites configurable for all Web UIs
>
>
> Key: HBASE-25993
> URL: https://issues.apache.org/jira/browse/HBASE-25993
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> When starting a jetty http server, one can explicitly exclude certain
> (unsecure) SSL cipher suites. This can be especially important, when the
> HBase cluster needs to be compliant with security regulations (e.g. FIPS).
> Currently it is possible to set the excluded ciphers for the ThriftServer
> ("hbase.thrift.ssl.exclude.cipher.suites") or for the RestServer
> ("hbase.rest.ssl.exclude.cipher.suites"), but one can not configure it for
> the regular InfoServer started by e.g. the master or region servers.
> In this commit I want to introduce a new configuration
> "ssl.server.exclude.cipher.list" to configure the excluded cipher suites for
> the http server started by the InfoServer. This parameter has the same name
> and will work in the same way, as it was already implemented in hadoop (e.g.
> for hdfs/yarn). See: HADOOP-12668, HADOOP-14341
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25993) Make excluded cipher suites configurable for all Web UIs
Mate Szalay-Beko created HBASE-25993:
Summary: Make excluded cipher suites configurable for all Web UIs
Key: HBASE-25993
URL: https://issues.apache.org/jira/browse/HBASE-25993
Project: HBase
Issue Type: Improvement
Affects Versions: 2.4.4, 2.3.5, 2.2.7, 3.0.0-alpha-1, 2.5.0
Reporter: Mate Szalay-Beko
Assignee: Mate Szalay-Beko
When starting a jetty http server, one can explicitly exclude certain
(unsecure) SSL cipher suites. This can be especially important, when the HBase
cluster needs to be compliant with security regulations (e.g. FIPS).
Currently it is possible to set the excluded ciphers for the ThriftServer
("hbase.thrift.ssl.exclude.cipher.suites") or for the RestServer
("hbase.rest.ssl.exclude.cipher.suites"), but one can not configure it for the
regular InfoServer started by e.g. the master or region servers.
In this commit I want to introduce a new configuration
"ssl.server.exclude.cipher.list" to configure the excluded cipher suites for
the http server started by the InfoServer. This parameter has the same name and
will work in the same way, as it was already implemented in hadoop (e.g. for
hdfs/yarn). See: HADOOP-12668, HADOOP-14341
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25987) Make SSL keystore type configurable for HBase ThriftServer
[
https://issues.apache.org/jira/browse/HBASE-25987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25987:
-
Description:
We get the following exception, when trying to start Hbase Thrift Server in
http mode (hbase.regionserver.thrift.http=true) and use non default (not "jks")
keystore type:
{noformat}
2021-06-08 07:40:10,275 ERROR org.apache.hadoop.hbase.thrift.ThriftServer:
Cannot run ThriftServer
java.io.IOException: Invalid keystore format
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:663)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1197)
at
org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:321)
at
org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:243)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.apache.hadoop.hbase.thrift.ThriftServer$2.run(ThriftServer.java:861)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at
org.apache.hadoop.hbase.thrift.ThriftServer.run(ThriftServer.java:855)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:882){noformat}
This problem appeared after we applied HBASE-25930 to our local HBase version.
It looks, we never had a parameter to specify the keystore type for thrift http
server. Before HBASE-25930, the keystore type used by the thrift http server
was accidentally defined based on the InfoServer (web ui) configuration of
"ssl.server.keystore.type". Before HBASE-25930, the InfoServer was started
first and it set the keystore type in the global keystore manager, which
setting propagated to the thrift http server too, without any override. In
HBASE-25930 the startup order changed, and the thrift http server configuration
happens before the InfoServer start, so we lack this accidental configuration
change now.
Given that we have independent keystore file path / password parameters already
for the thrift http server, the proper solution is to create a new parameter
also for the keystore type of the thrift http server:
*hbase.thrift.ssl.keystore.type* (defaulting to "jks").
was:
We get the following exception, when trying to start Hbase Thrift Server in
http mode (
hbase.regionserver.thrift.http=true) and use non default (not "jks") keystore
type:
{noformat}
2021-06-08 07:40:10,275 ERROR org.apache.hadoop.hbase.thrift.ThriftServer:
Cannot run ThriftServer
java.io.IOException: Invalid keystore format
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:663)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
[jira] [Work started] (HBASE-25987) Make SSL keystore type configurable for HBase ThriftServer
[
https://issues.apache.org/jira/browse/HBASE-25987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25987 started by Mate Szalay-Beko.
> Make SSL keystore type configurable for HBase ThriftServer
> --
>
> Key: HBASE-25987
> URL: https://issues.apache.org/jira/browse/HBASE-25987
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Affects Versions: 2.4.4
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> We get the following exception, when trying to start Hbase Thrift Server in
> http mode (
> hbase.regionserver.thrift.http=true) and use non default (not "jks") keystore
> type:
>
> {noformat}
> 2021-06-08 07:40:10,275 ERROR org.apache.hadoop.hbase.thrift.ThriftServer:
> Cannot run ThriftServer
> java.io.IOException: Invalid keystore format
> at
> sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:663)
> at
> sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
> at
> sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
> at
> sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
> at java.security.KeyStore.load(KeyStore.java:1445)
> at
> org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
> at
> org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1197)
> at
> org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:321)
> at
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:243)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
> at
> org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
> at
> org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.eclipse.jetty.server.Server.doStart(Server.java:401)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.apache.hadoop.hbase.thrift.ThriftServer$2.run(ThriftServer.java:861)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hadoop.hbase.thrift.ThriftServer.run(ThriftServer.java:855)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:882){noformat}
> This problem appeared after we applied HBASE-25930 to our local HBase
> version. It looks, we never had a parameter to specify the keystore type for
> thrift http server. Before HBASE-25930, the keystore type used by the thrift
> http server was accidentally defined based on the InfoServer (web ui)
> configuration of "ssl.server.keystore.type". Before HBASE-25930, the
> InfoServer was started first and it set the keystore type in the global
> keystore manager, which setting propagated to the thrift http server too,
> without any override. In HBASE-25930 the startup order changed, and the
> thrift http server configuration happens before the InfoServer start, so we
> lack this accidental configuration change now.
> The proper solution is to create a new parameter to setup keystore type for
> the thrift http server: hbase.thrift.ssl.keystore.type (defaulting to "jks")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25987) Make SSL keystore type configurable for HBase ThriftServer
Mate Szalay-Beko created HBASE-25987:
Summary: Make SSL keystore type configurable for HBase ThriftServer
Key: HBASE-25987
URL: https://issues.apache.org/jira/browse/HBASE-25987
Project: HBase
Issue Type: Improvement
Components: Thrift
Affects Versions: 2.4.4
Reporter: Mate Szalay-Beko
Assignee: Mate Szalay-Beko
We get the following exception, when trying to start Hbase Thrift Server in
http mode (
hbase.regionserver.thrift.http=true) and use non default (not "jks") keystore
type:
{noformat}
2021-06-08 07:40:10,275 ERROR org.apache.hadoop.hbase.thrift.ThriftServer:
Cannot run ThriftServer
java.io.IOException: Invalid keystore format
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:663)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1197)
at
org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:321)
at
org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:243)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.apache.hadoop.hbase.thrift.ThriftServer$2.run(ThriftServer.java:861)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at
org.apache.hadoop.hbase.thrift.ThriftServer.run(ThriftServer.java:855)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:882){noformat}
This problem appeared after we applied HBASE-25930 to our local HBase version.
It looks, we never had a parameter to specify the keystore type for thrift http
server. Before HBASE-25930, the keystore type used by the thrift http server
was accidentally defined based on the InfoServer (web ui) configuration of
"ssl.server.keystore.type". Before HBASE-25930, the InfoServer was started
first and it set the keystore type in the global keystore manager, which
setting propagated to the thrift http server too, without any override. In
HBASE-25930 the startup order changed, and the thrift http server configuration
happens before the InfoServer start, so we lack this accidental configuration
change now.
The proper solution is to create a new parameter to setup keystore type for the
thrift http server: hbase.thrift.ssl.keystore.type (defaulting to "jks")
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-25606) Backporting HBASE-24885 to branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-25606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25606 started by Mate Szalay-Beko. > Backporting HBASE-24885 to branch-2.2 > - > > Key: HBASE-25606 > URL: https://issues.apache.org/jira/browse/HBASE-25606 > Project: HBase > Issue Type: Bug > Components: hbck2, Region Assignment >Affects Versions: 2.2.6 >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > On a 2.2 HBase cluster we just hit the very same "HBCK2 assigns" issue > already fixed in HBASE-24885. This improvement would be very useful on > branch-2.2 too. I'll submit a PR soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25607) [HBCK2] New check to prevent assign of already opened regions
[ https://issues.apache.org/jira/browse/HBASE-25607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25607: - Summary: [HBCK2] New check to prevent assign of already opened regions (was: New check in hbck2 to prevent assign of already opened regions) > [HBCK2] New check to prevent assign of already opened regions > - > > Key: HBASE-25607 > URL: https://issues.apache.org/jira/browse/HBASE-25607 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Affects Versions: hbase-operator-tools-1.1.0 >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > Currently, if we don't have HBASE-24885 on the cluster and we use HBCK2 > assigns command to assign an already OPEN region (even without the --override > flag), then it can cause a stuck Region-In-Transition and a wrong OPENING > state in the meta (or even a double assignment of the same region on two RS). > This happens because the HBase master RPC endpoint doesn't check if the > region is already opened or not. The issue was fixed in HBASE-24885, but it > still remain a problem in previous HBase versions (2.2 and also 2.3.0 and > 2.3.1. ) > We could add some mechanism into HBCK2 to do a check and don't execute the > assign command if the region is OPEN in the meta. (we could also add a > 'force' flag to bypass this check). > The new check should run only if HBASE-24885 is not present yet on the > cluster (we could validate the cluster version) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25607) New check in hbck2 to prevent assign of already opened regions
Mate Szalay-Beko created HBASE-25607: Summary: New check in hbck2 to prevent assign of already opened regions Key: HBASE-25607 URL: https://issues.apache.org/jira/browse/HBASE-25607 Project: HBase Issue Type: Improvement Components: hbase-operator-tools, hbck2 Affects Versions: hbase-operator-tools-1.1.0 Reporter: Mate Szalay-Beko Assignee: Mate Szalay-Beko Currently, if we don't have HBASE-24885 on the cluster and we use HBCK2 assigns command to assign an already OPEN region (even without the --override flag), then it can cause a stuck Region-In-Transition and a wrong OPENING state in the meta (or even a double assignment of the same region on two RS). This happens because the HBase master RPC endpoint doesn't check if the region is already opened or not. The issue was fixed in HBASE-24885, but it still remain a problem in previous HBase versions (2.2 and also 2.3.0 and 2.3.1. ) We could add some mechanism into HBCK2 to do a check and don't execute the assign command if the region is OPEN in the meta. (we could also add a 'force' flag to bypass this check). The new check should run only if HBASE-24885 is not present yet on the cluster (we could validate the cluster version) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25606) Backporting HBASE-24885 to branch-2.2
Mate Szalay-Beko created HBASE-25606: Summary: Backporting HBASE-24885 to branch-2.2 Key: HBASE-25606 URL: https://issues.apache.org/jira/browse/HBASE-25606 Project: HBase Issue Type: Bug Components: hbck2, Region Assignment Affects Versions: 2.2.6 Reporter: Mate Szalay-Beko Assignee: Mate Szalay-Beko On a 2.2 HBase cluster we just hit the very same "HBCK2 assigns" issue already fixed in HBASE-24885. This improvement would be very useful on branch-2.2 too. I'll submit a PR soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24815) hbase-connectors mvn install error
[
https://issues.apache.org/jira/browse/HBASE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-24815:
-
Fix Version/s: connector-1.1.0
> hbase-connectors mvn install error
> --
>
> Key: HBASE-24815
> URL: https://issues.apache.org/jira/browse/HBASE-24815
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Reporter: leookok
>Assignee: Mate Szalay-Beko
>Priority: Blocker
> Fix For: connector-1.1.0
>
>
> *when maven command-line*
> mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> will return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216:
> overloaded method value addTaskCompletionListener with alternatives:
> (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext
> (listener:
> org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
> does not take type parameters
> {color:red}[ERROR] {color}one error found
> *but use the spark.version=2.4.0 is ok*
> mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
>
> *other try*
> mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439:
> object SparkHadoopUtil in package deploy cannot be accessed in package
> org.apache.spark.deploy
> [ERROR] [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487:
> not found: value SparkHadoopUtil
> {color:red}[ERROR]{color} two errors found
> go to the [spark
> @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
> define SparkHadoopUtil to private[spark]
> {code:java}
> private[spark] class SparkHadoopUtil extends Logging {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24815) hbase-connectors mvn install error
[
https://issues.apache.org/jira/browse/HBASE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko resolved HBASE-24815.
--
Resolution: Duplicate
> hbase-connectors mvn install error
> --
>
> Key: HBASE-24815
> URL: https://issues.apache.org/jira/browse/HBASE-24815
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Reporter: leookok
>Assignee: Mate Szalay-Beko
>Priority: Blocker
>
> *when maven command-line*
> mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> will return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216:
> overloaded method value addTaskCompletionListener with alternatives:
> (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext
> (listener:
> org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
> does not take type parameters
> {color:red}[ERROR] {color}one error found
> *but use the spark.version=2.4.0 is ok*
> mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
>
> *other try*
> mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439:
> object SparkHadoopUtil in package deploy cannot be accessed in package
> org.apache.spark.deploy
> [ERROR] [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487:
> not found: value SparkHadoopUtil
> {color:red}[ERROR]{color} two errors found
> go to the [spark
> @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
> define SparkHadoopUtil to private[spark]
> {code:java}
> private[spark] class SparkHadoopUtil extends Logging {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24815) hbase-connectors mvn install error
[
https://issues.apache.org/jira/browse/HBASE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276329#comment-17276329
]
Mate Szalay-Beko commented on HBASE-24815:
--
I tested the patch available on HBASE-25326, and it solves the issue. I'm going
to close this HBASE-24815 as a duplicate in the hope that the PR in HBASE-25326
gets merged soon.
> hbase-connectors mvn install error
> --
>
> Key: HBASE-24815
> URL: https://issues.apache.org/jira/browse/HBASE-24815
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Reporter: leookok
>Assignee: Mate Szalay-Beko
>Priority: Blocker
>
> *when maven command-line*
> mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> will return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216:
> overloaded method value addTaskCompletionListener with alternatives:
> (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext
> (listener:
> org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
> does not take type parameters
> {color:red}[ERROR] {color}one error found
> *but use the spark.version=2.4.0 is ok*
> mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
>
> *other try*
> mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439:
> object SparkHadoopUtil in package deploy cannot be accessed in package
> org.apache.spark.deploy
> [ERROR] [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487:
> not found: value SparkHadoopUtil
> {color:red}[ERROR]{color} two errors found
> go to the [spark
> @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
> define SparkHadoopUtil to private[spark]
> {code:java}
> private[spark] class SparkHadoopUtil extends Logging {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25326) Allow hbase-connector to be used with Apache Spark 3.0
[ https://issues.apache.org/jira/browse/HBASE-25326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko reassigned HBASE-25326: Assignee: (was: Mate Szalay-Beko) > Allow hbase-connector to be used with Apache Spark 3.0 > --- > > Key: HBASE-25326 > URL: https://issues.apache.org/jira/browse/HBASE-25326 > Project: HBase > Issue Type: Improvement >Reporter: Luca Canali >Priority: Minor > > Currently hbase-spark connector only works with Spark 2.x. > Apache Spark 3.0 has been relesead in June 2020. > This addresses the changes needed to run the connector with Spark 3.0 and > also to be able to compile the connector using Spark 3.0 as a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25326) Allow hbase-connector to be used with Apache Spark 3.0
[ https://issues.apache.org/jira/browse/HBASE-25326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko reopened HBASE-25326: -- > Allow hbase-connector to be used with Apache Spark 3.0 > --- > > Key: HBASE-25326 > URL: https://issues.apache.org/jira/browse/HBASE-25326 > Project: HBase > Issue Type: Improvement >Reporter: Luca Canali >Assignee: Mate Szalay-Beko >Priority: Minor > > Currently hbase-spark connector only works with Spark 2.x. > Apache Spark 3.0 has been relesead in June 2020. > This addresses the changes needed to run the connector with Spark 3.0 and > also to be able to compile the connector using Spark 3.0 as a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25326) Allow hbase-connector to be used with Apache Spark 3.0
[ https://issues.apache.org/jira/browse/HBASE-25326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko resolved HBASE-25326. -- Assignee: Mate Szalay-Beko Resolution: Fixed > Allow hbase-connector to be used with Apache Spark 3.0 > --- > > Key: HBASE-25326 > URL: https://issues.apache.org/jira/browse/HBASE-25326 > Project: HBase > Issue Type: Improvement >Reporter: Luca Canali >Assignee: Mate Szalay-Beko >Priority: Minor > > Currently hbase-spark connector only works with Spark 2.x. > Apache Spark 3.0 has been relesead in June 2020. > This addresses the changes needed to run the connector with Spark 3.0 and > also to be able to compile the connector using Spark 3.0 as a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24815) hbase-connectors mvn install error
[
https://issues.apache.org/jira/browse/HBASE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273518#comment-17273518
]
Mate Szalay-Beko commented on HBASE-24815:
--
I see there is already an other ticket with a pull request to fix this:
HBASE-25326
> hbase-connectors mvn install error
> --
>
> Key: HBASE-24815
> URL: https://issues.apache.org/jira/browse/HBASE-24815
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Reporter: leookok
>Assignee: Mate Szalay-Beko
>Priority: Blocker
>
> *when maven command-line*
> mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> will return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216:
> overloaded method value addTaskCompletionListener with alternatives:
> (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext
> (listener:
> org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
> does not take type parameters
> {color:red}[ERROR] {color}one error found
> *but use the spark.version=2.4.0 is ok*
> mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
>
> *other try*
> mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439:
> object SparkHadoopUtil in package deploy cannot be accessed in package
> org.apache.spark.deploy
> [ERROR] [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487:
> not found: value SparkHadoopUtil
> {color:red}[ERROR]{color} two errors found
> go to the [spark
> @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
> define SparkHadoopUtil to private[spark]
> {code:java}
> private[spark] class SparkHadoopUtil extends Logging {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-24815) hbase-connectors mvn install error
[
https://issues.apache.org/jira/browse/HBASE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-24815 started by Mate Szalay-Beko.
> hbase-connectors mvn install error
> --
>
> Key: HBASE-24815
> URL: https://issues.apache.org/jira/browse/HBASE-24815
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Reporter: leookok
>Assignee: Mate Szalay-Beko
>Priority: Blocker
>
> *when maven command-line*
> mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> will return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216:
> overloaded method value addTaskCompletionListener with alternatives:
> (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext
> (listener:
> org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
> does not take type parameters
> {color:red}[ERROR] {color}one error found
> *but use the spark.version=2.4.0 is ok*
> mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
>
> *other try*
> mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439:
> object SparkHadoopUtil in package deploy cannot be accessed in package
> org.apache.spark.deploy
> [ERROR] [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487:
> not found: value SparkHadoopUtil
> {color:red}[ERROR]{color} two errors found
> go to the [spark
> @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
> define SparkHadoopUtil to private[spark]
> {code:java}
> private[spark] class SparkHadoopUtil extends Logging {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24815) hbase-connectors mvn install error
[
https://issues.apache.org/jira/browse/HBASE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273496#comment-17273496
]
Mate Szalay-Beko commented on HBASE-24815:
--
I just faced this build issue when trying to build hbase-connector with spark3
and scala 2.12
I think the issue is caused by https://issues.apache.org/jira/browse/SPARK-26043
I'll try to fix this by copying these now-private utility functions from the
spark class. That way we could compile both with spark2 and spark3.
> hbase-connectors mvn install error
> --
>
> Key: HBASE-24815
> URL: https://issues.apache.org/jira/browse/HBASE-24815
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Reporter: leookok
>Assignee: Mate Szalay-Beko
>Priority: Blocker
>
> *when maven command-line*
> mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> will return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216:
> overloaded method value addTaskCompletionListener with alternatives:
> (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext
> (listener:
> org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
> does not take type parameters
> {color:red}[ERROR] {color}one error found
> *but use the spark.version=2.4.0 is ok*
> mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
>
> *other try*
> mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439:
> object SparkHadoopUtil in package deploy cannot be accessed in package
> org.apache.spark.deploy
> [ERROR] [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487:
> not found: value SparkHadoopUtil
> {color:red}[ERROR]{color} two errors found
> go to the [spark
> @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
> define SparkHadoopUtil to private[spark]
> {code:java}
> private[spark] class SparkHadoopUtil extends Logging {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24815) hbase-connectors mvn install error
[
https://issues.apache.org/jira/browse/HBASE-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko reassigned HBASE-24815:
Assignee: Mate Szalay-Beko
> hbase-connectors mvn install error
> --
>
> Key: HBASE-24815
> URL: https://issues.apache.org/jira/browse/HBASE-24815
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Reporter: leookok
>Assignee: Mate Szalay-Beko
>Priority: Blocker
>
> *when maven command-line*
> mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> will return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\datasources\HBaseTableScanRDD.scala:216:
> overloaded method value addTaskCompletionListener with alternatives:
> (f: org.apache.spark.TaskContext => Unit)org.apache.spark.TaskContext
> (listener:
> org.apache.spark.util.TaskCompletionListener)org.apache.spark.TaskContext
> does not take type parameters
> {color:red}[ERROR] {color}one error found
> *but use the spark.version=2.4.0 is ok*
> mvn -Dspark.version=2.4.0 -Dscala.version=2.11.7 -Dscala.binary.version=2.11
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
>
> *other try*
> mvn -Dspark.version=3.0.0 -Dscala.version=2.12.12 -Dscala.binary.version=2.12
> -Dcheckstyle.skip=true -Dmaven.test.skip=true clean install
> return error
> {color:red}[ERROR]{color} [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:439:
> object SparkHadoopUtil in package deploy cannot be accessed in package
> org.apache.spark.deploy
> [ERROR] [Error]
> F:\hbase-connectors\spark\hbase-spark\src\main\scala\org\apache\hadoop\hbase\spark\HBaseContext.scala:487:
> not found: value SparkHadoopUtil
> {color:red}[ERROR]{color} two errors found
> go to the [spark
> @github|https://github.com/apache/spark/blob/e1ea806b3075d279b5f08a29fe4c1ad6d3c4191a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala]
> define SparkHadoopUtil to private[spark]
> {code:java}
> private[spark] class SparkHadoopUtil extends Logging {}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25430) Refresh in-memory table-info cache in HBase
Mate Szalay-Beko created HBASE-25430: Summary: Refresh in-memory table-info cache in HBase Key: HBASE-25430 URL: https://issues.apache.org/jira/browse/HBASE-25430 Project: HBase Issue Type: Bug Components: hbase-operator-tools, hbck2 Reporter: Mate Szalay-Beko In HBASE-25297 we intorduced the feature in hbck2 to fix orphan tables by generating the missing .tableinfo file. If the table descriptor can not be found in the tableinfo cache of the master, then hbck2 will generate a new file with default table configurations. In this case hbck2 could re-trigger a refresh of the in-memory table info cache of the master node. We would need to add a new master RPC API to refresh the cache for a given table. Alternatively we should also test if a table disable / enable would do the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25430) Refresh in-memory table-info cache in HBase
[ https://issues.apache.org/jira/browse/HBASE-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25430: - Issue Type: Improvement (was: Bug) > Refresh in-memory table-info cache in HBase > --- > > Key: HBASE-25430 > URL: https://issues.apache.org/jira/browse/HBASE-25430 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Reporter: Mate Szalay-Beko >Priority: Major > > In HBASE-25297 we intorduced the feature in hbck2 to fix orphan tables by > generating the missing .tableinfo file. If the table descriptor can not be > found in the tableinfo cache of the master, then hbck2 will generate a new > file with default table configurations. In this case hbck2 could re-trigger a > refresh of the in-memory table info cache of the master node. > We would need to add a new master RPC API to refresh the cache for a given > table. Alternatively we should also test if a table disable / enable would do > the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25297) [HBCK2] Regenerate missing table descriptors by hbck2
[
https://issues.apache.org/jira/browse/HBASE-25297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252868#comment-17252868
]
Mate Szalay-Beko commented on HBASE-25297:
--
Thanks for the review and the merge [~wchevreuil]!
I created a new Jira task about the improvements you mentioned on the PR:
https://issues.apache.org/jira/browse/HBASE-25430
> [HBCK2] Regenerate missing table descriptors by hbck2
> -
>
> Key: HBASE-25297
> URL: https://issues.apache.org/jira/browse/HBASE-25297
> Project: HBase
> Issue Type: Improvement
> Components: hbase-operator-tools, hbck2
>Affects Versions: hbase-operator-tools-1.0.0
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: hbase-operator-tools-1.1.0
>
>
> We just have been in a situation, when we saw many ServerCrashProcedure to be
> blocked, because of the AssignProcedure is blocked because of the missing
> .tabledesc files.
> {code:java}
> java.io.IOException: Missing table descriptor for
> dd0676f57bdbff5d04ab735b7daf5e9b
> {code}
>
> In our case it was OK for us to get rid of these tables and we were able to
> use {{setRegionState}} to move all these regions to {{FAILED_OPEN}} state,
> then disable and drop the tables. But this took a lot of time and we might
> not always have the option to drop these tables.
> HBCK 1 had a functionality (\{{fixTableOrphans}}) to regenerate the table
> descriptors "according to the memory cache or hdfs table directory
> structure". We should investigate if we can implement something like this
> with hbck2. We might not always be able to reproduce the table
> configurations, but we can probably make a "basic" one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25297) [HBCK2] Regenerate missing table descriptors by hbck2
[
https://issues.apache.org/jira/browse/HBASE-25297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252868#comment-17252868
]
Mate Szalay-Beko edited comment on HBASE-25297 at 12/21/20, 1:35 PM:
-
Thanks for the review and the merge [~wchevreuil]!
I created a new Jira task about the improvements you mentioned on the PR:
HBASE-25430
was (Author: symat):
Thanks for the review and the merge [~wchevreuil]!
I created a new Jira task about the improvements you mentioned on the PR:
https://issues.apache.org/jira/browse/HBASE-25430
> [HBCK2] Regenerate missing table descriptors by hbck2
> -
>
> Key: HBASE-25297
> URL: https://issues.apache.org/jira/browse/HBASE-25297
> Project: HBase
> Issue Type: Improvement
> Components: hbase-operator-tools, hbck2
>Affects Versions: hbase-operator-tools-1.0.0
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: hbase-operator-tools-1.1.0
>
>
> We just have been in a situation, when we saw many ServerCrashProcedure to be
> blocked, because of the AssignProcedure is blocked because of the missing
> .tabledesc files.
> {code:java}
> java.io.IOException: Missing table descriptor for
> dd0676f57bdbff5d04ab735b7daf5e9b
> {code}
>
> In our case it was OK for us to get rid of these tables and we were able to
> use {{setRegionState}} to move all these regions to {{FAILED_OPEN}} state,
> then disable and drop the tables. But this took a lot of time and we might
> not always have the option to drop these tables.
> HBCK 1 had a functionality (\{{fixTableOrphans}}) to regenerate the table
> descriptors "according to the memory cache or hdfs table directory
> structure". We should investigate if we can implement something like this
> with hbck2. We might not always be able to reproduce the table
> configurations, but we can probably make a "basic" one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-25297) [HBCK2] Regenerate missing table descriptors by hbck2
[
https://issues.apache.org/jira/browse/HBASE-25297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25297 started by Mate Szalay-Beko.
> [HBCK2] Regenerate missing table descriptors by hbck2
> -
>
> Key: HBASE-25297
> URL: https://issues.apache.org/jira/browse/HBASE-25297
> Project: HBase
> Issue Type: Improvement
> Components: hbase-operator-tools, hbck2
>Affects Versions: 2.2.6, 2.3.2
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> We just have been in a situation, when we saw many ServerCrashProcedure to be
> blocked, because of the AssignProcedure is blocked because of the missing
> .tabledesc files.
> {code:java}
> java.io.IOException: Missing table descriptor for
> dd0676f57bdbff5d04ab735b7daf5e9b
> {code}
>
> In our case it was OK for us to get rid of these tables and we were able to
> use {{setRegionState}} to move all these regions to {{FAILED_OPEN}} state,
> then disable and drop the tables. But this took a lot of time and we might
> not always have the option to drop these tables.
> HBCK 1 had a functionality (\{{fixTableOrphans}}) to regenerate the table
> descriptors "according to the memory cache or hdfs table directory
> structure". We should investigate if we can implement something like this
> with hbck2. We might not always be able to reproduce the table
> configurations, but we can probably make a "basic" one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25318) Configure where IntegrationTestImportTsv generates HFiles
[
https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25318:
-
Description:
Currently IntegrationTestImportTsv is generating HFiles under the working
directory of the hdfs user executing the tool, before bulkloading it into HBase.
Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself.
When IntegrationTestImportTsv is executed against a real distributed cluster
and the working directory of the current user (e.g. /user/hbase) is not in the
same encryption zone as the HBase root directory (e.g. /hbase/data) then you
will get an exception:
{code:java}
ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial failure
due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
{code}
In this ticket I make it configurable where the IntegrationTestImportTsv
generates the HFiles. From now, one can execute this integration test on HDFS
Transparent Encryption enabled clusters, like:
{code:java}
./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D
IntegrationTestImportTsv.generatedHFileFolder=//testdata{code}
was:
IntegrationTestImportTsv is generating HFiles under the working directory of
the current hdfs user executing the tool, before bulkloading it into HBase.
Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself.
When IntegrationTestImportTsv is executed against a real distributed cluster
and the working directory of the current user (e.g. /user/hbase) is not in the
same encryption zone as the HBase root directory (e.g. /hbase/data) then you
will get an exception:
{code:java}
2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
There was a partial failure due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
{code}
We should make it configurable where the IntegrationTestImportTsv generates the
HFiles.
> Configure where IntegrationTestImportTsv generates HFiles
> -
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Minor
>
> Currently IntegrationTestImportTsv is generating HFiles under the working
> directory of the hdfs user executing the tool, before bulkloading it into
> HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS
> Transparent Encryption), you can bulkload HFiles only if they sit in the same
> encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster
> and the working directory of the current user (e.g. /user/hbase) is not in
> the same encryption zone as the HBase root directory (e.g. /hbase/data) then
> you will get an exception:
>
> {code:java}
> ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial
> failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> can't be moved into an encryption zone.
> {code}
>
> In this ticket I make it configurable where the IntegrationTestImportTsv
> generates the HFiles. From now, one can execute this integration test on HDFS
> Transparent Encryption enabled clusters, like:
> {code:java}
> ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D
> IntegrationTestImportTsv.generatedHFileFolder=/ path>/testdata{code}
--
This message was sent by
[jira] [Work started] (HBASE-25318) Configure where IntegrationTestImportTsv generates HFiles
[
https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25318 started by Mate Szalay-Beko.
> Configure where IntegrationTestImportTsv generates HFiles
> -
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Minor
>
> Currently IntegrationTestImportTsv is generating HFiles under the working
> directory of the hdfs user executing the tool, before bulkloading it into
> HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS
> Transparent Encryption), you can bulkload HFiles only if they sit in the same
> encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster
> and the working directory of the current user (e.g. /user/hbase) is not in
> the same encryption zone as the HBase root directory (e.g. /hbase/data) then
> you will get an exception:
>
> {code:java}
> ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial
> failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> can't be moved into an encryption zone.
> {code}
>
> In this ticket I make it configurable where the IntegrationTestImportTsv
> generates the HFiles. From now, one can execute this integration test on HDFS
> Transparent Encryption enabled clusters, like:
> {code:java}
> ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv -D
> IntegrationTestImportTsv.generatedHFileFolder=/ path>/testdata{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25318) Configure where IntegrationTestImportTsv generates HFiles
[
https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25318:
-
Summary: Configure where IntegrationTestImportTsv generates HFiles (was:
Config option for IntegrationTestImportTsv where to generate HFiles to bulkload)
> Configure where IntegrationTestImportTsv generates HFiles
> -
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Minor
>
> IntegrationTestImportTsv is generating HFiles under the working directory of
> the current hdfs user executing the tool, before bulkloading it into HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS
> Transparent Encryption), you can bulkload HFiles only if they sit in the same
> encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster
> and the working directory of the current user (e.g. /user/hbase) is not in
> the same encryption zone as the HBase root directory (e.g. /hbase/data) then
> you will get an exception:
>
> {code:java}
> 2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
> There was a partial failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> can't be moved into an encryption zone.
> {code}
> We should make it configurable where the IntegrationTestImportTsv generates
> the HFiles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25263:
-
Release Note: Since the backward-compatible change we introduced in
HBASE-25263, we use the more secure PBKDF2WithHmacSHA384 key generation
algorithm (instead of PBKDF2WithHmacSHA1) to generate a secret key for HFile /
WalFile encryption, when the user is defining a string encryption key in the
hbase shell.
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> This PR is a follow-up of HBASE-25181 (#2539), where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). We are changing it to {{PBKDF2WithHmacSHA384}}. It will not break
> backward-compatibility, as even the tables created by the shell using the new
> algorithm will be able to load (e.g. during bulkload / replication) the
> HFiles serialized with the key generated by an old algorithm, as the HFiles
> themselves already contain the key necessary for their decryption.
> Smaller issues to be fixed:
> 2. Improve the documentation e.g. with the changes introduced by HBASE-25181
> and also by some points discussed on the Jira ticket of HBASE-25263.
> 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw {{IllegalStateExceptions}} instead of
> {{RuntimeExceptions}}.
> 4. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25318) Config option for IntegrationTestImportTsv where to generate HFiles to bulkload
[
https://issues.apache.org/jira/browse/HBASE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25318:
-
Description:
IntegrationTestImportTsv is generating HFiles under the working directory of
the current hdfs user executing the tool, before bulkloading it into HBase.
Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself.
When IntegrationTestImportTsv is executed against a real distributed cluster
and the working directory of the current user (e.g. /user/hbase) is not in the
same encryption zone as the HBase root directory (e.g. /hbase/data) then you
will get an exception:
{code:java}
2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
There was a partial failure due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
{code}
We should make it configurable where the IntegrationTestImportTsv generates the
HFiles.
was:
When IntegrationTestImportTsv is executed against a real distributed cluster,
it is generating HFiles under the working directory of the current hdfs user
executing the tool, before bulkloading it into HBase.
Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself. If the working
directory of the current user (e.g. /user/hbase) is not in the same encryption
zone as the HBase root directory (e.g. /hbase/data) then you will get an
exception when executing the IntegrationTestImportTsv tool:
{code:java}
2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
There was a partial failure due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
{code}
We should make it configurable where the IntegrationTestImportTsv generates the
HFiles.
> Config option for IntegrationTestImportTsv where to generate HFiles to
> bulkload
> ---
>
> Key: HBASE-25318
> URL: https://issues.apache.org/jira/browse/HBASE-25318
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Minor
>
> IntegrationTestImportTsv is generating HFiles under the working directory of
> the current hdfs user executing the tool, before bulkloading it into HBase.
> Assuming you encrypt the HBase root directory within HDFS (using HDFS
> Transparent Encryption), you can bulkload HFiles only if they sit in the same
> encryption zone in HDFS as the HBase root directory itself.
> When IntegrationTestImportTsv is executed against a real distributed cluster
> and the working directory of the current user (e.g. /user/hbase) is not in
> the same encryption zone as the HBase root directory (e.g. /hbase/data) then
> you will get an exception:
>
> {code:java}
> 2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
> There was a partial failure due to IO when attempting to load d :
> hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> /user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
> can't be moved into an encryption zone.
> {code}
> We should make it configurable where the IntegrationTestImportTsv generates
> the HFiles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25318) Config option for IntegrationTestImportTsv where to generate HFiles to bulkload
Mate Szalay-Beko created HBASE-25318:
Summary: Config option for IntegrationTestImportTsv where to
generate HFiles to bulkload
Key: HBASE-25318
URL: https://issues.apache.org/jira/browse/HBASE-25318
Project: HBase
Issue Type: Improvement
Components: integration tests
Reporter: Mate Szalay-Beko
Assignee: Mate Szalay-Beko
When IntegrationTestImportTsv is executed against a real distributed cluster,
it is generating HFiles under the working directory of the current hdfs user
executing the tool, before bulkloading it into HBase.
Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself. If the working
directory of the current user (e.g. /user/hbase) is not in the same encryption
zone as the HBase root directory (e.g. /hbase/data) then you will get an
exception when executing the IntegrationTestImportTsv tool:
{code:java}
2020-11-21 01:06:28,963 ERROR org.apache.hadoop.hbase.regionserver.HRegion:
There was a partial failure due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
{code}
We should make it configurable where the IntegrationTestImportTsv generates the
HFiles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25304) Support AES-192 and AES-256 in DefaultCipherProvider
Mate Szalay-Beko created HBASE-25304:
Summary: Support AES-192 and AES-256 in DefaultCipherProvider
Key: HBASE-25304
URL: https://issues.apache.org/jira/browse/HBASE-25304
Project: HBase
Issue Type: Improvement
Components: encryption
Reporter: Mate Szalay-Beko
Assignee: Mate Szalay-Beko
The DefaultCipherProvider currently supports AES-128. In some security policies
(such as the Application Security and Development STIG), AES-256 is required in
certain situations.
I want to add AES-192 and AES-256 support. I quickly tried to implement this as
part of HBASE-25263, but after 1-2 days I realized that it worths a separate
task in Jira. The main challenge is that the key length and the algorithm needs
to be decoupled in the code, and also some more tests need to be added to make
sure we are backward-compatible and also supporting AES-192 and AES-256.
Beside defining a new algorithm and key on the Java API, I also want to make
the usage of e.g. AES-256 in the shell, like:
{code:java}
create 'test', {NAME => 'cf', ENCRYPTION => 'AES-256', ENCRYPTION_KEY =>
'mysecret'}
{code}
Also we should support AES-192 and AES-256 in master encryption keys. And we
need to document how the users can configure / use it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25263 started by Mate Szalay-Beko.
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This PR is a follow-up of HBASE-25181 (#2539), where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). We are changing it to {{PBKDF2WithHmacSHA384}}. It will not break
> backward-compatibility, as even the tables created by the shell using the new
> algorithm will be able to load (e.g. during bulkload / replication) the
> HFiles serialized with the key generated by an old algorithm, as the HFiles
> themselves already contain the key necessary for their decryption.
> Smaller issues to be fixed:
> 2. Improve the documentation e.g. with the changes introduced by HBASE-25181
> and also by some points discussed on the Jira ticket of HBASE-25263.
> 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw {{IllegalStateExceptions}} instead of
> {{RuntimeExceptions}}.
> 4. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25263:
-
Description:
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:
1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to generate
a secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and not
allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to {{PBKDF2WithHmacSHA384}}. It will not break
backward-compatibility, as even the tables created by the shell using the new
algorithm will be able to load (e.g. during bulkload / replication) the HFiles
serialized with the key generated by an old algorithm, as the HFiles themselves
already contain the key necessary for their decryption.
Smaller issues to be fixed:
2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.
3. In {{EncryptionUtil.createEncryptionContext}} the various encryption config
checks should throw {{IllegalStateExceptions}} instead of {{RuntimeExceptions}}.
4. Test cases in {{TestEncryptionTest.java}} should be broken down into smaller
tests.
5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit rule
to validate exceptions.
was:
This ticket is a follow-up of HBASE-25181, where several issues were discussed
on the PR:
1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to generate
a secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and not
allowed in certain environments (e.g. on FIPS compliant clusters). Our plan is
to change it to e.g. {{PBKDF2WithHmacSHA384}}. If will not break
backward-compatibility, as even the tables created by the shell using the new
algorithm will be able to load (e.g. during bulkload / replication) the HFiles
serialized with the key generated by an old algorithm, as the HFiles themselves
already contain the key necessary for their decryption.
Smaller issues:
2. Updating the documentation about changes introduced both by HBASE-25181 and
by this change.
3. In {{EncryptionUtil.createEncryptionContext}} the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.
4. Test cases in {{TestEncryptionTest.java}} should be broken down into smaller
tests.
5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit rule
to validate exceptions.
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This PR is a follow-up of HBASE-25181 (#2539), where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). We are changing it to {{PBKDF2WithHmacSHA384}}. It will not break
> backward-compatibility, as even the tables created by the shell using the new
> algorithm will be able to load (e.g. during bulkload / replication) the
> HFiles serialized with the key generated by an old algorithm, as the HFiles
> themselves already contain the key necessary for their decryption.
> Smaller issues to be fixed:
> 2. Improve the documentation e.g. with the changes introduced by HBASE-25181
> and also by some points discussed on the Jira ticket of HBASE-25263.
> 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw {{IllegalStateExceptions}} instead of
> {{RuntimeExceptions}}.
> 4. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25263:
-
Description:
This ticket is a follow-up of HBASE-25181, where several issues were discussed
on the PR:
1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to generate
a secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and not
allowed in certain environments (e.g. on FIPS compliant clusters). Our plan is
to change it to e.g. {{PBKDF2WithHmacSHA384}}. If will not break
backward-compatibility, as even the tables created by the shell using the new
algorithm will be able to load (e.g. during bulkload / replication) the HFiles
serialized with the key generated by an old algorithm, as the HFiles themselves
already contain the key necessary for their decryption.
Smaller issues:
2. Updating the documentation about changes introduced both by HBASE-25181 and
by this change.
3. In {{EncryptionUtil.createEncryptionContext}} the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.
4. Test cases in {{TestEncryptionTest.java}} should be broken down into smaller
tests.
5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit rule
to validate exceptions.
was:
This ticket is a follow-up of HBASE-25181, where several issues were discussed
on the PR:
1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to generate
a secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and not
allowed in certain environments (e.g. on FIPS compliant clusters). Our plan is
to change it to e.g. {{PBKDF2WithHmacSHA384}}. If this would break backward
compatibility, then we should make this algorithm configurable.
Smaller issues:
2. In {{EncryptionUtil.createEncryptionContext}} the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.
3. Test cases in {{TestEncryptionTest.java}} should be broken down into smaller
tests.
4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit rule
to validate exceptions.
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If will
> not break backward-compatibility, as even the tables created by the shell
> using the new algorithm will be able to load (e.g. during bulkload /
> replication) the HFiles serialized with the key generated by an old
> algorithm, as the HFiles themselves already contain the key necessary for
> their decryption.
> Smaller issues:
> 2. Updating the documentation about changes introduced both by HBASE-25181
> and by this change.
> 3. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 4. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 5. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25261) Upgrade Bootstrap to 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25261: - Description: HBase UI is currently using bootstrap 3.3.7. This version is vulnerable to 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is here: [https://snyk.io/vuln/npm:bootstrap] Upgrading to bootstrap 4 would be nice, but potentially more work to do. To avoid these CVE issues, we should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently. was: HBase UI is currently using bootstrap 3.3.7. This version is vulnerable to 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is here: [https://snyk.io/vuln/npm:bootstrap] Upgrading to bootstrap 4 would be nice, but potentially more work to do. We should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently. > Upgrade Bootstrap to 3.4.1 > -- > > Key: HBASE-25261 > URL: https://issues.apache.org/jira/browse/HBASE-25261 > Project: HBase > Issue Type: Improvement > Components: security, UI >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > HBase UI is currently using bootstrap 3.3.7. This version is vulnerable to 4 > medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and > CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is > here: [https://snyk.io/vuln/npm:bootstrap] > Upgrading to bootstrap 4 would be nice, but potentially more work to do. To > avoid these CVE issues, we should at least upgrade to the latest bootstrap 3, > which is 3.4.1 currently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25261) Upgrade Bootstrap to 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25261: - Description: HBase UI is currently using bootstrap 3.3.7. This version is vulnerable to 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is here: [https://snyk.io/vuln/npm:bootstrap] Upgrading to bootstrap 4 would be nice, but potentially more work to do. We should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently. was: HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is here: [https://snyk.io/vuln/npm:bootstrap] Upgrading to bootstrap 4 would be nice, but potentially more work to do. We should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently. > Upgrade Bootstrap to 3.4.1 > -- > > Key: HBASE-25261 > URL: https://issues.apache.org/jira/browse/HBASE-25261 > Project: HBase > Issue Type: Improvement > Components: security, UI >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > HBase UI is currently using bootstrap 3.3.7. This version is vulnerable to 4 > medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and > CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is > here: [https://snyk.io/vuln/npm:bootstrap] > Upgrading to bootstrap 4 would be nice, but potentially more work to do. We > should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25297) Regenerate missing table descriptors by hbck2
[
https://issues.apache.org/jira/browse/HBASE-25297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25297:
-
Summary: Regenerate missing table descriptors by hbck2 (was: Regenerate a
missing table descriptors by hbck2)
> Regenerate missing table descriptors by hbck2
> -
>
> Key: HBASE-25297
> URL: https://issues.apache.org/jira/browse/HBASE-25297
> Project: HBase
> Issue Type: Improvement
> Components: hbase-operator-tools, hbck2
>Affects Versions: 2.2.6, 2.3.2
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> We just have been in a situation, when we saw many ServerCrashProcedure to be
> blocked, because of the AssignProcedure is blocked because of the missing
> .tabledesc files.
> {code:java}
> java.io.IOException: Missing table descriptor for
> dd0676f57bdbff5d04ab735b7daf5e9b
> {code}
>
> In our case it was OK for us to get rid of these tables and we were able to
> use {{setRegionState}} to move all these regions to {{FAILED_OPEN}} state,
> then disable and drop the tables. But this took a lot of time and we might
> not always have the option to drop these tables.
> HBCK 1 had a functionality (\{{fixTableOrphans}}) to regenerate the table
> descriptors "according to the memory cache or hdfs table directory
> structure". We should investigate if we can implement something like this
> with hbck2. We might not always be able to reproduce the table
> configurations, but we can probably make a "basic" one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25297) Regenerate a missing table descriptors by hbck2
[
https://issues.apache.org/jira/browse/HBASE-25297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25297:
-
Summary: Regenerate a missing table descriptors by hbck2 (was: regenerate
a missing table descriptors by hbck2)
> Regenerate a missing table descriptors by hbck2
> ---
>
> Key: HBASE-25297
> URL: https://issues.apache.org/jira/browse/HBASE-25297
> Project: HBase
> Issue Type: Improvement
> Components: hbase-operator-tools, hbck2
>Affects Versions: 2.2.6, 2.3.2
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> We just have been in a situation, when we saw many ServerCrashProcedure to be
> blocked, because of the AssignProcedure is blocked because of the missing
> .tabledesc files.
> {code:java}
> java.io.IOException: Missing table descriptor for
> dd0676f57bdbff5d04ab735b7daf5e9b
> {code}
>
> In our case it was OK for us to get rid of these tables and we were able to
> use {{setRegionState}} to move all these regions to {{FAILED_OPEN}} state,
> then disable and drop the tables. But this took a lot of time and we might
> not always have the option to drop these tables.
> HBCK 1 had a functionality (\{{fixTableOrphans}}) to regenerate the table
> descriptors "according to the memory cache or hdfs table directory
> structure". We should investigate if we can implement something like this
> with hbck2. We might not always be able to reproduce the table
> configurations, but we can probably make a "basic" one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25297) regenerate a missing table descriptors by hbck2
Mate Szalay-Beko created HBASE-25297:
Summary: regenerate a missing table descriptors by hbck2
Key: HBASE-25297
URL: https://issues.apache.org/jira/browse/HBASE-25297
Project: HBase
Issue Type: Improvement
Components: hbase-operator-tools, hbck2
Affects Versions: 2.3.2, 2.2.6
Reporter: Mate Szalay-Beko
Assignee: Mate Szalay-Beko
We just have been in a situation, when we saw many ServerCrashProcedure to be
blocked, because of the AssignProcedure is blocked because of the missing
.tabledesc files.
{code:java}
java.io.IOException: Missing table descriptor for
dd0676f57bdbff5d04ab735b7daf5e9b
{code}
In our case it was OK for us to get rid of these tables and we were able to use
{{setRegionState}} to move all these regions to {{FAILED_OPEN}} state, then
disable and drop the tables. But this took a lot of time and we might not
always have the option to drop these tables.
HBCK 1 had a functionality (\{{fixTableOrphans}}) to regenerate the table
descriptors "according to the memory cache or hdfs table directory structure".
We should investigate if we can implement something like this with hbck2. We
might not always be able to reproduce the table configurations, but we can
probably make a "basic" one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25261) Upgrade Bootstrap to 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25261: - Description: HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is here: [https://snyk.io/vuln/npm:bootstrap] Upgrading to bootstrap 4 would be nice, but potentially more work to do. We should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently. was: HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is here: https://snyk.io/vuln/npm:bootstrap Upgrading to bootstrap 4 would be nice, but potentially more work to do. We should upgrade to the latest bootstrap 3, which is 3.4.1 currently. > Upgrade Bootstrap to 3.4.1 > -- > > Key: HBASE-25261 > URL: https://issues.apache.org/jira/browse/HBASE-25261 > Project: HBase > Issue Type: Improvement > Components: security, UI >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to > 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and > CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is > here: [https://snyk.io/vuln/npm:bootstrap] > Upgrading to bootstrap 4 would be nice, but potentially more work to do. We > should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230669#comment-17230669
]
Mate Szalay-Beko commented on HBASE-25263:
--
These are very good points, thanks!
{quote}... each HFile has all of the necessary metadata for supporting its
decryption stored within it ...
{quote}
Yes, I didn't think it through deep enough before asking the question. You are
right and this also means that we can change the derivation function without
causing problems with backward compatibility. We can even bulkload HFiles /
move snapshots to other cluster and all the "old" hfiles will be readable by
the "new" cluster (assuming the master encryption key is the same / still
known).
{quote}If our older key derivation function has been deemed insufficient then
all of the derived DEKs should probably also be treated as insufficient. The
guidance we give there is compaction will rewrite the files with newly derived
DEKs. Major compaction will rewrite most files, but not all, so there is scope
for further improvement here... We may want an unconditional major compaction
to ensure all DEKs are rotated.
{quote}
This is a valid point. Also an unconditional major compaction could be useful
for other things too (like changing HFile compression maybe?). I can check the
code and create a follow-up ticket later.
BTW in the documentation
([http://hbase.apache.org/book.html#hbase.encryption.server)] we only recommend
major compaction, we don't mention that this won't necessary rewrite all the
HFiles (I might need to update this):
{quote}Rotate the Data Key
To rotate the data key, first change the ColumnFamily key in the column
descriptor, then trigger a major compaction. When compaction is complete, all
HFiles will be re-encrypted using the new data key. Until the compaction
completes, the old HFiles will still be readable using the old key.
{quote}
Also a note for me: I should also update the documentation mentioning the key
derivation function we use in the HBase shell (and also mention that the users
will have a better control over the encryption key if they provide it from Java
code and not from the shell)
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If
> this would break backward compatibility, then we should make this algorithm
> configurable.
> Smaller issues:
> 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 3. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25267) Add SSL keystore type and truststore related configs for HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25267:
-
Description:
The RESTServer currently relies on the following parameters to configure SSL on
the REST API:
* {{hbase.rest.ssl.enabled}}
* {{hbase.rest.ssl.keystore.store}}
* {{hbase.rest.ssl.keystore.password}}
* {{hbase.rest.ssl.keystore.keypassword}}
* {{hbase.rest.ssl.exclude.cipher.suites}}
* {{hbase.rest.ssl.include.cipher.suites}}
* {{hbase.rest.ssl.exclude.protocols}}
* {{hbase.rest.ssl.include.protocols}}
In this patch I want to introduce the following new parameters:
* {{hbase.rest.ssl.keystore.type}}
* {{hbase.rest.ssl.truststore.store}}
* {{hbase.rest.ssl.truststore.password}}
* {{hbase.rest.ssl.truststore.type}}
If any of the new the parameter is not provided, then we should fall-back to
the current behaviour (e.g. assuming JKS keystore/truststore types, or no
passwords, or no custom trust store file).
was:
The keystore configuration of the RESTServer currently relies on the following
parameters to configure SSL:
* hbase.rest.ssl.enabled
* hbase.rest.ssl.keystore.store
* hbase.rest.ssl.keystore.password
* hbase.rest.ssl.keystore.keypassword
* hbase.rest.ssl.exclude.cipher.suites
* hbase.rest.ssl.include.cipher.suites
* hbase.rest.ssl.exclude.protocols
* hbase.rest.ssl.include.protocols
In this patch I want to introduce the following new parameters:
* {{hbase.rest.ssl.keystore.type}}
* {{hbase.rest.ssl.truststore.store}}
* {{hbase.rest.ssl.}}{{truststore}}{{.password}}
* {{hbase.rest.ssl.}}{{truststore}}{{.type}}
If any of the new the parameter is not provided, then we should fall-back to
the current behaviour (e.g. assuming JKS keystore/truststore types, or no
passwords, or no custom trust store file).
> Add SSL keystore type and truststore related configs for HBase RESTServer
> -
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> The RESTServer currently relies on the following parameters to configure SSL
> on the REST API:
> * {{hbase.rest.ssl.enabled}}
> * {{hbase.rest.ssl.keystore.store}}
> * {{hbase.rest.ssl.keystore.password}}
> * {{hbase.rest.ssl.keystore.keypassword}}
> * {{hbase.rest.ssl.exclude.cipher.suites}}
> * {{hbase.rest.ssl.include.cipher.suites}}
> * {{hbase.rest.ssl.exclude.protocols}}
> * {{hbase.rest.ssl.include.protocols}}
> In this patch I want to introduce the following new parameters:
> * {{hbase.rest.ssl.keystore.type}}
> * {{hbase.rest.ssl.truststore.store}}
> * {{hbase.rest.ssl.truststore.password}}
> * {{hbase.rest.ssl.truststore.type}}
> If any of the new the parameter is not provided, then we should fall-back to
> the current behaviour (e.g. assuming JKS keystore/truststore types, or no
> passwords, or no custom trust store file).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25267) Add SSL keystore type and truststore related configs for HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25267:
-
Summary: Add SSL keystore type and truststore related configs for HBase
RESTServer (was: Make SSL truststore and keystore type configurable in HBase
RESTServer)
> Add SSL keystore type and truststore related configs for HBase RESTServer
> -
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> The keystore configuration of the RESTServer currently relies on the
> following parameters to configure SSL:
> * hbase.rest.ssl.enabled
> * hbase.rest.ssl.keystore.store
> * hbase.rest.ssl.keystore.password
> * hbase.rest.ssl.keystore.keypassword
> * hbase.rest.ssl.exclude.cipher.suites
> * hbase.rest.ssl.include.cipher.suites
> * hbase.rest.ssl.exclude.protocols
> * hbase.rest.ssl.include.protocols
> In this patch I want to introduce the following new parameters:
> * {{hbase.rest.ssl.keystore.type}}
> * {{hbase.rest.ssl.truststore.store}}
> * {{hbase.rest.ssl.}}{{truststore}}{{.password}}
> * {{hbase.rest.ssl.}}{{truststore}}{{.type}}
> If any of the new the parameter is not provided, then we should fall-back to
> the current behaviour (e.g. assuming JKS keystore/truststore types, or no
> passwords, or no custom trust store file).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25267) Make SSL truststore and keystore type configurable in HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25267:
-
Description:
The keystore configuration of the RESTServer currently relies on the following
parameters to configure SSL:
* hbase.rest.ssl.enabled
* hbase.rest.ssl.keystore.store
* hbase.rest.ssl.keystore.password
* hbase.rest.ssl.keystore.keypassword
* hbase.rest.ssl.exclude.cipher.suites
* hbase.rest.ssl.include.cipher.suites
* hbase.rest.ssl.exclude.protocols
* hbase.rest.ssl.include.protocols
In this patch I want to introduce the following new parameters:
* {{hbase.rest.ssl.keystore.type}}
* {{hbase.rest.ssl.truststore.store}}
* {{hbase.rest.ssl.}}{{truststore}}{{.password}}
* {{hbase.rest.ssl.}}{{truststore}}{{.type}}
If any of the new the parameter is not provided, then we should fall-back to
the current behaviour (e.g. assuming JKS keystore/truststore types, or no
passwords, or no custom trust store file).
was:
The keystore configuration of the RESTServer currently relies on the following
parameters to configure SSL:
* hbase.rest.ssl.enabled
* hbase.rest.ssl.keystore.store
* hbase.rest.ssl.keystore.password
* hbase.rest.ssl.keystore.keypassword
* hbase.rest.ssl.exclude.cipher.suites
* hbase.rest.ssl.include.cipher.suites
* hbase.rest.ssl.exclude.protocols
* hbase.rest.ssl.include.protocols
In this patch I want to introduce the {{hbase.rest.ssl.keystore.type}}
parameter, enabling us to customize the keystore type for the REST server. If
the parameter is not provided, then we should fall-back to the current
behaviour (which assumes keystore type JKS).
(this is similar how we already do in the InfoServer with the
\{{ssl.server.keystore.type}} parameter)
> Make SSL truststore and keystore type configurable in HBase RESTServer
> --
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> The keystore configuration of the RESTServer currently relies on the
> following parameters to configure SSL:
> * hbase.rest.ssl.enabled
> * hbase.rest.ssl.keystore.store
> * hbase.rest.ssl.keystore.password
> * hbase.rest.ssl.keystore.keypassword
> * hbase.rest.ssl.exclude.cipher.suites
> * hbase.rest.ssl.include.cipher.suites
> * hbase.rest.ssl.exclude.protocols
> * hbase.rest.ssl.include.protocols
> In this patch I want to introduce the following new parameters:
> * {{hbase.rest.ssl.keystore.type}}
> * {{hbase.rest.ssl.truststore.store}}
> * {{hbase.rest.ssl.}}{{truststore}}{{.password}}
> * {{hbase.rest.ssl.}}{{truststore}}{{.type}}
> If any of the new the parameter is not provided, then we should fall-back to
> the current behaviour (e.g. assuming JKS keystore/truststore types, or no
> passwords, or no custom trust store file).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25267) Make SSL truststore and keystore type configurable in HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25267:
-
Summary: Make SSL truststore and keystore type configurable in HBase
RESTServer (was: Make SSL keystore type configurable in HBase RESTServer)
> Make SSL truststore and keystore type configurable in HBase RESTServer
> --
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> The keystore configuration of the RESTServer currently relies on the
> following parameters to configure SSL:
> * hbase.rest.ssl.enabled
> * hbase.rest.ssl.keystore.store
> * hbase.rest.ssl.keystore.password
> * hbase.rest.ssl.keystore.keypassword
> * hbase.rest.ssl.exclude.cipher.suites
> * hbase.rest.ssl.include.cipher.suites
> * hbase.rest.ssl.exclude.protocols
> * hbase.rest.ssl.include.protocols
> In this patch I want to introduce the {{hbase.rest.ssl.keystore.type}}
> parameter, enabling us to customize the keystore type for the REST server. If
> the parameter is not provided, then we should fall-back to the current
> behaviour (which assumes keystore type JKS).
> (this is similar how we already do in the InfoServer with the
> \{{ssl.server.keystore.type}} parameter)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-25267) Make SSL keystore type configurable in HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-25267 started by Mate Szalay-Beko.
> Make SSL keystore type configurable in HBase RESTServer
> ---
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> The keystore configuration of the RESTServer currently relies on the
> following parameters to configure SSL:
> * hbase.rest.ssl.enabled
> * hbase.rest.ssl.keystore.store
> * hbase.rest.ssl.keystore.password
> * hbase.rest.ssl.keystore.keypassword
> * hbase.rest.ssl.exclude.cipher.suites
> * hbase.rest.ssl.include.cipher.suites
> * hbase.rest.ssl.exclude.protocols
> * hbase.rest.ssl.include.protocols
> In this patch I want to introduce the {{hbase.rest.ssl.keystore.type}}
> parameter, enabling us to customize the keystore type for the REST server. If
> the parameter is not provided, then we should fall-back to the current
> behaviour (which assumes keystore type JKS).
> (this is similar how we already do in the InfoServer with the
> \{{ssl.server.keystore.type}} parameter)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25267) Make SSL keystore type configurable in HBase RESTServer
Mate Szalay-Beko created HBASE-25267:
Summary: Make SSL keystore type configurable in HBase RESTServer
Key: HBASE-25267
URL: https://issues.apache.org/jira/browse/HBASE-25267
Project: HBase
Issue Type: Improvement
Components: REST
Reporter: Mate Szalay-Beko
Assignee: Mate Szalay-Beko
The keystore configuration of the RESTServer currently relies on the following
parameters to configure SSL:
* hbase.rest.ssl.enabled
* hbase.rest.ssl.keystore.store
* hbase.rest.ssl.keystore.password
* hbase.rest.ssl.keystore.keypassword
* hbase.rest.ssl.exclude.cipher.suites
* hbase.rest.ssl.include.cipher.suites
* hbase.rest.ssl.exclude.protocols
* hbase.rest.ssl.include.protocols
In this patch I want to introduce the {{hbase.rest.ssl.keystore.type}}
parameter, enabling us to customize the keystore type for the REST server. If
the parameter is not provided, then we should fall-back to the current
behaviour (which assumes keystore type JKS).
(this is similar how we already do in the InfoServer with the
\{{ssl.server.keystore.type}} parameter)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229080#comment-17229080
]
Mate Szalay-Beko commented on HBASE-25263:
--
regarding changing {{PBKDF2WithHmacSHA1}} to {{PBKDF2WithHmacSHA384}}:
[~apurtell] I saw you did something similar in HBASE-10951. On that ticket the
assumption was that there is no compatibility issue with simply changing this
algorithm:
{quote}No compatability issues that I can see given this isn't the way to
generate data keys for production. This is so one can use the shell to create a
schema with all encryption related attributes set up properly for basic
functional testing or integration tests.
{quote}
Do you think this is still true? Isn't is possible that someone is relying in
production on the schema defined with the HBase shell?
Alternatively (instead of simply changing this argument) I can introduce a
configuration variable, defaulting to the old algorithm.
What do you think?
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If
> this would break backward compatibility, then we should make this algorithm
> configurable.
> Smaller issues:
> 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 3. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HBASE-25263:
-
Description:
This ticket is a follow-up of HBASE-25181, where several issues were discussed
on the PR:
1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to generate
a secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and not
allowed in certain environments (e.g. on FIPS compliant clusters). Our plan is
to change it to e.g. {{PBKDF2WithHmacSHA384}}. If this would break backward
compatibility, then we should make this algorithm configurable.
Smaller issues:
2. In {{EncryptionUtil.createEncryptionContext}} the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.
3. Test cases in {{TestEncryptionTest.java}} should be broken down into smaller
tests.
4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit rule
to validate exceptions.
was:
This ticket is a follow-up of HBASE-25181, where several issues were discussed
on the PR:
1. Currently we use `PBKDF2WithHmacSHA1` key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and not
allowed in certain environments (e.g. on FIPS compliant clusters). Our plan is
to change it to e.g. `PBKDF2WithHmacSHA384`. If this would break backward
compatibility, then we should make this algorithm configurable.
Smaller issues:
2. In `EncryptionUtil.createEncryptionContext` the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.
3. Test cases in `TestEncryptionTest.java` should be broken down into smaller
tests.
4. `TestEncryptionDisabled.java` should use `ExpectedException` JUnit rule to
validate exceptions.
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If
> this would break backward compatibility, then we should make this algorithm
> configurable.
> Smaller issues:
> 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 3. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[ https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25263: - Description: This ticket is a follow-up of HBASE-25181, where several issues were discussed on the PR: 1. Currently we use `PBKDF2WithHmacSHA1` key generation algorithm to generate a secret key for HFile / WalFile encryption, when the user is defining a string encryption key in the hbase shell. This algorithm is not secure enough and not allowed in certain environments (e.g. on FIPS compliant clusters). Our plan is to change it to e.g. `PBKDF2WithHmacSHA384`. If this would break backward compatibility, then we should make this algorithm configurable. Smaller issues: 2. In `EncryptionUtil.createEncryptionContext` the various encryption config checks should throw IllegalStateExceptions instead of RuntimeExceptions. 3. Test cases in `TestEncryptionTest.java` should be broken down into smaller tests. 4. `TestEncryptionDisabled.java` should use `ExpectedException` JUnit rule to validate exceptions. was: This ticket is a follow-up of HBASE-25181, where several issues were discussed on the PR: 1. Currently we use `PBKDF2WithHmacSHA1` key generation algorithm to generate a secret key for HFile / WalFile encryption, when the user is defining a string encryption key in the hbase shell. This algorithm is not secure enough and not allowed in certain environments (like on FIPS compliant clusters). Our plan is to change it to e.g. `PBKDF2WithHmacSHA384`. If this would break backward compatibility, then we should make this algorithm configurable. 2. In `EncryptionUtil.createEncryptionContext` the various encryption config checks should throw IllegalStateExceptions instead of RuntimeExceptions. 3. Test cases in `TestEncryptionTest.java` should be broken down into smaller tests. 4. `TestEncryptionDisabled.java` should use `ExpectedException` JUnit rule to validate exceptions. > Change encryption key generation algorithm used in the HBase shell > -- > > Key: HBASE-25263 > URL: https://issues.apache.org/jira/browse/HBASE-25263 > Project: HBase > Issue Type: Improvement > Components: encryption, shell >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > This ticket is a follow-up of HBASE-25181, where several issues were > discussed on the PR: > 1. Currently we use `PBKDF2WithHmacSHA1` key generation algorithm to generate > a secret key for HFile / WalFile encryption, when the user is defining a > string encryption key in the hbase shell. This algorithm is not secure enough > and not allowed in certain environments (e.g. on FIPS compliant clusters). > Our plan is to change it to e.g. `PBKDF2WithHmacSHA384`. If this would break > backward compatibility, then we should make this algorithm configurable. > Smaller issues: > 2. In `EncryptionUtil.createEncryptionContext` the various encryption config > checks should throw IllegalStateExceptions instead of RuntimeExceptions. > 3. Test cases in `TestEncryptionTest.java` should be broken down into smaller > tests. > 4. `TestEncryptionDisabled.java` should use `ExpectedException` JUnit rule to > validate exceptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
Mate Szalay-Beko created HBASE-25263: Summary: Change encryption key generation algorithm used in the HBase shell Key: HBASE-25263 URL: https://issues.apache.org/jira/browse/HBASE-25263 Project: HBase Issue Type: Improvement Components: encryption, shell Reporter: Mate Szalay-Beko Assignee: Mate Szalay-Beko This ticket is a follow-up of HBASE-25181, where several issues were discussed on the PR: 1. Currently we use `PBKDF2WithHmacSHA1` key generation algorithm to generate a secret key for HFile / WalFile encryption, when the user is defining a string encryption key in the hbase shell. This algorithm is not secure enough and not allowed in certain environments (like on FIPS compliant clusters). Our plan is to change it to e.g. `PBKDF2WithHmacSHA384`. If this would break backward compatibility, then we should make this algorithm configurable. 2. In `EncryptionUtil.createEncryptionContext` the various encryption config checks should throw IllegalStateExceptions instead of RuntimeExceptions. 3. Test cases in `TestEncryptionTest.java` should be broken down into smaller tests. 4. `TestEncryptionDisabled.java` should use `ExpectedException` JUnit rule to validate exceptions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-25261) Upgrade Bootstrap to 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25261 started by Mate Szalay-Beko. > Upgrade Bootstrap to 3.4.1 > -- > > Key: HBASE-25261 > URL: https://issues.apache.org/jira/browse/HBASE-25261 > Project: HBase > Issue Type: Improvement > Components: security, UI >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to > 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and > CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is > here: https://snyk.io/vuln/npm:bootstrap > Upgrading to bootstrap 4 would be nice, but potentially more work to do. We > should upgrade to the latest bootstrap 3, which is 3.4.1 currently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25261) Upgrade Bootstrap to 3.4.1
Mate Szalay-Beko created HBASE-25261: Summary: Upgrade Bootstrap to 3.4.1 Key: HBASE-25261 URL: https://issues.apache.org/jira/browse/HBASE-25261 Project: HBase Issue Type: Improvement Components: security, UI Reporter: Mate Szalay-Beko Assignee: Mate Szalay-Beko HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to 4 medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331). Details on all the bootstrap versions and vulnerabilities is here: https://snyk.io/vuln/npm:bootstrap Upgrading to bootstrap 4 would be nice, but potentially more work to do. We should upgrade to the latest bootstrap 3, which is 3.4.1 currently. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25181) Configure hash algorithm in wrapped encryption keys
[ https://issues.apache.org/jira/browse/HBASE-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25181: - Description: Currently we are using MD5 hash algorithm to store a hash for encryption keys. This hash is needed to verify the secret key of the subject. (e.g. making sure that the same secret key is used during encrypted HFile / WalFile read and write). The MD5 algorithm is considered weak, and can not be used in some (e.g. FIPS compliant) clusters. However, currently it is not possible to use different hash algorithm, or to disable the whole column family encryption globally on the cluster. In this patch: * I introduce a backward compatible way of specifying the hash algorithm. This enable us to use newer and/or more secure hash algorithms like SHA-384 or SHA-512 (which are FIPS compliant). * I added a configuration parameter to globally enable / disable the column family encryption feature. (enabled by default for backward compatibility). This is handy if someone wants to operate an HBase cluster making sure that uses are only relying on other (e.g. HDFS based) encryption mechanisms. was: Currently we are using MD5 hash algorithm to store a hash for encryption keys. This hash is needed to verify the secret key of the subject. (e.g. making sure that the same secrey key is used during encrypted HFile read and write). The MD5 algorithm is considered weak, and can not be used in some (e.g. FIPS compliant) clusters. In the patch I plan to: * introduce a backward compatible way of specifying the hash algorithm. This enable us to use newer and more secure hash algorithms like SHA-384 or SHA-512 (which are FIPS compliant). * change the algoritm used by the hbase shell to generate secure keys for column family encryption (this is only used for testing schema in the shell, the proper data keys are generated by the Java API, see e.g. HBASE-10951) > Configure hash algorithm in wrapped encryption keys > --- > > Key: HBASE-25181 > URL: https://issues.apache.org/jira/browse/HBASE-25181 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > Currently we are using MD5 hash algorithm to store a hash for encryption > keys. This hash is needed to verify the secret key of the subject. (e.g. > making sure that the same secret key is used during encrypted HFile / WalFile > read and write). The MD5 algorithm is considered weak, and can not be used in > some (e.g. FIPS compliant) clusters. However, currently it is not possible to > use different hash algorithm, or to disable the whole column family > encryption globally on the cluster. > In this patch: > * I introduce a backward compatible way of specifying the hash algorithm. > This enable us to use newer and/or more secure hash algorithms like SHA-384 > or SHA-512 (which are FIPS compliant). > * I added a configuration parameter to globally enable / disable the column > family encryption feature. (enabled by default for backward compatibility). > This is handy if someone wants to operate an HBase cluster making sure that > uses are only relying on other (e.g. HDFS based) encryption mechanisms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-25181) Configure hash algorithm in wrapped encryption keys
[ https://issues.apache.org/jira/browse/HBASE-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25181 started by Mate Szalay-Beko. > Configure hash algorithm in wrapped encryption keys > --- > > Key: HBASE-25181 > URL: https://issues.apache.org/jira/browse/HBASE-25181 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > Currently we are using MD5 hash algorithm to store a hash for encryption > keys. This hash is needed to verify the secret key of the subject. (e.g. > making sure that the same secrey key is used during encrypted HFile read and > write). The MD5 algorithm is considered weak, and can not be used in some > (e.g. FIPS compliant) clusters. > In the patch I plan to: > * introduce a backward compatible way of specifying the hash algorithm. This > enable us to use newer and more secure hash algorithms like SHA-384 or > SHA-512 (which are FIPS compliant). > * change the algoritm used by the hbase shell to generate secure keys for > column family encryption (this is only used for testing schema in the shell, > the proper data keys are generated by the Java API, see e.g. HBASE-10951) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25181) Configure hash algorithm in wrapped encryption keys
Mate Szalay-Beko created HBASE-25181: Summary: Configure hash algorithm in wrapped encryption keys Key: HBASE-25181 URL: https://issues.apache.org/jira/browse/HBASE-25181 Project: HBase Issue Type: Improvement Affects Versions: 2.3.2 Reporter: Mate Szalay-Beko Assignee: Mate Szalay-Beko Currently we are using MD5 hash algorithm to store a hash for encryption keys. This hash is needed to verify the secret key of the subject. (e.g. making sure that the same secrey key is used during encrypted HFile read and write). The MD5 algorithm is considered weak, and can not be used in some (e.g. FIPS compliant) clusters. In the patch I plan to: * introduce a backward compatible way of specifying the hash algorithm. This enable us to use newer and more secure hash algorithms like SHA-384 or SHA-512 (which are FIPS compliant). * change the algoritm used by the hbase shell to generate secure keys for column family encryption (this is only used for testing schema in the shell, the proper data keys are generated by the Java API, see e.g. HBASE-10951) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25146) Add extra logging at info level to HFileCorruptionChecker in order to report progress
[
https://issues.apache.org/jira/browse/HBASE-25146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17207181#comment-17207181
]
Mate Szalay-Beko commented on HBASE-25146:
--
Also we just saw that the tool doesn't print out the number of mob files
checked. It is counting them with a different counter than the normal HFiles
({{hfilesChecked}}), but in the end it doesn't print out the value of the
{{mobFilesChecked}} counter. (I found this in 2.1, haven't check this on the
master branch)
> Add extra logging at info level to HFileCorruptionChecker in order to report
> progress
> -
>
> Key: HBASE-25146
> URL: https://issues.apache.org/jira/browse/HBASE-25146
> Project: HBase
> Issue Type: Improvement
> Components: hbck, hbck2
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
>
> Currently there's no progress reporting in HFileCorruptionChecker: neither in
> the logs nor in stdout. It only creates a report about the entire operation
> at the end of the process and emits some warning messages is corruption is
> found.
> Adding some logging about the progress would be beneficial for long running
> checks indicating that the process is running healthy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24556) HBase upgrade from 2.2 to 2.3 fails due to failure in ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-24556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135843#comment-17135843
]
Mate Szalay-Beko commented on HBASE-24556:
--
This is indeed a ZooKeeper Server issue. In a real production environment no
HBase miniZooKeeperCluster will be used I guess, but HBase will only act as a
ZooKeeper Client and will connect to a real distributed ZooKeeper Server.
BTW if someone would like to upgrade his/her ZooKeeper cluster from 3.4 to 3.5,
then this FAQ can be important:
[https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ]
> HBase upgrade from 2.2 to 2.3 fails due to failure in ZooKeeper
> ---
>
> Key: HBASE-24556
> URL: https://issues.apache.org/jira/browse/HBASE-24556
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 2.3.0
>Reporter: Zhuqi Jin
>Priority: Major
>
> When we tested upgrading HBase from 2.2 to 2.3, the upgraded node failed to
> start due to a known zookeeper failure - ZOOKEEPER-3056.
> The error message is shown below:
>
> {code:java}
> 2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
> exiting2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
> exitingjava.io.IOException: No snapshot found, but there are log entries.
> Something is broken! at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
> at
> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
> at
> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
> at
> org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
> at
> org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259)
> at
> org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:189)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:210)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3035)
> {code}
>
> It can be reproduced through the following steps:
> * Start a single-node HBase 2.2 with the following configuration.
> {code:java}
>
> hbase.rootdir
> /var/lib/hbase
>
> hbase.zookeeper.property.dataDir
> /var/lib/zookeeper
> {code}
> * Use HBase performance evaluation tool to generate data.
> {code:java}
> /hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
> sequentialWrite 1{code}
> * Upgrade the node to 2.3 with the same configuration. The new version node
> failed to start because of the zookeeper.
> HBase 2.2 is using zookeeper-3.4.10.jar, and Kafka 2.3 are using
> zookeeper-3.5.7.jar.
> The bug has been fixed by ZOOKEEPER-3056 in zookeeper-3.6.0. Should we
> upgrade the dependency of HBase 2.3 to use zookeeper-3.6.0.jar? or upgrade
> the configuration file of zookeeper-3.5.7.jar?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24132) Upgrade to Apache ZooKeeper 3.5.7
[ https://issues.apache.org/jira/browse/HBASE-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092134#comment-17092134 ] Mate Szalay-Beko commented on HBASE-24132: -- I would recommend this page on the ZooKeeper wiki. We are collecting the upgrade related issues here: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ > Upgrade to Apache ZooKeeper 3.5.7 > - > > Key: HBASE-24132 > URL: https://issues.apache.org/jira/browse/HBASE-24132 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.3 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Apache ZooKeeper 3.5.7 has been released, HDFS and other projects have > updated the dependent zookeeper verison. Perhaps Hbase may update as well. > Some of the interfaces are changed in this zookeeper version . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24139) Balancer should avoid leaving idle region servers
[
https://issues.apache.org/jira/browse/HBASE-24139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079290#comment-17079290
]
Mate Szalay-Beko commented on HBASE-24139:
--
> I am inclined to take the approach of adding a short circuit to needsBalance
is there any benefit for the user to fine-tune the weight of this rule? If not,
then I think the simpler approach (adding short circuit) is preferable.
Maybe there is a case, when the user first want to add 5 new RS to the cluster,
but doesn't want to re-balance 5 times. But I think in this case he should
disable the loadbalancer before before adding the new nodes then re-enable it
in the end.
> Balancer should avoid leaving idle region servers
> -
>
> Key: HBASE-24139
> URL: https://issues.apache.org/jira/browse/HBASE-24139
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, Operability
>Reporter: Sean Busbey
>Assignee: Beata Sudi
>Priority: Critical
> Labels: beginner
>
> After HBASE-15529 the StochasticLoadBalancer makes the decision to run based
> on its internal cost functions rather than the simple region count skew of
> BaseLoadBalancer.
> Given the default weights for those cost functions, the default minimum cost
> to indicate a need to rebalance, and a regions per region server density of
> ~90 we are not very responsive to adding additional region servers for
> non-trivial cluster sizes:
> * For clusters ~10 nodes, the defaults think a single RS at 0 regions means
> we need to balance
> * For clusters >20 nodes, the defaults will not consider a single RS at 0
> regions to mean we need to balance. 2 RS at 0 will cause it to balance.
> * For clusters ~100 nodes, having 6 RS with no regions will still not meet
> the threshold to cause a balance.
> Note that this is the decision to look at balancer plans at all. The
> calculation is severely dominated by the region count skew (it has weight 500
> and all other weights are ~105), so barring a very significant change in all
> other cost functions this condition will persist indefinitely.
> Two possible approaches:
> * add a new cost function that's essentially "don't have RS with 0 regions"
> that an operator can tune
> * add a short circuit condition for the {{needsBalance}} method that checks
> for empty RS similar to the check we do for colocated region replicas
> For those currently hitting this an easy work around is to set
> {{hbase.master.balancer.stochastic.minCostNeedBalance}} to {{0.01}}. This
> will mean that a single RS having 0 regions will cause the balancer to run
> for clusters of up to ~90 region servers. It's essentially the same as the
> default slop of 0.01 used by the BaseLoadBalancer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24139) Balancer should avoid leaving idle region servers
[
https://issues.apache.org/jira/browse/HBASE-24139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079283#comment-17079283
]
Mate Szalay-Beko commented on HBASE-24139:
--
> Given the other caveats against using "balancer per table" maybe we could
> expressly only do the short circuit when we are not doing per-table?
what about changing the new rule like: "don't have RS with 0 regions if there
is any RS with more than 1 regions"
this would be compatible also with "balancer per table"
> Balancer should avoid leaving idle region servers
> -
>
> Key: HBASE-24139
> URL: https://issues.apache.org/jira/browse/HBASE-24139
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, Operability
>Reporter: Sean Busbey
>Assignee: Beata Sudi
>Priority: Critical
> Labels: beginner
>
> After HBASE-15529 the StochasticLoadBalancer makes the decision to run based
> on its internal cost functions rather than the simple region count skew of
> BaseLoadBalancer.
> Given the default weights for those cost functions, the default minimum cost
> to indicate a need to rebalance, and a regions per region server density of
> ~90 we are not very responsive to adding additional region servers for
> non-trivial cluster sizes:
> * For clusters ~10 nodes, the defaults think a single RS at 0 regions means
> we need to balance
> * For clusters >20 nodes, the defaults will not consider a single RS at 0
> regions to mean we need to balance. 2 RS at 0 will cause it to balance.
> * For clusters ~100 nodes, having 6 RS with no regions will still not meet
> the threshold to cause a balance.
> Note that this is the decision to look at balancer plans at all. The
> calculation is severely dominated by the region count skew (it has weight 500
> and all other weights are ~105), so barring a very significant change in all
> other cost functions this condition will persist indefinitely.
> Two possible approaches:
> * add a new cost function that's essentially "don't have RS with 0 regions"
> that an operator can tune
> * add a short circuit condition for the {{needsBalance}} method that checks
> for empty RS similar to the check we do for colocated region replicas
> For those currently hitting this an easy work around is to set
> {{hbase.master.balancer.stochastic.minCostNeedBalance}} to {{0.01}}. This
> will mean that a single RS having 0 regions will cause the balancer to run
> for clusters of up to ~90 region servers. It's essentially the same as the
> default slop of 0.01 used by the BaseLoadBalancer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24132) Upgrade to Apache ZooKeeper 3.5.7
[ https://issues.apache.org/jira/browse/HBASE-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078216#comment-17078216 ] Mate Szalay-Beko commented on HBASE-24132: -- Thanks for the PR! We are already building HBase with ZooKeeper 3.5.x, not only because the 3.4 EOL, but also because the SSL support in ZooKeeper is important for us (and for our customers). I added a comment to the PR. > Upgrade to Apache ZooKeeper 3.5.7 > - > > Key: HBASE-24132 > URL: https://issues.apache.org/jira/browse/HBASE-24132 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.3 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Apache ZooKeeper 3.5.7 has been released, HDFS and other projects have > updated the dependent zookeeper verison. Perhaps Hbase may update as well. > Some of the interfaces are changed in this zookeeper version . -- This message was sent by Atlassian Jira (v8.3.4#803005)
