[jira] [Resolved] (HBASE-27337) [hbase-thirdparty] Add nimbus-jose-jwt library to hbase-shaded-miscellaneous
[ https://issues.apache.org/jira/browse/HBASE-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andor Molnar resolved HBASE-27337. -- Resolution: Won't Fix > [hbase-thirdparty] Add nimbus-jose-jwt library to hbase-shaded-miscellaneous > > > Key: HBASE-27337 > URL: https://issues.apache.org/jira/browse/HBASE-27337 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Andor Molnar >Assignee: Andor Molnar >Priority: Major > > Required for OAuthBearer SASL auth plugin to work properly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28339) HBaseReplicationEndpoint creates new ZooKeeper client every time it tries to reconnect
[
https://issues.apache.org/jira/browse/HBASE-28339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andor Molnar resolved HBASE-28339.
--
Resolution: Invalid
> HBaseReplicationEndpoint creates new ZooKeeper client every time it tries to
> reconnect
> --
>
> Key: HBASE-28339
> URL: https://issues.apache.org/jira/browse/HBASE-28339
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.6.0, 2.4.17, 3.0.0-beta-1, 2.5.7, 2.7.0
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
>
> Asbtract base class {{HBaseReplicationEndpoint}} and therefore
> {{HBaseInterClusterReplicationEndpoint}} creates new ZooKeeper client
> instance every time there's an error occurs in communication and it tries to
> reconnect. This was not a problem with ZooKeeper 3.4.x versions, because the
> TGT Login thread was a static reference and only created once for all clients
> in the same JVM. With the upgrade to ZooKeeper 3.5.x the login thread is
> dedicated to the client instance, hence we have a new login thread every time
> the replication endpoint reconnects.
> {code:java}
> /**
> * A private method used to re-establish a zookeeper session with a peer
> cluster.
> */
> protected void reconnect(KeeperException ke) {
> if (
> ke instanceof ConnectionLossException || ke instanceof
> SessionExpiredException
> || ke instanceof AuthFailedException
> ) {
> String clusterKey = ctx.getPeerConfig().getClusterKey();
> LOG.warn("Lost the ZooKeeper connection for peer " + clusterKey, ke);
> try {
> reloadZkWatcher();
> } catch (IOException io) {
> LOG.warn("Creation of ZookeeperWatcher failed for peer " + clusterKey,
> io);
> }
> }
> }{code}
> {code:java}
> /**
> * Closes the current ZKW (if not null) and creates a new one
> * @throws IOException If anything goes wrong connecting
> */
> synchronized void reloadZkWatcher() throws IOException {
> if (zkw != null) zkw.close();
> zkw = new ZKWatcher(ctx.getConfiguration(), "connection to cluster: " +
> ctx.getPeerId(), this);
> getZkw().registerListener(new PeerRegionServerListener(this));
> } {code}
> If the target cluster of replication is unavailable for some reason, the
> replication endpoint keeps trying to reconnect to ZooKeeper destroying and
> creating new Login threads constantly which will carpet bomb the KDC host
> with login requests.
>
> I'm not sure how to fix this yet, trying to create a unit test first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28340) Add trust/key store type settings to ZK TLS settings handled by HBase
Andor Molnar created HBASE-28340:
Summary: Add trust/key store type settings to ZK TLS settings
handled by HBase
Key: HBASE-28340
URL: https://issues.apache.org/jira/browse/HBASE-28340
Project: HBase
Issue Type: Sub-task
Components: Zookeeper
Affects Versions: 2.5.7, 3.0.0-beta-1, 2.4.17
Reporter: Andor Molnar
Assignee: Andor Molnar
Let's add the following settings as well. Last time we missed it.
{noformat}
zookeeper.ssl.keyStore.type
zookeeper.ssl.trustStore.type{noformat}
Handle them in hbase-site.xml as:
{noformat}
hbase.zookeeper.property.ssl.keyStore.type
hbase.zookeeper.property.ssl.trustStore.type{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28339) HBaseReplicationEndpoint creates new ZooKeeper client every time it tries to reconnect
Andor Molnar created HBASE-28339:
Summary: HBaseReplicationEndpoint creates new ZooKeeper client
every time it tries to reconnect
Key: HBASE-28339
URL: https://issues.apache.org/jira/browse/HBASE-28339
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 2.5.7, 3.0.0-beta-1, 2.4.17, 2.6.0, 2.7.0
Reporter: Andor Molnar
Assignee: Andor Molnar
Asbtract base class {{HBaseReplicationEndpoint}} and therefore
{{HBaseInterClusterReplicationEndpoint}} creates new ZooKeeper client instance
every time there's an error occurs in communication and it tries to reconnect.
This was not a problem with ZooKeeper 3.4.x versions, because the TGT Login
thread was a static reference and only created once for all clients in the same
JVM. With the upgrade to ZooKeeper 3.5.x the login thread is dedicated to the
client instance, hence we have a new login thread every time the replication
endpoint reconnects.
{code:java}
/**
* A private method used to re-establish a zookeeper session with a peer
cluster.
*/
protected void reconnect(KeeperException ke) {
if (
ke instanceof ConnectionLossException || ke instanceof
SessionExpiredException
|| ke instanceof AuthFailedException
) {
String clusterKey = ctx.getPeerConfig().getClusterKey();
LOG.warn("Lost the ZooKeeper connection for peer " + clusterKey, ke);
try {
reloadZkWatcher();
} catch (IOException io) {
LOG.warn("Creation of ZookeeperWatcher failed for peer " + clusterKey,
io);
}
}
}{code}
{code:java}
/**
* Closes the current ZKW (if not null) and creates a new one
* @throws IOException If anything goes wrong connecting
*/
synchronized void reloadZkWatcher() throws IOException {
if (zkw != null) zkw.close();
zkw = new ZKWatcher(ctx.getConfiguration(), "connection to cluster: " +
ctx.getPeerId(), this);
getZkw().registerListener(new PeerRegionServerListener(this));
} {code}
If the target cluster of replication is unavailable for some reason, the
replication endpoint keeps trying to reconnect to ZooKeeper destroying and
creating new Login threads constantly which will carpet bomb the KDC host with
login requests.
I'm not sure how to fix this yet, trying to create a unit test first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28337) Positive connection test in TestShadeSaslAuthenticationProvider runs with Kerberos instead of Shade
Andor Molnar created HBASE-28337: Summary: Positive connection test in TestShadeSaslAuthenticationProvider runs with Kerberos instead of Shade Key: HBASE-28337 URL: https://issues.apache.org/jira/browse/HBASE-28337 Project: HBase Issue Type: Test Affects Versions: 2.6.0, 2.4.18, 3.0.0, 2.7.0, 2.5.8 Reporter: Andor Molnar Assignee: Andor Molnar The positive test (testPositiveAuthentication) in TestShadeSaslAuthenticationProvider doesn't create a new user in user1.doAs(), so it will use the already Kerberos authenticated user instead of re-authenticating with the token. As a consequence it doesn't reveal a problem introduced with HBASE-23881 which will cause clients to timeout if authenticated with a SASL mech which doesn't create a reply token in case of successful authentication. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28274) Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)
Andor Molnar created HBASE-28274:
Summary: Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)
Key: HBASE-28274
URL: https://issues.apache.org/jira/browse/HBASE-28274
Project: HBase
Issue Type: Test
Components: flakies, integration tests, test
Reporter: Andor Molnar
Assignee: Andor Molnar
The following test sometimes fails for me when running locally with Maven:
TestFanOutOneBlockAsyncDFSOutput.testRecover()
I can't really figure out the reason, but it's probably a side effect of the
preceding test: testConnectToDatanodeFailed(). This test also restarts one of
the datanodes in the MiniDFS cluster just like testRecover() and it somehow
causes the failure.
{noformat}
java.lang.AssertionError: flush should fail
at org.junit.Assert.fail(Assert.java:89)
at
org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput.testRecover(TestFanOutOneBlockAsyncDFSOutput.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method){noformat}
The flush() call is testRecover() should fail, because we restart one of the DN
in the DFS cluster which is expected to brake connection. It succeeds though if
the preceding test already restarted a DN. No matter which DN we restart, even
if they're different, the error occurs.
I also tried to add CLUSTER.waitDatanodeFullyStarted() at the end of
testConnectToDatanodeFailed(), looks like it made the tests slightly more
stable, but didn't help fully.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28254) Flaky test: TestTableShell
Andor Molnar created HBASE-28254:
Summary: Flaky test: TestTableShell
Key: HBASE-28254
URL: https://issues.apache.org/jira/browse/HBASE-28254
Project: HBase
Issue Type: Test
Components: flakies, integration tests
Reporter: Andor Molnar
Assignee: Andor Molnar
The test is running the following Ruby commands:
{noformat}
# Instert test data
@test_table.put(1, "x:a", 1)
@test_table.put(2, "x:raw1", 11)
@test_table.put(2, "x:raw1", 11)
@test_table.put(2, "x:raw1", 11)
@test_table.put(2, "x:raw1", 11)
{noformat}
And validate the versions with:
{noformat}
args = { VERSIONS => 10, RAW => true } # Since 4 versions of row with rowkey 2
is been added, we can use any number >= 4 for VERSIONS to scan all 4 versions.
num_rows = 0
@test_table._scan_internal(args) do # Raw Scan
num_rows += 1
end
# 5 since , 1 from row key '1' and other 4 from row key '4'
assert_equal(num_rows, 5,
'Num rows scanned without RAW/VERSIONS are not 5')
{noformat}
Which sometimes (almost always on fast machines) fails, because it only finds 3
versions out of 4. I believe this due to commands are running too fast and
inserts data with same timestamp, so HBase cannot distinguish them. I'd like to
add some sleep between puts to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-28223) Include shaded netty-all in hbase-shaded-mapreduce
[ https://issues.apache.org/jira/browse/HBASE-28223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andor Molnar resolved HBASE-28223. -- Resolution: Invalid > Include shaded netty-all in hbase-shaded-mapreduce > -- > > Key: HBASE-28223 > URL: https://issues.apache.org/jira/browse/HBASE-28223 > Project: HBase > Issue Type: Improvement > Components: mapreduce, shading >Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.6 >Reporter: Andor Molnar >Assignee: Andor Molnar >Priority: Major > > Since ZooKeeper TLS connection is supported within HBase, we need netty-all > on the classpath. It's already present in hbase-shaded-client and we need it > in hbase-shaded-mapreduce for MR clients too. netty-all is already included > in pom.xml, but only with test-scope, so shading plugin doesn't add it to the > jar. Patch will override it to compile-scope. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28223) Including shaded netty-all in hbase-shaded-mapreduce
Andor Molnar created HBASE-28223: Summary: Including shaded netty-all in hbase-shaded-mapreduce Key: HBASE-28223 URL: https://issues.apache.org/jira/browse/HBASE-28223 Project: HBase Issue Type: Improvement Components: mapreduce, shading Reporter: Andor Molnar Assignee: Andor Molnar Since ZooKeeper TLS connection is supported within HBase, we need netty-all on the classpath. It's already present in hbase-shaded-client and we need it in hbase-shaded-mapreduce for MR clients too. netty-all is already included in pom.xml, but only with test-scope, so shading plugin doesn't add it to the jar. Patch will override it to compile-scope. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28063) Add documentation to HBase book
Andor Molnar created HBASE-28063: Summary: Add documentation to HBase book Key: HBASE-28063 URL: https://issues.apache.org/jira/browse/HBASE-28063 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Andor Molnar Assignee: Andor Molnar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28038) Add TLS settings to ZooKeeper client
Andor Molnar created HBASE-28038:
Summary: Add TLS settings to ZooKeeper client
Key: HBASE-28038
URL: https://issues.apache.org/jira/browse/HBASE-28038
Project: HBase
Issue Type: Improvement
Components: Zookeeper
Affects Versions: 2.5.5, 2.4.17, 3.0.0-alpha-4
Reporter: Andor Molnar
Assignee: Andor Molnar
ZooKeeper supports TLS connection from its clients. Currently the only way to
set up HBase for this is to add the following Java properties to the HBase
process:
{noformat}
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=/path/to/keystore.jks
-Dzookeeper.ssl.keyStore.password=password
-Dzookeeper.ssl.trustStore.location=/path/to/truststore.jks
-Dzookeeper.ssl.trustStore.password=password
{noformat}
KeyStore is only needed if ZooKeeper server wants client certificate to be
provided.
I'd like to add these options to hbase-site.xml in the following way:
{noformat}
hbase.zookeeper.clientCnxnSocket=...
hbase.zookeeper.client.secure=...
hbase.zookeeper.ssl.keyStore=...
...{noformat}
It will follow the way that we already do for ZooKeeper clientPort and quorum
settings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27810) HBCK throws RejectedExecutionException when closing ZooKeeper resources
Andor Molnar created HBASE-27810: Summary: HBCK throws RejectedExecutionException when closing ZooKeeper resources Key: HBASE-27810 URL: https://issues.apache.org/jira/browse/HBASE-27810 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 2.5.4, 2.4.17, 3.0.0-alpha-3, 2.6.0 Reporter: Andor Molnar Assignee: Andor Molnar HBCK throws RejectedExecutionException at the end of run, because the order of closing ZooKeeper resources has been swapped in HBASE-27426. In ZKWatcher.java close() method first it shuts down the zkEventProcessor and when it fully shut down, it closes the RecoverableZooKeeper (the ZK client). The watcher receives the close event which cannot be submitted to the event processor and throws exception. I think we need to check whether the executor is able to receive jobs before submitting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27538) Document auto cert file reloading HBASE-27347
Andor Molnar created HBASE-27538: Summary: Document auto cert file reloading HBASE-27347 Key: HBASE-27538 URL: https://issues.apache.org/jira/browse/HBASE-27538 Project: HBase Issue Type: Task Components: documentation Reporter: Andor Molnar Assignee: Andor Molnar Documentation hasn't been updated in HBASE-27347 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-26667) Integrate user-experience for hbase-client
[
https://issues.apache.org/jira/browse/HBASE-26667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andor Molnar reopened HBASE-26667:
--
> Integrate user-experience for hbase-client
> --
>
> Key: HBASE-26667
> URL: https://issues.apache.org/jira/browse/HBASE-26667
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Andor Molnar
>Priority: Major
> Fix For: HBASE-26553
>
>
> Today, we have two mechanism in order to get the tokens needed to
> authenticate:
> # Kerberos, we rely on a Kerberos ticket being present in a well-known
> location (defined by JVM properties) or via programmatic invocation of
> UserGroupInformation
> # Delegation tokens, we rely on special API to be called (our mapreduce API)
> which loads the token into the current UserGroupInformation "context" (the
> JAAS PrivilegedAction).
> The JWT bearer token approach is very similar to the delegation token
> mechanism, but HBase does not generate this JWT (as we do with delegation
> tokens). How does a client provide this token to the hbase-client (i.e.
> {{ConnectionFactory.getConnection()}} or a {{UserGroupInformation}} call)? We
> should be mindful of all of the different "entrypoints" to HBase ({{{}hbase
> ...{}}} commands, {{java -cp}} commands, Phoenix commands, Spark comands,
> etc). Our solution should be effective for all of these approaches and not
> require downstream changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27347) Port FileWatcher from ZK to autodetect keystore/truststore changes in TLS connections
Andor Molnar created HBASE-27347: Summary: Port FileWatcher from ZK to autodetect keystore/truststore changes in TLS connections Key: HBASE-27347 URL: https://issues.apache.org/jira/browse/HBASE-27347 Project: HBase Issue Type: New Feature Components: IPC/RPC, security Reporter: Andor Molnar Assignee: Andor Molnar As a follow-up feature I'd like to port the file watcher functionality from ZooKeeper in order to avoid restarts when keystore/truststore is updated on a running cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27346) Autodetect key/truststore file type from file extension
Andor Molnar created HBASE-27346: Summary: Autodetect key/truststore file type from file extension Key: HBASE-27346 URL: https://issues.apache.org/jira/browse/HBASE-27346 Project: HBase Issue Type: Improvement Components: security Reporter: Andor Molnar Assignee: Andor Molnar Noticed that file type autodetection hasn't been properly ported from ZooKeeper although the comment says otherwise. Instead of defaulting to JKS we should check the file extension. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27342) Use Hadoop Credentials API to retrieve passwords of TLS key/trust stores
Andor Molnar created HBASE-27342: Summary: Use Hadoop Credentials API to retrieve passwords of TLS key/trust stores Key: HBASE-27342 URL: https://issues.apache.org/jira/browse/HBASE-27342 Project: HBase Issue Type: Improvement Components: IPC/RPC, security Reporter: Andor Molnar Assignee: Andor Molnar Based on a discussion in the TLS Jira and mailing list, it would be beneficial to protect the password of trust and key stores for TLS encryption support in Netty RPC. [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27337) [hbase-thirdparty] Add nimbus-jose-jwt library to hbase-shaded-miscellaneous
Andor Molnar created HBASE-27337: Summary: [hbase-thirdparty] Add nimbus-jose-jwt library to hbase-shaded-miscellaneous Key: HBASE-27337 URL: https://issues.apache.org/jira/browse/HBASE-27337 Project: HBase Issue Type: Task Components: thirdparty Reporter: Andor Molnar Assignee: Andor Molnar Required for OAuthBearer SASL auth plugin to work properly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27226) Document native TLS support in Netty RPC
Andor Molnar created HBASE-27226:
Summary: Document native TLS support in Netty RPC
Key: HBASE-27226
URL: https://issues.apache.org/jira/browse/HBASE-27226
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Andor Molnar
Assignee: Andor Molnar
Add a new section to the HBase book on how a developer can get this going.
Should include
* relevant TLS properties added in X509Util.java which need to be added to
hbase-site.xml
* how to generate a self-signed CA and certs using
{{{}keytool{}}}/{{{}openssl{}}}
* any known limitations
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27118) Add security headers to Thrift/HTTP server
Andor Molnar created HBASE-27118: Summary: Add security headers to Thrift/HTTP server Key: HBASE-27118 URL: https://issues.apache.org/jira/browse/HBASE-27118 Project: HBase Issue Type: Improvement Components: Thrift Affects Versions: 3.0.0-alpha-2, 2.2.7, 2.1.10, 2.0.6 Reporter: Andor Molnar Assignee: Andor Molnar In order to further improve HBase exposed service for vulnerability scanners, would be nice to implement additional security headers for the Thrift server when HTTP transport is enabled. Similarly to REST and Web UIs, related tickets are attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-26908) Remove warnings from meta replicas feature references in the HBase book
Andor Molnar created HBASE-26908: Summary: Remove warnings from meta replicas feature references in the HBase book Key: HBASE-26908 URL: https://issues.apache.org/jira/browse/HBASE-26908 Project: HBase Issue Type: Task Components: documentation Reporter: Andor Molnar Assignee: Andor Molnar Meta replicas is a new feature in HBase 2.4 and mentioned in "Use with caution" in the docs. Given that the feature and the related "async wal replication for meta" is actively used in production already, I'd like to remove these warnings from the docs. With this change, users will have more confidence in the feature. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26789) Automatically add default security headers to http/rest if SSL enabled
Andor Molnar created HBASE-26789: Summary: Automatically add default security headers to http/rest if SSL enabled Key: HBASE-26789 URL: https://issues.apache.org/jira/browse/HBASE-26789 Project: HBase Issue Type: Improvement Components: REST, UI Affects Versions: 3.0.0-alpha-2, 2.2.7, 2.1.10, 2.0.6 Reporter: Andor Molnar Assignee: Andor Molnar In the previous ticket https://issues.apache.org/jira/browse/HBASE-23303 we implemented these security headers as optional which had to explicitly enabled in the config. With this change the headers will automatically be added with meaningful default values if SSL is enabled. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26655) Initial commit with basic functionality and example code
Andor Molnar created HBASE-26655: Summary: Initial commit with basic functionality and example code Key: HBASE-26655 URL: https://issues.apache.org/jira/browse/HBASE-26655 Project: HBase Issue Type: Sub-task Components: security Reporter: Andor Molnar Assignee: Andor Molnar -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26553) OAuth Bearer authentication mech plugin for SASL
Andor Molnar created HBASE-26553: Summary: OAuth Bearer authentication mech plugin for SASL Key: HBASE-26553 URL: https://issues.apache.org/jira/browse/HBASE-26553 Project: HBase Issue Type: New Feature Components: security Reporter: Andor Molnar Assignee: Andor Molnar Implementation of a new SASL plugin to add support for OAuth Bearer token authentication for HBase client RPC. * The plugin supports secured (cryptographically signed) JSON Web Token authentication as defined in [RFC-7628|https://datatracker.ietf.org/doc/html/rfc7628] and the JWT format in [RFC-7519|https://datatracker.ietf.org/doc/html/rfc7519] . * The implementation is inspired by [Apache Kafka's OAuth Bearer token|https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_oauth.html] support with the important difference that HBase version is intended for production usage. The two main differences are that Kafka supports unsecured tokens only and it issues the tokens for itself which breaks the principle of OAuth token authentication. * We use the [Nimbus JOSE + JWT|https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/] Java library for signature verification and token processing and we add it as a new dependency to HBase. * We add secure JWT support and verification of digital signatures with multiple algorithms as supported by Nimbus. Json-formatted JWK set is required for the signature verification as defined in [RFC-7517|https://datatracker.ietf.org/doc/html/rfc7517]. * The impl is verified with Apache Knox issued tokens, because that's the primary use case of this new feature. * New client example is added to the hbase-examples project to showcase the feature. * It's important that this Jira does not cover the solution for obtaining a token from Knox. The assumption is that the client already has a valid token in base64 encoded string and we only provide a helper method for adding it to user's credentials. * Renewing expired tokens is also the responsibility of the client. We don't provide a mechanism for that in this Jira, but it's planned to be covered in a follow-up ticket. The following new parameters are introduced in hbase-site.xml: * hbase.security.oauth.jwt.jwks.file - Path of a local file for JWK set. (required if URL not specified) * hbase.security.oauth.jwt.jwks.url - URL to download the JWK set. (required if File not specified) * hbase.security.oauth.jwt.requiredaudience - Required audience of the JWT. (optional) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-25146) Add extra logging at info level to HFileCorruptionChecker in order to report progress
Andor Molnar created HBASE-25146: Summary: Add extra logging at info level to HFileCorruptionChecker in order to report progress Key: HBASE-25146 URL: https://issues.apache.org/jira/browse/HBASE-25146 Project: HBase Issue Type: Improvement Components: hbck, hbck2 Reporter: Andor Molnar Assignee: Andor Molnar Currently there's no progress reporting in HFileCorruptionChecker: neither in the logs nor in stdout. It only creates a report about the entire operation at the end of the process and emits some warning messages is corruption is found. Adding some logging about the progress would be beneficial for long running checks indicating that the process is running healthy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23303) Add security headers to REST server/info page
Andor Molnar created HBASE-23303:
Summary: Add security headers to REST server/info page
Key: HBASE-23303
URL: https://issues.apache.org/jira/browse/HBASE-23303
Project: HBase
Issue Type: Improvement
Components: REST
Affects Versions: 2.2.2, 2.1.7, 2.0.6, 3.0.0
Reporter: Andor Molnar
Assignee: Andor Molnar
Vulnerability scanners suggest that the following extra headers should be added
to both Info/Rest server endpoints which are exposed by {{hbase-rest}} project.
* X-Content-Type-Options: nosniff
* X-XSS-Protection: 1; mode=block
* X-Frame-Options: SAMEORIGIN
Info server already has "X-Frame-Options: DENY" which is more restrictive than
"SAMEORIGIN", so it's probably fine. All of three headers are missing from REST
responses.
I'll put together a patch to resolve this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-22759) Add user info to AUDITLOG events when doing grant/revoke
Andor Molnar created HBASE-22759:
Summary: Add user info to AUDITLOG events when doing grant/revoke
Key: HBASE-22759
URL: https://issues.apache.org/jira/browse/HBASE-22759
Project: HBase
Issue Type: Improvement
Components: logging, security
Affects Versions: 2.1.5, 2.2.0, 3.0.0
Reporter: Andor Molnar
Assignee: Andor Molnar
Fix For: 3.0.0, 2.3.0, 2.2.1, 2.1.6
On *branch-2.1* the AUDITLOG events is raised like this:
{noformat}
AUDITLOG.trace("Granted permission " + perm.toString());{noformat}
I'd like to extend this line with "caller" user info like this:
{noformat}
AUDITLOG.trace("User {} granted permission {}", caller,
perm.toString());{noformat}
Similar change is proposed for Revoke event.
On branch-2.2+ grant() and revoke() methods in AccessController have been
deprecated and logic was moved to {{MasterRpcServices}}, but that class doesn't
do any audit logging. I'm not sure about why audit logging has been removed and
about any replacement in the refactored logic, but Audit logging is a crucial
security tool in our environment to track change events on ACLs.
I'm planning to add AUDITLOG to {{MasterRpcServices}} to bring back this
functionality, but please FIXME and point me in the right direction if needed.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (HBASE-22382) Refactor tests in TestFromClientSide
Andor Molnar created HBASE-22382:
Summary: Refactor tests in TestFromClientSide
Key: HBASE-22382
URL: https://issues.apache.org/jira/browse/HBASE-22382
Project: HBase
Issue Type: Task
Components: test
Reporter: Andor Molnar
Assignee: Andor Molnar
The following tests in {{TestFromClientSide}} needs to be refactored:
- {{testNull}} - should be several tests instead of one,
- {{testVersionLimits}} - is too long, should be split into multiple,
- {{testDeletesWithReverseScan}} - is too long, should be split into multiple
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
