Good questions. Looks the API hasn't changed since when it was added in 2014: https://github.com/Jerry-Xin/hadoop/commit/c79728478caadd8374bce2bc3f466db1da1e3ad1
I'll create a patch to support this, apart from the CLI extension this looks like super easy to integrate. Filed jira: https://issues.apache.org/jira/browse/HBASE-27342 Andor On Thu, 2022-08-25 at 20:15 +0200, Nick Dimiduk wrote: > Is the Hadoop Credentials API protected by their compatibility > guidelines? > If not, can we convince them to support it formally? > > On Thu, Aug 25, 2022 at 17:15 Andrew Purtell < > andrew.purt...@gmail.com> > wrote: > > > We can import the code from our sister Apache project if long term > > dependency and compatibility are concerns. The concerns apply in > > both > > directions: > > > > Depend, do not import: Hadoop may break us with a change that we > > have to > > incorporate by updating to a new minimum version probably because > > of a > > security issue. How likely is this? I don’t think the SPI has > > changed much. > > Cannot do a decent review at this time, on phone only. > > > > Import: If we do not watch the Hadoop implementation, then our > > semantics > > may diverge from theirs in a way that is confusing and inconvenient > > for > > operators who have to manage credentials at both Hadoop and HBase > > layers. > > > > > On Aug 25, 2022, at 11:00 AM, 张铎 <palomino...@gmail.com> wrote: > > > > > > +1 > > > > > > But I'm still not sure whether we should just use the code in > > > hadoop, > > > or we should just use the mechanism but write(copy) the code by > > > our > > > own? > > > > > > Andrew Purtell <andrew.purt...@gmail.com> 于2022年8月25日周四 22:13写道： > > > > I agree the Credential SPI provided by Hadoop is direct and > > > > expedient. > > > > > > > > I would ask that a patch integrating it, if this is the > > > > selected > > approach, should also add support to bin/hbase so “hbase credential > > …” > > command line support is available and identical to that provided by > > the > > Hadoop bin script. This is for convenience and also a concession to > > users > > that ship HBase binaries/packages disaggregated from Hadoop ones. > > > > > > On Aug 25, 2022, at 9:50 AM, Andor Molnar <an...@apache.org > > > > > > > wrote: > > > > > > > > > > As Bryan mentioned there's a nice, extensible API already > > > > > available in > > > > > Hadoop, the Credentials API. > > > > > > > > > > > > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html > > > > > I think that would be the quickest and easiest approach to > > > > > resolve this > > > > > problem. Is there any objection or downside of that? > > > > > > > > > > Andor > > > > > > > > > > > > > > > > > > > > > On Tue, 2022-08-23 at 21:39 +0800, 张铎(Duo Zhang) wrote: > > > > > > Maybe we could introduce two configs for a password, > > > > > > password- > > > > > > provider > > > > > > and password-provider-parameter > > > > > > > > > > > > The default implementation is FileBasedPasswordProvider, > > > > > > where the > > > > > > parameter is just a location. And also an > > > > > > EnvVarPasswordProvider, > > > > > > where the parameter is the name of the environment > > > > > > variable. > > > > > > > > > > > > And users could also implement their own providers. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > Bryan Beaudreault <bbeaudrea...@hubspot.com.invalid> > > > > > > 于2022年8月23日周二 > > > > > > 21:12写道： > > > > > > > I agree that it seems insecure to put it directly into > > > > > > > the hbase- > > > > > > > site.xml. > > > > > > > Another reason is due to the RS UI which (helpfully) can > > > > > > > print the > > > > > > > entire > > > > > > > site configuration. We’d need to make sure the password > > > > > > > is excluded > > > > > > > from > > > > > > > that, but better to remove it from site xml altogether. > > > > > > > > > > > > > > That said, we already have the concept of keystore > > > > > > > passwords in > > > > > > > hbase -- > > > > > > > search our refguide for "password" and you'll see two > > > > > > > examples: for > > > > > > > jmx ssl > > > > > > > and for encryption-at-rest. Both cases seem to take the > > > > > > > approach > > > > > > > of > > > > > > > allowing an explicit password or a password file. Another > > > > > > > example > > > > > > > we can > > > > > > > take inspiration from is Hadoop Credentials API[1] which > > > > > > > allows > > > > > > > specifying > > > > > > > by environment variable or password file. Searching > > > > > > > around for > > > > > > > other > > > > > > > opensource projects, these options seem to be the most > > > > > > > common for > > > > > > > the > > > > > > > keystore password. See Cassandra[2] and Zookeeper[3] as > > > > > > > further > > > > > > > examples. > > > > > > > > > > > > > > Elastic takes the approach of allowing "secure settings" > > > > > > > [4], which > > > > > > > are > > > > > > > stored in a separate keystore managed via elasticsearch- > > > > > > > keystore > > > > > > > command. > > > > > > > This just pushes the problem down a level, as that > > > > > > > keystore needs > > > > > > > to be > > > > > > > password protected as well. In which case, you are > > > > > > > expected to > > > > > > > provide the > > > > > > > path to a password file using an environment variable at > > > > > > > startup > > > > > > > [5]. This > > > > > > > approach is very similar to Hadoop Credentials API. > > > > > > > > > > > > > > Personally I think we should go with the password file > > > > > > > path > > > > > > > approach. This > > > > > > > gives a lot of flexibility, for example one could delete > > > > > > > it after > > > > > > > startup > > > > > > > like mentioned in [5]. I like the idea of providing a > > > > > > > decryption > > > > > > > interface > > > > > > > option for advanced users, but I think we still need to > > > > > > > provide an > > > > > > > option > > > > > > > which doesn't require writing a bunch of code. > > > > > > > > > > > > > > Alternatively I think a case could be made for unifying > > > > > > > on Hadoop's > > > > > > > Credential API. IMO, if we did that, it should be a > > > > > > > separate > > > > > > > initiative > > > > > > > since we'd probably want to unify our existing keystore > > > > > > > configurations into > > > > > > > it and it'd probably need a major version release as a > > > > > > > result. > > > > > > > > > > > > > > [1] > > > > > > > > > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html > > > > > > > [2] > > > > > > > > > https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/configuration/secureSSLClientToNode.html > > > > > > > [3] > > > > > > > https://zookeeper.apache.org/doc/r3.8.0/zookeeperAdmin.html > > > > > > > (search keyStore.password) > > > > > > > [4] > > > > > > > > > https://www.elastic.co/guide/en/elasticsearch/reference/master/secure-settings.html > > > > > > > [5] > > > > > > > > > https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-running-systemd > > > > > > > On Mon, Aug 22, 2022 at 11:33 PM 张铎(Duo Zhang) < > > > > > > > palomino...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > In real production deployment, usually we will store an > > > > > > > > encrypted > > > > > > > > password in the configuration file, and then decrypt it > > > > > > > > after > > > > > > > > loading, > > > > > > > > to actually use it. > > > > > > > > > > > > > > > > And how to get the decryption will depend on the > > > > > > > > environment. On > > > > > > > > cloud > > > > > > > > VMs, usually you can use an encryption service to > > > > > > > > decrypt the > > > > > > > > password. On K8s, you can mount the key using secret. > > > > > > > > > > > > > > > > So maybe we should abstract a decryption interface, so > > > > > > > > users > > > > > > > > could > > > > > > > > implement it on their own to find a suitable way to > > > > > > > > decrypt the > > > > > > > > encrypted password? > > > > > > > > > > > > > > > > Andor Molnar <an...@apache.org> 于2022年8月23日周二 05:55写道： > > > > > > > > > > > > > > > > > Hi team, > > > > > > > > > > > > > > > > > > Netty TLS support is now merged into master and > > > > > > > > > branch-2 > > > > > > > > > branches. > > > > > > > > > Currently keystore/truststore passwords can only be > > > > > > > > > stored in > > > > > > > > > hbase- > > > > > > > > > site.xml which is not the best approach from security > > > > > > > > > perspective. > > > > > > > > > > > > > > > > > > In the docs review Sergey Soldatov mentioned ( > > > > > > > > > https://github.com/apache/hbase/pull/4717/files#r951768699 > > > > > > > > < > > > > > > > > https://github.com/apache/hbase/pull/4717/files#r951768699> > > > > > > > > ;;) > > > > > > > > an approach > > > > > > > > > in HDFS where password can be stored in special files > > > > > > > > > or in > > > > > > > > > environment > > > > > > > > > variables. > > > > > > > > > > > > > > > > > > Sergey, would you please point me to the details of > > > > > > > > > that > > > > > > > > > implementation? Sounds like it would be acceptable > > > > > > > > > for HBase > > > > > > > > > too. > > > > > > > > > > > > > > > > > > Is there any other idea that folks could recommend? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Andor > > > > > > > > > > > > > > > > > > > > > > > > > > >
