I can't quite tell whether this part is successful or not: sudo curl --negotiate -u : -ivk https://{knoxhost}:8443/ gateway/default/hbase/version/rest
This returns "rest 0.0.3..." It seems like it was - so I assume that you did set up Knox to be kerberized to the backend services even though you don't seem to provide those steps in your mail. Just to be clear though... did you? Perhaps the version call doesn't require SPNEGO? I'd also like to understand why you are going through Knox but still requiring kerberos. Also, you cannot remove identity assertion providers in 0.12.0 - it may grab a random provider to use. I made this provider truly default to something in 0.14.0 prior to that it is a mandatory provider in every topology. If you already have Knox properly kerberized then I would start by adding the identity assertion provider back. On Fri, Dec 15, 2017 at 5:05 PM, Rick Kellogg <[email protected]> wrote: > Problem Description: > > > > Unable to successfully communicate via Curl to HBase via Knox using > Kerberos secured environment. > > > > Environment: HDP 2.6.0.3 (Knox 0.12.0 & HBase 1.1.2) > > Kerberos Protected Environment > > RHEL 6.x > > Oracle JDK 1.8.0_91 > > > > I have deployed a standalone instance of HDP 2.6.0.3 with Kerberos > enabled. Can confirm access to HBase is > > operational based on tests using HBase Shell. > > > > Secure access to HBase via Curl and Knox requires several prerequisites: > > > > * Configuration of HBASE REST Server > > * Startup of HBase REST Server with Kerberos credentials > > * Configuration of Apache Knox > > > > HBASE REST SERVER CONFIGURATION > > > > The Apache HBase REST Server is not managed via Ambari. As such it must > be configured and started manually. > > Incomplete documentation can be found within the HBase Reference Guide: > http://hbase.apache.org/book.html#_rest. > > One crucial omission is the requirement for a Kerberos kinit prior to > launch of the HBase REST daemon. After > > much frustration, I discovered the following HortonWorks Article which was > very helpful "HOWTO: Start and Test > > HBase REST Server in a kerberized environment". > > > > https://community.hortonworks.com/articles/91425/howto- > start-and-test-hbase-rest-server-in-a-kerber.html > > > > Changes listed in this article were completed. I also needed to add the > following to start up the REST Server > > without errors: > > > > hbase.rest.keytab.file=/etc/security/keytabs/hbase.service.keytab > > > > > > HBASE REST SERVER STARTUP > > > > From one of the HBase Region Servers, determine the principal listed in > the hbase.service.keytab file. > > > > sudo ktlist -kt /etc/security/keytabs/hbase.service.keytab > > > > Default principal should be of the form: hbase/{hostname}@{DOMAIN.COM} > > > > Purge existing Kerberos tickets using: > > > > sudo kdestroy > > > > Obtain Kerberos ticket using: > > > > sudo kinit hbase/{hostname}@{DOMAIN.COM} -kt /etc/security/keytabs/hbase. > service.keytab > > > > Startup HBASE REST Server using the following: > > > > sudo /usr/hdp/current/hbase-master/bin/hbase-daemon.sh start rest -p 8080 > > > > Review HBASE REST Server logs (typically found in > /var/log/hbase/hbase-root*.log) for any errors. > > > > > > VERIFY KERBEROS ACCESS TO HBASE VIA HBASE REST SERVER > > > > Using a different HBase Region Server, determine the principals listed in > the hbase.service.keytab file. > > > > sudo ktlist -kt /etc/security/keytabs/hbase.service.keytab > > > > Default principal should be of the form: hbase/{hostname}@{DOMAIN.COM} > > > > Purge existing Kerberos tickets using: > > > > sudo kdestroy > > > > Obtain Kerberos ticket using: > > > > sudo kinit hbase/{hostname}@{DOMAIN.COM} -kt /etc/security/keytabs/hbase. > service.keytab > > > > Verify connectivity to HBase REST Server > > > > sudo curl --negotiate -u : -ivk http://{hostname}:8080/version/rest > > This returns "rest 0.0.3..." > > > > Verify connectivity to an existing HBase table > > > > sudo curl --negotiate -u : -ivk http://{hostname}:8080/{ > existingtable}/schema > > This returns metadata > > > > > > CONFIGURE KNOX > > > > Within Knox Advanced Topology, disable ShireProvider and > identity-assertion providers and then add > > HadoopAuth authentication provider using example in Knox User Guide > > (http://knox.apache.org/books/knox-0-14-0/user-guide.html# > HadoopAuth+Authentication+Provider). > > The hadoop.auth.kerberos.principal is set to a value of the form: > > HTTP/{knoxhost}@{DOMAIN.COM}. The hadoop.auth.config.kerberis.keytab is > set to: > > /etc/security/keytabs/spnego.service.keytab > > > > Restart Knox > > > > VERIFY KNOX TO HBASE CONNECTIVITY > > > > Using a different HBase Region Server establish a Kerberos ticket: > > > > sudo kdestroy > > sudo kinit hbase/{hostname}@{DOMAIN.COM} -kt /etc/security/keytabs/hbase. > service.keytab > > > > Verify connectivity to HBase REST Server via Knox > > > > sudo curl --negotiate -u : -ivk https://{knoxhost}:8443/ > gateway/default/hbase/version/rest > > This returns "rest 0.0.3..." > > > > The following attempt to retrieve HBase metadata fails: > > > > sudo curl --negotiate -u : -ivk https://{knoxhost}:8443/ > gateway/default/hbase/{existingtable}/schema > > Process hangs > > > > > > Any ideas what might be going on here? Have I missed any important steps? > > > > Thanks, > > Rick Kellogg > > > > >
