Thanks. See inline below
From: larry mccay [mailto:[email protected]] Sent: Friday, December 15, 2017 5:38 PM To: [email protected] Subject: Re: Issue with Knox/HBase Kerberos integration I can't quite tell whether this part is successful or not: sudo curl --negotiate -u : -ivk https://{knoxhost}:8443/gateway/default/hbase/version/rest <https://%7bknoxhost%7d:8443/gateway/default/hbase/version/rest> This returns "rest 0.0.3..." RK – Yes this was successful and correct. It seems like it was - so I assume that you did set up Knox to be kerberized to the backend services even though you don't seem to provide those steps in your mail. Just to be clear though... did you? RK – We used the Kerberos feature within Ambari to protect the cluster. I then did a kinit and ran the HBase Shell. From there I can see my tables, etc. Perhaps the version call doesn't require SPNEGO? I'd also like to understand why you are going through Knox but still requiring kerberos. RK – We want to use Kerberos all the way from our Tomcat client through Knox to the HBase REST Server to HBase. All the back end services are Kerberos protected. Configuration of the HBase REST proxy and HBase was tricky and poorly documented. Also, you cannot remove identity assertion providers in 0.12.0 - it may grab a random provider to use. I made this provider truly default to something in 0.14.0 prior to that it is a mandatory provider in every topology. If you already have Knox properly kerberized then I would start by adding the identity assertion provider back. RK - Thanks for the suggestion. It might be something quite simple. Will test again on Monday when I return to work. Thanks for your help. Rick On Fri, Dec 15, 2017 at 5:05 PM, Rick Kellogg <[email protected] <mailto:[email protected]> > wrote: Problem Description: Unable to successfully communicate via Curl to HBase via Knox using Kerberos secured environment. Environment: HDP 2.6.0.3 (Knox 0.12.0 & HBase 1.1.2) Kerberos Protected Environment RHEL 6.x Oracle JDK 1.8.0_91 I have deployed a standalone instance of HDP 2.6.0.3 with Kerberos enabled. Can confirm access to HBase is operational based on tests using HBase Shell. Secure access to HBase via Curl and Knox requires several prerequisites: * Configuration of HBASE REST Server * Startup of HBase REST Server with Kerberos credentials * Configuration of Apache Knox HBASE REST SERVER CONFIGURATION The Apache HBase REST Server is not managed via Ambari. As such it must be configured and started manually. Incomplete documentation can be found within the HBase Reference Guide: http://hbase.apache.org/book.html#_rest. One crucial omission is the requirement for a Kerberos kinit prior to launch of the HBase REST daemon. After much frustration, I discovered the following HortonWorks Article which was very helpful "HOWTO: Start and Test HBase REST Server in a kerberized environment". https://community.hortonworks.com/articles/91425/howto-start-and-test-hbase-rest-server-in-a-kerber.html Changes listed in this article were completed. I also needed to add the following to start up the REST Server without errors: hbase.rest.keytab.file=/etc/security/keytabs/hbase.service.keytab HBASE REST SERVER STARTUP >From one of the HBase Region Servers, determine the principal listed in the >hbase.service.keytab file. sudo ktlist -kt /etc/security/keytabs/hbase.service.keytab Default principal should be of the form: hbase/{hostname}@{DOMAIN.COM <http://DOMAIN.COM> } Purge existing Kerberos tickets using: sudo kdestroy Obtain Kerberos ticket using: sudo kinit hbase/{hostname}@{DOMAIN.COM <http://DOMAIN.COM> } -kt /etc/security/keytabs/hbase.service.keytab Startup HBASE REST Server using the following: sudo /usr/hdp/current/hbase-master/bin/hbase-daemon.sh start rest -p 8080 Review HBASE REST Server logs (typically found in /var/log/hbase/hbase-root*.log) for any errors. VERIFY KERBEROS ACCESS TO HBASE VIA HBASE REST SERVER Using a different HBase Region Server, determine the principals listed in the hbase.service.keytab file. sudo ktlist -kt /etc/security/keytabs/hbase.service.keytab Default principal should be of the form: hbase/{hostname}@{DOMAIN.COM <http://DOMAIN.COM> } Purge existing Kerberos tickets using: sudo kdestroy Obtain Kerberos ticket using: sudo kinit hbase/{hostname}@{DOMAIN.COM <http://DOMAIN.COM> } -kt /etc/security/keytabs/hbase.service.keytab Verify connectivity to HBase REST Server sudo curl --negotiate -u : -ivk http://{hostname}:8080/version/rest <http://%7bhostname%7d:8080/version/rest> This returns "rest 0.0.3..." Verify connectivity to an existing HBase table sudo curl --negotiate -u : -ivk http://{hostname}:8080/{existingtable}/schema <http://%7bhostname%7d:8080/%7bexistingtable%7d/schema> This returns metadata CONFIGURE KNOX Within Knox Advanced Topology, disable ShireProvider and identity-assertion providers and then add HadoopAuth authentication provider using example in Knox User Guide (http://knox.apache.org/books/knox-0-14-0/user-guide.html#HadoopAuth+Authentication+Provider). The hadoop.auth.kerberos.principal is set to a value of the form: HTTP/{knoxhost}@{DOMAIN.COM <http://DOMAIN.COM> }. The hadoop.auth.config.kerberis.keytab is set to: /etc/security/keytabs/spnego.service.keytab Restart Knox VERIFY KNOX TO HBASE CONNECTIVITY Using a different HBase Region Server establish a Kerberos ticket: sudo kdestroy sudo kinit hbase/{hostname}@{DOMAIN.COM <http://DOMAIN.COM> } -kt /etc/security/keytabs/hbase.service.keytab Verify connectivity to HBase REST Server via Knox sudo curl --negotiate -u : -ivk https://{knoxhost}:8443/gateway/default/hbase/version/rest <https://%7bknoxhost%7d:8443/gateway/default/hbase/version/rest> This returns "rest 0.0.3..." The following attempt to retrieve HBase metadata fails: sudo curl --negotiate -u : -ivk https://{knoxhost}:8443/gateway/default/hbase/{existingtable}/schema <https://%7bknoxhost%7d:8443/gateway/default/hbase/%7bexistingtable%7d/schema> Process hangs Any ideas what might be going on here? Have I missed any important steps? Thanks, Rick Kellogg
