Problem Description:
Unable to successfully communicate via Curl to HBase via Knox using Kerberos
secured environment.
Environment: HDP 2.6.0.3 (Knox 0.12.0 & HBase 1.1.2)
Kerberos Protected Environment
RHEL 6.x
Oracle JDK 1.8.0_91
I have deployed a standalone instance of HDP 2.6.0.3 with Kerberos enabled.
Can confirm access to HBase is
operational based on tests using HBase Shell.
Secure access to HBase via Curl and Knox requires several prerequisites:
* Configuration of HBASE REST Server
* Startup of HBase REST Server with Kerberos credentials
* Configuration of Apache Knox
HBASE REST SERVER CONFIGURATION
The Apache HBase REST Server is not managed via Ambari. As such it must be
configured and started manually.
Incomplete documentation can be found within the HBase Reference Guide:
http://hbase.apache.org/book.html#_rest.
One crucial omission is the requirement for a Kerberos kinit prior to launch
of the HBase REST daemon. After
much frustration, I discovered the following HortonWorks Article which was
very helpful "HOWTO: Start and Test
HBase REST Server in a kerberized environment".
https://community.hortonworks.com/articles/91425/howto-start-and-test-hbase-
rest-server-in-a-kerber.html
Changes listed in this article were completed. I also needed to add the
following to start up the REST Server
without errors:
hbase.rest.keytab.file=/etc/security/keytabs/hbase.service.keytab
HBASE REST SERVER STARTUP
>From one of the HBase Region Servers, determine the principal listed in the
hbase.service.keytab file.
sudo ktlist -kt /etc/security/keytabs/hbase.service.keytab
Default principal should be of the form: hbase/{hostname}@{DOMAIN.COM}
Purge existing Kerberos tickets using:
sudo kdestroy
Obtain Kerberos ticket using:
sudo kinit hbase/{hostname}@{DOMAIN.COM} -kt
/etc/security/keytabs/hbase.service.keytab
Startup HBASE REST Server using the following:
sudo /usr/hdp/current/hbase-master/bin/hbase-daemon.sh start rest -p 8080
Review HBASE REST Server logs (typically found in
/var/log/hbase/hbase-root*.log) for any errors.
VERIFY KERBEROS ACCESS TO HBASE VIA HBASE REST SERVER
Using a different HBase Region Server, determine the principals listed in
the hbase.service.keytab file.
sudo ktlist -kt /etc/security/keytabs/hbase.service.keytab
Default principal should be of the form: hbase/{hostname}@{DOMAIN.COM}
Purge existing Kerberos tickets using:
sudo kdestroy
Obtain Kerberos ticket using:
sudo kinit hbase/{hostname}@{DOMAIN.COM} -kt
/etc/security/keytabs/hbase.service.keytab
Verify connectivity to HBase REST Server
sudo curl --negotiate -u : -ivk http://{hostname}:8080/version/rest
This returns "rest 0.0.3..."
Verify connectivity to an existing HBase table
sudo curl --negotiate -u : -ivk
http://{hostname}:8080/{existingtable}/schema
This returns metadata
CONFIGURE KNOX
Within Knox Advanced Topology, disable ShireProvider and identity-assertion
providers and then add
HadoopAuth authentication provider using example in Knox User Guide
(http://knox.apache.org/books/knox-0-14-0/user-guide.html#HadoopAuth+Authent
ication+Provider).
The hadoop.auth.kerberos.principal is set to a value of the form:
HTTP/{knoxhost}@{DOMAIN.COM}. The hadoop.auth.config.kerberis.keytab is set
to:
/etc/security/keytabs/spnego.service.keytab
Restart Knox
VERIFY KNOX TO HBASE CONNECTIVITY
Using a different HBase Region Server establish a Kerberos ticket:
sudo kdestroy
sudo kinit hbase/{hostname}@{DOMAIN.COM} -kt
/etc/security/keytabs/hbase.service.keytab
Verify connectivity to HBase REST Server via Knox
sudo curl --negotiate -u : -ivk
https://{knoxhost}:8443/gateway/default/hbase/version/rest
This returns "rest 0.0.3..."
The following attempt to retrieve HBase metadata fails:
sudo curl --negotiate -u : -ivk
https://{knoxhost}:8443/gateway/default/hbase/{existingtable}/schema
Process hangs
Any ideas what might be going on here? Have I missed any important steps?
Thanks,
Rick Kellogg
