Hi Darpan -
The following topology is probably a better starting point for you AD
configuration - I've tried to merge yours with it as best I can:
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapContextFactory</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<!-- ADJUST host, port for your AD setup-->
<value>ldap://IP_OF_WINDOWS_AD:389</value>
</param>
<!-- ignored due to use of
main.ldapRealm.userSearchAttributeName -->
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>cn={0},CN=users,DC=test,DC=com</value>
<!-- also tried following values -->
<value>uid={0},CN=users,DC=test,DC=com</value>
<value>cn={0},DC=test,DC=com</value>
</param>
<!-- Param above is ignored sAMAccount is usually used for AD -->
<param>
<name>main.ldapRealm.userSearchAttributeName</name>
<value>sAMAccountName</value>
</param>
<!-- adjust as appropriate -->
<param>
<name>main.ldapRealm.userObjectClass</name>
<value>person</value>
</param>
<!-- adjust the dn below to match your environment -->
<param>
<name>main.ldapRealm.contextFactory.systemUsername</name>
<value>cn={systemuser},ou=process,ou=accounts,dc=test,dc=com</value>
</param>
<!-- should be moved to the credential store for the gateway to be more
secure -->
<param>
<name>main.ldapRealm.contextFactory.systemPassword</name>
<value>{systemuser_password}/value>
</param>
<!-- let's disable for now since you have no authorization
policies defined anyway -->
<param>
<name>main.ldapRealm.authorizationEnabled</name>
<value>false</value>
</param>
<param>
<name>main.ldapRealm.searchBase</name>
<value>cn=users,dc=test,dc=com</value>
</param>
<param>
<param>
<name>main.ldapRealm.memberAttributeValueTemplate</name>
<value>cn={0},cn=users,dc=test,dc=com</value>
<!-- also tried uid={0} -->
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<!-- the group principal mapping below is not likely what you want
note that mapping of the hdfs group to admin. Also, we have
disabled authorization above so there is no need for groups
-->
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
<!--param>
<name>group.principal.mapping</name>
<value>*=users;hdfs=admin</value>
</param-->
</provider>
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
</provider>
</gateway>
We need to better document the difference between LDAP and AD for such
deployments.
I've also tried to document some of the changes that I made.
Note that you don't have any authorization ACLs defined in the AclsAuthz
provider so I disabled group lookup.
That will only add complexity to your config - we can re-enable once
authentication is working.
Please go through this config and ensure that DNs, host and ports and
system usernames match your environment.
Hope this helps.
--larry
On Tue, Dec 8, 2015 at 5:16 AM, Darpan Patel <[email protected]> wrote:
> Hi All,
>
> For this blocker issue let more information if it can help fixing the
> authorization problem.
> Please let me know if more details required.
> (+ dev list)
>
> */etc/krb5.conf*
>
> [libdefaults]
> renew_lifetime = 7d
> forwardable = true
> default_realm = HORTONWORKS.COM
> ticket_lifetime = 24h
> dns_lookup_realm = false
> dns_lookup_kdc = false
> #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
> #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
>
> [domain_realm]
> .hortonworks.com = HORTONWORKS.COM
> HORTONWORKS.COm = HORTONWORKS.COM
>
> [logging]
> default = FILE:/var/log/krb5kdc.log
> admin_server = FILE:/var/log/kadmind.log
> kdc = FILE:/var/log/krb5kdc.log
>
> [realms]
> HORTONWORKS.COM = {
> admin_server = KDC_SERVER_HOST
> kdc = KDC_SERVER_HOST
> }
> *TEST.COM <http://TEST.COM>* = {
> admin_server = WINDOWS_12_SERVER_AD_HOST
> kdc = WINDOWS_12_SERVER_AD_HOST
> }
>
>
> */usr/hdp/current/knox-server/conf/gateway-site.xml*
>
> <configuration>
> <property>
> <name>*gateway.gateway.conf.dir*</name>
> <value>deployments</value>
> </property>
> <property>
> <name>*gateway.hadoop.kerberos.secured*</name>
> <value>true</value>
> </property>
> <property>
> <name>*gateway.path*</name>
> <value>gateway</value>
> </property>
> <property>
> <name>*gateway.port*</name>
> <value>8443</value>
> </property>
> <property>
> <name>*java.security.auth.login.config*</name>
> <value>/*etc/knox/conf/krb5JAASLogin.conf*</value>
> </property>
> <property>
> <name>*java.security.krb5.conf*</name>
> <value>*/etc/krb5.conf*</value>
> </property>
> <property>
> <name>sun.security.krb5.debug</name>
> <value>true</value>
> </property>
> </configuration>
>
>
> */etc/knox/conf/krb5JAASLogin.conf*
>
> com.sun.security.jgss.initiate {
> com.sun.security.auth.module.Krb5LoginModule required
> renewTGT=true
> doNotPrompt=true
> useKeyTab=true
> keyTab="/etc/security/keytabs/knox.service.keytab"
> principal="knox/[email protected]"
> isInitiator=true
> storeKey=true
> useTicketCache=true
> client=true;
> };
>
> Regards,
> DP
>
> ---------- Forwarded message ----------
> From: Darpan Patel <[email protected]>
> Date: 7 December 2015 at 17:59
> Subject: Need help setting up Knox for A/D integrated Kerberized Cluster
> To: [email protected]
>
>
> Hi All,
>
> I am stuck on an issue from last two days. I would be really grateful if
> someone can help on this.
>
> We have HDP 2.3 implemented over 8 node cluster and the same cluster has
> been Kerberized and later on we have integrated it with Active Directory
> (Which runs in the same VPN). We also verified that Windows 2012 A/D
> integration with Ranger works fine for defining policies and audit log. But
> I am stuck at Knox bit. I am trying to replicate the same configuration
> properties which I have set for Ranger LDAP-AD Integration.
>
> I am taking reference of the Hortonworks documentation and also Apache
> Knox documentation.
>
> The A/D domain name is TEST.COM and all the users are under Users
>
> [image: Inline images 1]
>
>
> Under the Users we have few users one of the them is knox, darpan,
> test,etc.
>
> When we issue following command on the node on which Knox Server is
> running (topology name is default)
>
>
>
> *curl -iv -k -u [email protected]:#123Password -X GET
> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS
> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>" OR*
>
> *curl -iv -k -u knox:#123Password -X GET
> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS
> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>"*
>
>
> Every time I see < HTTP/1.1 401 Unauthorized HTTP/1.1 401 Unauthorized on
> the console.
>
>
> Entries in the *gateway-audit.log *are like this :
>
> gateway-audit.log
> ==================
> 15/12/07 17:11:08
> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable|
> 15/12/07 17:11:09
> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||authentication|
> *principal*|*[email protected] <[email protected]>*|failure|*LDAP authentication
> failed.*
> 15/12/07 17:11:09
> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response
> status: 401
>
>
> 15/12/07 17:05:28
> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable|
> 15/12/07 17:05:29
> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||authentication|
> *principal*|knox|failure|*LDAP authentication failed.*
> 15/12/07 17:05:29
> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response
> status: 401
>
>
> *Gateway.log*
> *===========*
> 2015-12-07 17:05:28,620 INFO hadoop.gateway
> (KnoxLdapRealm.java:getUserDn(550)) - Computed userDn:
> cn=knox,CN=users,DC=test,DC=com using dnTemplate for principal: knox
>
>
> Following is the part of our *default.xml *topology:
>
>
> <gateway>
> <provider>
> <role>authentication</role>
> <name>ShiroProvider</name>
> <enabled>true</enabled>
> <param>
> <name>sessionTimeout</name>
> <value>30</value>
> </param>
> <param>
> <name>*main.ldapRealm*</name>
>
> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
> </param>
>
> <param>
> <name>*main.ldapContextFactory*</name>
>
> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
> </param>
>
> <param>
> <name>*main.ldapRealm.userDnTemplate*</name>
> <value>cn={0},CN=users,DC=test,DC=com</value>
> <!-- also tried following values -->
> <value>uid={0},CN=users,DC=test,DC=com</value>
> <value>cn={0},DC=test,DC=com</value>
> </param>
> <param>
> <name>*main.ldapRealm.contextFactory.url*</name>
> <!-- IP Address of the WINDOSWS 2012 Acive
> Directory Server which works for Ranger -->
> <value>*ldap://IP_OF_WINDOWS_AD:389*</value>
> </param>
> <param>
> <name>*main.ldapRealm.authorizationEnabled*</name>
> <value>true</value>
> </param>
> <param>
> <name>*main.ldapRealm.searchBase*</name>
> <value>cn=users,dc=test,dc=com</value>
> </param>
> <param>
> <param>
> <name>*main.ldapRealm.memberAttributeValueTemplate*
> </name>
> <value>cn={0},cn=users,dc=test,dc=com</value>
> <!-- also tried uid={0} -->
> </param>
> <param>
> <name>
> *main.ldapRealm.contextFactory.authenticationMechanism<*/name>
> <value>simple</value>
> </param>
> <param>
> <name>urls./**</name>
> <value>authcBasic</value>
> </param>
> </provider>
>
> <provider>
> <role>*identity-assertion*</role>
> <name>Default</name>
> <enabled>true</enabled>
> <param>
> <name>*group.principal.mapping*</name>
> <value>*=users;hdfs=admin</value>
> </param>
> </provider>
>
> <provider>
> <role>*authorization*</role>
> <name>AclsAuthz</name>
> <enabled>true</enabled>
> </provider>
>
> </gateway>
>
>
> And following is the console output while trying to access webhdfs using
> curl
>
> curl -iv -k -u knox:#123Password -X GET "
> https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS";
>
>
> *Console Output:*
> ----------------
>
> * About to connect() to localhost port 8443 (#0)
> * Trying ::1...
> * Connected to localhost (::1) port 8443 (#0)
> * Initializing NSS with certpath: sql:/etc/pki/nssdb
> * skipping SSL peer certificate verification
> * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
> * Server certificate:
> * subject:
> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
> * start date: Nov 27 20:36:22 2015 GMT
> * expire date: Nov 26 20:36:22 2016 GMT
> * common name: FQDN_OF_My_gateway_HOST
> * issuer:
> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
> * Server auth using Basic with user 'knox'
> > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1
> > Authorization: Basic a25veDojMTIzUGFzc3dvcmQ=
> > User-Agent: curl/7.29.0
> > Host: localhost:8443
> > Accept: */*
> >
> < HTTP/1.1 401 Unauthorized
> HTTP/1.1 401 Unauthorized
> * Authentication problem. Ignoring this.
> < WWW-Authenticate: BASIC realm="application"
> WWW-Authenticate: BASIC realm="application"
> < Content-Length: 0
> Content-Length: 0
> < Server: Jetty(8.1.14.v20131031)
> Server: Jetty(8.1.14.v20131031)
>
>
> Please let me know if any additional information is required.
>
> Thanks,
> DP
>
>
