Well when I am keeping the param to the following value we get an error.
<param>
> <name>main.ldapRealm.contextFactory</name>
> <value>$ldapContextFactory</value>
> </param>
>
>
Copying from the gateway.log. (It made me think we need to define the value
for ldapContextFactory)
2015-12-08 22:13:58,003 ERROR env.EnvironmentLoader
(EnvironmentLoader.java:initEnvironment(146)) - Shiro environment
initialization failed
org.apache.shiro.config.UnresolveableReferenceException: *The object with
id [ldapContextFactory] has not yet been defined and therefore cannot be
referenced. * Please ensure objects are defined in the order in which they
should be created and made available for future reference.
at
org.apache.shiro.config.ReflectionBuilder.getReferencedObject(ReflectionBuilder.java:224)
at
org.apache.shiro.config.ReflectionBuilder.resolveReference(ReflectionBuilder.java:239)
Regards,
DP
>
> On Tue, Dec 8, 2015 at 4:59 PM, Darpan Patel <[email protected]> wrote:
>
>> Thanks for the merged template. I made modifications to it and
>>
>> I am not sure what value should I fill for main.ldapRealm.contextFactory ?
>> We are running on windows 2008/2012 Active directory.
>>
>> <param>
>> <name>main.ldapRealm.contextFactory</name>
>> <value>$ldapContextFactory</value>
>> </param>
>>
>>
> I think that you leave it exactly like that.
> It is some sort of shiro injection thing - it references the value defined
> above it that way.
>
>
>> I removed this parameter and I see the in the logs:
>>
>> 2015-12-08 21:56:51,806 ERROR hadoop.gateway
>> (KnoxLdapRealm.java:getUserDn(574)) - Failed to get system ldap connection:
>> javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308:
>> LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e,
>> v1db1]
>>
>>
>> ( I am happy to see new error after 3 days phew!!!)
>>
>>
> Glad that you are happy but let's getting working and see how you feel. :)
> We'll also roll it into some better documentation for the AD specific
> usecase.
>
>
>> Regards,
>> DP
>>
>> On 8 December 2015 at 14:52, Darpan Patel <[email protected]> wrote:
>>
>>> Thanks Larray.
>>> I will check this and update you.
>>>
>>> Regards,
>>> DP
>>>
>>> On 8 December 2015 at 12:18, larry mccay <[email protected]> wrote:
>>>
>>>> Hi Darpan -
>>>>
>>>> The following topology is probably a better starting point for you AD
>>>> configuration - I've tried to merge yours with it as best I can:
>>>>
>>>> <gateway>
>>>> <provider>
>>>> <role>authentication</role>
>>>> <name>ShiroProvider</name>
>>>> <enabled>true</enabled>
>>>> <param>
>>>> <name>sessionTimeout</name>
>>>> <value>30</value>
>>>> </param>
>>>> <param>
>>>> <name>main.ldapRealm</name>
>>>>
>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
>>>> </param>
>>>>
>>>> <param>
>>>> <name>main.ldapContextFactory</name>
>>>>
>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
>>>> </param>
>>>>
>>>> <param>
>>>> <name>main.ldapRealm.contextFactory</name>
>>>> <value>$ldapContextFactory</value>
>>>> </param>
>>>> <param>
>>>> <name>main.ldapRealm.contextFactory.url</name>
>>>> <!-- ADJUST host, port for your AD setup-->
>>>> <value>ldap://IP_OF_WINDOWS_AD:389</value>
>>>> </param>
>>>> <!-- ignored due to use of
>>>> main.ldapRealm.userSearchAttributeName -->
>>>> <param>
>>>> <name>main.ldapRealm.userDnTemplate</name>
>>>> <value>cn={0},CN=users,DC=test,DC=com</value>
>>>> <!-- also tried following values -->
>>>> <value>uid={0},CN=users,DC=test,DC=com</value>
>>>> <value>cn={0},DC=test,DC=com</value>
>>>> </param>
>>>>
>>>> <!-- Param above is ignored sAMAccount is usually used for AD
>>>> -->
>>>> <param>
>>>> <name>main.ldapRealm.userSearchAttributeName</name>
>>>> <value>sAMAccountName</value>
>>>> </param>
>>>>
>>>> <!-- adjust as appropriate -->
>>>> <param>
>>>> <name>main.ldapRealm.userObjectClass</name>
>>>> <value>person</value>
>>>> </param>
>>>>
>>>> <!-- adjust the dn below to match your environment -->
>>>> <param>
>>>> <name>main.ldapRealm.contextFactory.systemUsername</name>
>>>> <value>cn={systemuser},ou=process,ou=accounts,dc=test,dc=com</value>
>>>> </param>
>>>>
>>>> <!-- should be moved to the credential store for the gateway to be more
>>>> secure -->
>>>> <param>
>>>> <name>main.ldapRealm.contextFactory.systemPassword</name>
>>>> <value>{systemuser_password}/value>
>>>> </param>
>>>>
>>>> <!-- let's disable for now since you have no authorization
>>>> policies defined anyway -->
>>>> <param>
>>>> <name>main.ldapRealm.authorizationEnabled</name>
>>>> <value>false</value>
>>>> </param>
>>>>
>>>> <param>
>>>> <name>main.ldapRealm.searchBase</name>
>>>> <value>cn=users,dc=test,dc=com</value>
>>>> </param>
>>>>
>>>> <param>
>>>>
>>>> <param>
>>>> <name>main.ldapRealm.memberAttributeValueTemplate</name>
>>>> <value>cn={0},cn=users,dc=test,dc=com</value>
>>>> <!-- also tried uid={0} -->
>>>> </param>
>>>>
>>>> <param>
>>>>
>>>> <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
>>>> <value>simple</value>
>>>> </param>
>>>>
>>>> <param>
>>>> <name>urls./**</name>
>>>> <value>authcBasic</value>
>>>> </param>
>>>> </provider>
>>>>
>>>> <!-- the group principal mapping below is not likely what you
>>>> want
>>>> note that mapping of the hdfs group to admin. Also, we have
>>>> disabled authorization above so there is no need for
>>>> groups -->
>>>> <provider>
>>>> <role>identity-assertion</role>
>>>> <name>Default</name>
>>>> <enabled>true</enabled>
>>>> <!--param>
>>>> <name>group.principal.mapping</name>
>>>> <value>*=users;hdfs=admin</value>
>>>> </param-->
>>>> </provider>
>>>>
>>>> <provider>
>>>> <role>authorization</role>
>>>> <name>AclsAuthz</name>
>>>> <enabled>true</enabled>
>>>> </provider>
>>>>
>>>> </gateway>
>>>>
>>>> We need to better document the difference between LDAP and AD for such
>>>> deployments.
>>>>
>>>> I've also tried to document some of the changes that I made.
>>>> Note that you don't have any authorization ACLs defined in the
>>>> AclsAuthz provider so I disabled group lookup.
>>>> That will only add complexity to your config - we can re-enable once
>>>> authentication is working.
>>>>
>>>> Please go through this config and ensure that DNs, host and ports and
>>>> system usernames match your environment.
>>>>
>>>> Hope this helps.
>>>>
>>>> --larry
>>>>
>>>> On Tue, Dec 8, 2015 at 5:16 AM, Darpan Patel <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> For this blocker issue let more information if it can help fixing the
>>>>> authorization problem.
>>>>> Please let me know if more details required.
>>>>> (+ dev list)
>>>>>
>>>>> */etc/krb5.conf*
>>>>>
>>>>> [libdefaults]
>>>>> renew_lifetime = 7d
>>>>> forwardable = true
>>>>> default_realm = HORTONWORKS.COM
>>>>> ticket_lifetime = 24h
>>>>> dns_lookup_realm = false
>>>>> dns_lookup_kdc = false
>>>>> #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
>>>>> #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
>>>>>
>>>>> [domain_realm]
>>>>> .hortonworks.com = HORTONWORKS.COM
>>>>> HORTONWORKS.COm = HORTONWORKS.COM
>>>>>
>>>>> [logging]
>>>>> default = FILE:/var/log/krb5kdc.log
>>>>> admin_server = FILE:/var/log/kadmind.log
>>>>> kdc = FILE:/var/log/krb5kdc.log
>>>>>
>>>>> [realms]
>>>>> HORTONWORKS.COM = {
>>>>> admin_server = KDC_SERVER_HOST
>>>>> kdc = KDC_SERVER_HOST
>>>>> }
>>>>> *TEST.COM <http://TEST.COM>* = {
>>>>> admin_server = WINDOWS_12_SERVER_AD_HOST
>>>>> kdc = WINDOWS_12_SERVER_AD_HOST
>>>>> }
>>>>>
>>>>>
>>>>> */usr/hdp/current/knox-server/conf/gateway-site.xml*
>>>>>
>>>>> <configuration>
>>>>> <property>
>>>>> <name>*gateway.gateway.conf.dir*</name>
>>>>> <value>deployments</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>*gateway.hadoop.kerberos.secured*</name>
>>>>> <value>true</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>*gateway.path*</name>
>>>>> <value>gateway</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>*gateway.port*</name>
>>>>> <value>8443</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>*java.security.auth.login.config*</name>
>>>>> <value>/*etc/knox/conf/krb5JAASLogin.conf*</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>*java.security.krb5.conf*</name>
>>>>> <value>*/etc/krb5.conf*</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>sun.security.krb5.debug</name>
>>>>> <value>true</value>
>>>>> </property>
>>>>> </configuration>
>>>>>
>>>>>
>>>>> */etc/knox/conf/krb5JAASLogin.conf*
>>>>>
>>>>> com.sun.security.jgss.initiate {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> renewTGT=true
>>>>> doNotPrompt=true
>>>>> useKeyTab=true
>>>>> keyTab="/etc/security/keytabs/knox.service.keytab"
>>>>> principal="knox/[email protected]"
>>>>> isInitiator=true
>>>>> storeKey=true
>>>>> useTicketCache=true
>>>>> client=true;
>>>>> };
>>>>>
>>>>> Regards,
>>>>> DP
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Darpan Patel <[email protected]>
>>>>> Date: 7 December 2015 at 17:59
>>>>> Subject: Need help setting up Knox for A/D integrated Kerberized
>>>>> Cluster
>>>>> To: [email protected]
>>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I am stuck on an issue from last two days. I would be really grateful
>>>>> if someone can help on this.
>>>>>
>>>>> We have HDP 2.3 implemented over 8 node cluster and the same cluster
>>>>> has been Kerberized and later on we have integrated it with Active
>>>>> Directory (Which runs in the same VPN). We also verified that Windows 2012
>>>>> A/D integration with Ranger works fine for defining policies and audit
>>>>> log.
>>>>> But I am stuck at Knox bit. I am trying to replicate the same
>>>>> configuration
>>>>> properties which I have set for Ranger LDAP-AD Integration.
>>>>>
>>>>> I am taking reference of the Hortonworks documentation and also Apache
>>>>> Knox documentation.
>>>>>
>>>>> The A/D domain name is TEST.COM and all the users are under Users
>>>>>
>>>>> [image: Inline images 1]
>>>>>
>>>>>
>>>>> Under the Users we have few users one of the them is knox, darpan,
>>>>> test,etc.
>>>>>
>>>>> When we issue following command on the node on which Knox Server is
>>>>> running (topology name is default)
>>>>>
>>>>>
>>>>>
>>>>> *curl -iv -k -u [email protected]:#123Password -X GET
>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS
>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>" OR*
>>>>>
>>>>> *curl -iv -k -u knox:#123Password -X GET
>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS
>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>"*
>>>>>
>>>>>
>>>>> Every time I see < HTTP/1.1 401 Unauthorized HTTP/1.1 401
>>>>> Unauthorized on the console.
>>>>>
>>>>>
>>>>> Entries in the *gateway-audit.log *are like this :
>>>>>
>>>>> gateway-audit.log
>>>>> ==================
>>>>> 15/12/07 17:11:08
>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable|
>>>>> 15/12/07 17:11:09
>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||authentication|
>>>>> *principal*|*[email protected] <[email protected]>*|failure|*LDAP
>>>>> authentication failed.*
>>>>> 15/12/07 17:11:09
>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response
>>>>> status: 401
>>>>>
>>>>>
>>>>> 15/12/07 17:05:28
>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable|
>>>>> 15/12/07 17:05:29
>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||authentication|
>>>>> *principal*|knox|failure|*LDAP authentication failed.*
>>>>> 15/12/07 17:05:29
>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response
>>>>> status: 401
>>>>>
>>>>>
>>>>> *Gateway.log*
>>>>> *===========*
>>>>> 2015-12-07 17:05:28,620 INFO hadoop.gateway
>>>>> (KnoxLdapRealm.java:getUserDn(550)) - Computed userDn:
>>>>> cn=knox,CN=users,DC=test,DC=com using dnTemplate for principal: knox
>>>>>
>>>>>
>>>>> Following is the part of our *default.xml *topology:
>>>>>
>>>>>
>>>>> <gateway>
>>>>> <provider>
>>>>> <role>authentication</role>
>>>>> <name>ShiroProvider</name>
>>>>> <enabled>true</enabled>
>>>>> <param>
>>>>> <name>sessionTimeout</name>
>>>>> <value>30</value>
>>>>> </param>
>>>>> <param>
>>>>> <name>*main.ldapRealm*</name>
>>>>>
>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
>>>>> </param>
>>>>>
>>>>> <param>
>>>>> <name>*main.ldapContextFactory*</name>
>>>>>
>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
>>>>> </param>
>>>>>
>>>>> <param>
>>>>> <name>*main.ldapRealm.userDnTemplate*</name>
>>>>> <value>cn={0},CN=users,DC=test,DC=com</value>
>>>>> <!-- also tried following values -->
>>>>> <value>uid={0},CN=users,DC=test,DC=com</value>
>>>>> <value>cn={0},DC=test,DC=com</value>
>>>>> </param>
>>>>> <param>
>>>>> <name>*main.ldapRealm.contextFactory.url*
>>>>> </name>
>>>>> <!-- IP Address of the WINDOSWS 2012 Acive
>>>>> Directory Server which works for Ranger -->
>>>>> <value>*ldap://IP_OF_WINDOWS_AD:389*</value>
>>>>> </param>
>>>>> <param>
>>>>> <name>*main.ldapRealm.authorizationEnabled*
>>>>> </name>
>>>>> <value>true</value>
>>>>> </param>
>>>>> <param>
>>>>> <name>*main.ldapRealm.searchBase*</name>
>>>>> <value>cn=users,dc=test,dc=com</value>
>>>>> </param>
>>>>> <param>
>>>>> <param>
>>>>> <name>
>>>>> *main.ldapRealm.memberAttributeValueTemplate*</name>
>>>>> <value>cn={0},cn=users,dc=test,dc=com</value>
>>>>> <!-- also tried uid={0} -->
>>>>> </param>
>>>>> <param>
>>>>> <name>
>>>>> *main.ldapRealm.contextFactory.authenticationMechanism<*/name>
>>>>> <value>simple</value>
>>>>> </param>
>>>>> <param>
>>>>> <name>urls./**</name>
>>>>> <value>authcBasic</value>
>>>>> </param>
>>>>> </provider>
>>>>>
>>>>> <provider>
>>>>> <role>*identity-assertion*</role>
>>>>> <name>Default</name>
>>>>> <enabled>true</enabled>
>>>>> <param>
>>>>> <name>*group.principal.mapping*</name>
>>>>> <value>*=users;hdfs=admin</value>
>>>>> </param>
>>>>> </provider>
>>>>>
>>>>> <provider>
>>>>> <role>*authorization*</role>
>>>>> <name>AclsAuthz</name>
>>>>> <enabled>true</enabled>
>>>>> </provider>
>>>>>
>>>>> </gateway>
>>>>>
>>>>>
>>>>> And following is the console output while trying to access webhdfs
>>>>> using curl
>>>>>
>>>>> curl -iv -k -u knox:#123Password -X GET "
>>>>> https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS";
>>>>>
>>>>>
>>>>> *Console Output:*
>>>>> ----------------
>>>>>
>>>>> * About to connect() to localhost port 8443 (#0)
>>>>> * Trying ::1...
>>>>> * Connected to localhost (::1) port 8443 (#0)
>>>>> * Initializing NSS with certpath: sql:/etc/pki/nssdb
>>>>> * skipping SSL peer certificate verification
>>>>> * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
>>>>> * Server certificate:
>>>>> * subject:
>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
>>>>> * start date: Nov 27 20:36:22 2015 GMT
>>>>> * expire date: Nov 26 20:36:22 2016 GMT
>>>>> * common name: FQDN_OF_My_gateway_HOST
>>>>> * issuer:
>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
>>>>> * Server auth using Basic with user 'knox'
>>>>> > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1
>>>>> > Authorization: Basic a25veDojMTIzUGFzc3dvcmQ=
>>>>> > User-Agent: curl/7.29.0
>>>>> > Host: localhost:8443
>>>>> > Accept: */*
>>>>> >
>>>>> < HTTP/1.1 401 Unauthorized
>>>>> HTTP/1.1 401 Unauthorized
>>>>> * Authentication problem. Ignoring this.
>>>>> < WWW-Authenticate: BASIC realm="application"
>>>>> WWW-Authenticate: BASIC realm="application"
>>>>> < Content-Length: 0
>>>>> Content-Length: 0
>>>>> < Server: Jetty(8.1.14.v20131031)
>>>>> Server: Jetty(8.1.14.v20131031)
>>>>>
>>>>>
>>>>> Please let me know if any additional information is required.
>>>>>
>>>>> Thanks,
>>>>> DP
>>>>>
>>>>>
>>>>
>>>
>>
>
