In the version that I sent you the main.ldapContextFactory is set before this entry. Is that true in the version that you are using?
On Tue, Dec 8, 2015 at 5:16 PM, Darpan Patel <[email protected]> wrote: > Well when I am keeping the param to the following value we get an error. > > <param> >> <name>main.ldapRealm.contextFactory</name> >> <value>$ldapContextFactory</value> >> </param> >> >> > > Copying from the gateway.log. (It made me think we need to define the > value for ldapContextFactory) > > 2015-12-08 22:13:58,003 ERROR env.EnvironmentLoader > (EnvironmentLoader.java:initEnvironment(146)) - Shiro environment > initialization failed > org.apache.shiro.config.UnresolveableReferenceException: *The object with > id [ldapContextFactory] has not yet been defined and therefore cannot be > referenced. * Please ensure objects are defined in the order in which > they should be created and made available for future reference. > at > org.apache.shiro.config.ReflectionBuilder.getReferencedObject(ReflectionBuilder.java:224) > at > org.apache.shiro.config.ReflectionBuilder.resolveReference(ReflectionBuilder.java:239) > > > Regards, > DP > > >> >> On Tue, Dec 8, 2015 at 4:59 PM, Darpan Patel <[email protected]> wrote: >> >>> Thanks for the merged template. I made modifications to it and >>> >>> I am not sure what value should I fill for main.ldapRealm.contextFactory >>> ? >>> We are running on windows 2008/2012 Active directory. >>> >>> <param> >>> <name>main.ldapRealm.contextFactory</name> >>> <value>$ldapContextFactory</value> >>> </param> >>> >>> >> I think that you leave it exactly like that. >> It is some sort of shiro injection thing - it references the value >> defined above it that way. >> >> >>> I removed this parameter and I see the in the logs: >>> >>> 2015-12-08 21:56:51,806 ERROR hadoop.gateway >>> (KnoxLdapRealm.java:getUserDn(574)) - Failed to get system ldap connection: >>> javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: >>> LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e, >>> v1db1] >>> >>> >>> ( I am happy to see new error after 3 days phew!!!) >>> >>> >> Glad that you are happy but let's getting working and see how you feel. :) >> We'll also roll it into some better documentation for the AD specific >> usecase. >> >> >>> Regards, >>> DP >>> >>> On 8 December 2015 at 14:52, Darpan Patel <[email protected]> wrote: >>> >>>> Thanks Larray. >>>> I will check this and update you. >>>> >>>> Regards, >>>> DP >>>> >>>> On 8 December 2015 at 12:18, larry mccay <[email protected]> wrote: >>>> >>>>> Hi Darpan - >>>>> >>>>> The following topology is probably a better starting point for you AD >>>>> configuration - I've tried to merge yours with it as best I can: >>>>> >>>>> <gateway> >>>>> <provider> >>>>> <role>authentication</role> >>>>> <name>ShiroProvider</name> >>>>> <enabled>true</enabled> >>>>> <param> >>>>> <name>sessionTimeout</name> >>>>> <value>30</value> >>>>> </param> >>>>> <param> >>>>> <name>main.ldapRealm</name> >>>>> >>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>>>> </param> >>>>> >>>>> <param> >>>>> <name>main.ldapContextFactory</name> >>>>> >>>>> >>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>>>> </param> >>>>> >>>>> <param> >>>>> <name>main.ldapRealm.contextFactory</name> >>>>> <value>$ldapContextFactory</value> >>>>> </param> >>>>> <param> >>>>> <name>main.ldapRealm.contextFactory.url</name> >>>>> <!-- ADJUST host, port for your AD setup--> >>>>> <value>ldap://IP_OF_WINDOWS_AD:389</value> >>>>> </param> >>>>> <!-- ignored due to use of >>>>> main.ldapRealm.userSearchAttributeName --> >>>>> <param> >>>>> <name>main.ldapRealm.userDnTemplate</name> >>>>> <value>cn={0},CN=users,DC=test,DC=com</value> >>>>> <!-- also tried following values --> >>>>> <value>uid={0},CN=users,DC=test,DC=com</value> >>>>> <value>cn={0},DC=test,DC=com</value> >>>>> </param> >>>>> >>>>> <!-- Param above is ignored sAMAccount is usually used for AD >>>>> --> >>>>> <param> >>>>> <name>main.ldapRealm.userSearchAttributeName</name> >>>>> <value>sAMAccountName</value> >>>>> </param> >>>>> >>>>> <!-- adjust as appropriate --> >>>>> <param> >>>>> <name>main.ldapRealm.userObjectClass</name> >>>>> <value>person</value> >>>>> </param> >>>>> >>>>> <!-- adjust the dn below to match your environment --> >>>>> <param> >>>>> <name>main.ldapRealm.contextFactory.systemUsername</name> >>>>> <value>cn={systemuser},ou=process,ou=accounts,dc=test,dc=com</value> >>>>> </param> >>>>> >>>>> <!-- should be moved to the credential store for the gateway to be >>>>> more secure --> >>>>> <param> >>>>> <name>main.ldapRealm.contextFactory.systemPassword</name> >>>>> <value>{systemuser_password}/value> >>>>> </param> >>>>> >>>>> <!-- let's disable for now since you have no >>>>> authorization policies defined anyway --> >>>>> <param> >>>>> <name>main.ldapRealm.authorizationEnabled</name> >>>>> <value>false</value> >>>>> </param> >>>>> >>>>> <param> >>>>> <name>main.ldapRealm.searchBase</name> >>>>> <value>cn=users,dc=test,dc=com</value> >>>>> </param> >>>>> >>>>> <param> >>>>> >>>>> <param> >>>>> <name>main.ldapRealm.memberAttributeValueTemplate</name> >>>>> <value>cn={0},cn=users,dc=test,dc=com</value> >>>>> <!-- also tried uid={0} --> >>>>> </param> >>>>> >>>>> <param> >>>>> >>>>> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> >>>>> <value>simple</value> >>>>> </param> >>>>> >>>>> <param> >>>>> <name>urls./**</name> >>>>> <value>authcBasic</value> >>>>> </param> >>>>> </provider> >>>>> >>>>> <!-- the group principal mapping below is not likely what you >>>>> want >>>>> note that mapping of the hdfs group to admin. Also, we have >>>>> disabled authorization above so there is no need for >>>>> groups --> >>>>> <provider> >>>>> <role>identity-assertion</role> >>>>> <name>Default</name> >>>>> <enabled>true</enabled> >>>>> <!--param> >>>>> <name>group.principal.mapping</name> >>>>> <value>*=users;hdfs=admin</value> >>>>> </param--> >>>>> </provider> >>>>> >>>>> <provider> >>>>> <role>authorization</role> >>>>> <name>AclsAuthz</name> >>>>> <enabled>true</enabled> >>>>> </provider> >>>>> >>>>> </gateway> >>>>> >>>>> We need to better document the difference between LDAP and AD for such >>>>> deployments. >>>>> >>>>> I've also tried to document some of the changes that I made. >>>>> Note that you don't have any authorization ACLs defined in the >>>>> AclsAuthz provider so I disabled group lookup. >>>>> That will only add complexity to your config - we can re-enable once >>>>> authentication is working. >>>>> >>>>> Please go through this config and ensure that DNs, host and ports and >>>>> system usernames match your environment. >>>>> >>>>> Hope this helps. >>>>> >>>>> --larry >>>>> >>>>> On Tue, Dec 8, 2015 at 5:16 AM, Darpan Patel <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> For this blocker issue let more information if it can help fixing the >>>>>> authorization problem. >>>>>> Please let me know if more details required. >>>>>> (+ dev list) >>>>>> >>>>>> */etc/krb5.conf* >>>>>> >>>>>> [libdefaults] >>>>>> renew_lifetime = 7d >>>>>> forwardable = true >>>>>> default_realm = HORTONWORKS.COM >>>>>> ticket_lifetime = 24h >>>>>> dns_lookup_realm = false >>>>>> dns_lookup_kdc = false >>>>>> #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>>>> #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>>>> >>>>>> [domain_realm] >>>>>> .hortonworks.com = HORTONWORKS.COM >>>>>> HORTONWORKS.COm = HORTONWORKS.COM >>>>>> >>>>>> [logging] >>>>>> default = FILE:/var/log/krb5kdc.log >>>>>> admin_server = FILE:/var/log/kadmind.log >>>>>> kdc = FILE:/var/log/krb5kdc.log >>>>>> >>>>>> [realms] >>>>>> HORTONWORKS.COM = { >>>>>> admin_server = KDC_SERVER_HOST >>>>>> kdc = KDC_SERVER_HOST >>>>>> } >>>>>> *TEST.COM <http://TEST.COM>* = { >>>>>> admin_server = WINDOWS_12_SERVER_AD_HOST >>>>>> kdc = WINDOWS_12_SERVER_AD_HOST >>>>>> } >>>>>> >>>>>> >>>>>> */usr/hdp/current/knox-server/conf/gateway-site.xml* >>>>>> >>>>>> <configuration> >>>>>> <property> >>>>>> <name>*gateway.gateway.conf.dir*</name> >>>>>> <value>deployments</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>*gateway.hadoop.kerberos.secured*</name> >>>>>> <value>true</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>*gateway.path*</name> >>>>>> <value>gateway</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>*gateway.port*</name> >>>>>> <value>8443</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>*java.security.auth.login.config*</name> >>>>>> <value>/*etc/knox/conf/krb5JAASLogin.conf*</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>*java.security.krb5.conf*</name> >>>>>> <value>*/etc/krb5.conf*</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>sun.security.krb5.debug</name> >>>>>> <value>true</value> >>>>>> </property> >>>>>> </configuration> >>>>>> >>>>>> >>>>>> */etc/knox/conf/krb5JAASLogin.conf* >>>>>> >>>>>> com.sun.security.jgss.initiate { >>>>>> com.sun.security.auth.module.Krb5LoginModule required >>>>>> renewTGT=true >>>>>> doNotPrompt=true >>>>>> useKeyTab=true >>>>>> keyTab="/etc/security/keytabs/knox.service.keytab" >>>>>> principal="knox/[email protected]" >>>>>> isInitiator=true >>>>>> storeKey=true >>>>>> useTicketCache=true >>>>>> client=true; >>>>>> }; >>>>>> >>>>>> Regards, >>>>>> DP >>>>>> >>>>>> ---------- Forwarded message ---------- >>>>>> From: Darpan Patel <[email protected]> >>>>>> Date: 7 December 2015 at 17:59 >>>>>> Subject: Need help setting up Knox for A/D integrated Kerberized >>>>>> Cluster >>>>>> To: [email protected] >>>>>> >>>>>> >>>>>> Hi All, >>>>>> >>>>>> I am stuck on an issue from last two days. I would be really grateful >>>>>> if someone can help on this. >>>>>> >>>>>> We have HDP 2.3 implemented over 8 node cluster and the same cluster >>>>>> has been Kerberized and later on we have integrated it with Active >>>>>> Directory (Which runs in the same VPN). We also verified that Windows >>>>>> 2012 >>>>>> A/D integration with Ranger works fine for defining policies and audit >>>>>> log. >>>>>> But I am stuck at Knox bit. I am trying to replicate the same >>>>>> configuration >>>>>> properties which I have set for Ranger LDAP-AD Integration. >>>>>> >>>>>> I am taking reference of the Hortonworks documentation and also >>>>>> Apache Knox documentation. >>>>>> >>>>>> The A/D domain name is TEST.COM and all the users are under Users >>>>>> >>>>>> [image: Inline images 1] >>>>>> >>>>>> >>>>>> Under the Users we have few users one of the them is knox, darpan, >>>>>> test,etc. >>>>>> >>>>>> When we issue following command on the node on which Knox Server is >>>>>> running (topology name is default) >>>>>> >>>>>> >>>>>> >>>>>> *curl -iv -k -u [email protected]:#123Password -X GET >>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>" OR* >>>>>> >>>>>> *curl -iv -k -u knox:#123Password -X GET >>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>"* >>>>>> >>>>>> >>>>>> Every time I see < HTTP/1.1 401 Unauthorized HTTP/1.1 401 >>>>>> Unauthorized on the console. >>>>>> >>>>>> >>>>>> Entries in the *gateway-audit.log *are like this : >>>>>> >>>>>> gateway-audit.log >>>>>> ================== >>>>>> 15/12/07 17:11:08 >>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>>>> 15/12/07 17:11:09 >>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||authentication| >>>>>> *principal*|*[email protected] <[email protected]>*|failure|*LDAP >>>>>> authentication failed.* >>>>>> 15/12/07 17:11:09 >>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>>>> status: 401 >>>>>> >>>>>> >>>>>> 15/12/07 17:05:28 >>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>>>> 15/12/07 17:05:29 >>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||authentication| >>>>>> *principal*|knox|failure|*LDAP authentication failed.* >>>>>> 15/12/07 17:05:29 >>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>>>> status: 401 >>>>>> >>>>>> >>>>>> *Gateway.log* >>>>>> *===========* >>>>>> 2015-12-07 17:05:28,620 INFO hadoop.gateway >>>>>> (KnoxLdapRealm.java:getUserDn(550)) - Computed userDn: >>>>>> cn=knox,CN=users,DC=test,DC=com using dnTemplate for principal: knox >>>>>> >>>>>> >>>>>> Following is the part of our *default.xml *topology: >>>>>> >>>>>> >>>>>> <gateway> >>>>>> <provider> >>>>>> <role>authentication</role> >>>>>> <name>ShiroProvider</name> >>>>>> <enabled>true</enabled> >>>>>> <param> >>>>>> <name>sessionTimeout</name> >>>>>> <value>30</value> >>>>>> </param> >>>>>> <param> >>>>>> <name>*main.ldapRealm*</name> >>>>>> >>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> <name>*main.ldapContextFactory*</name> >>>>>> >>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> <name>*main.ldapRealm.userDnTemplate*</name> >>>>>> <value>cn={0},CN=users,DC=test,DC=com</value> >>>>>> <!-- also tried following values --> >>>>>> <value>uid={0},CN=users,DC=test,DC=com</value> >>>>>> <value>cn={0},DC=test,DC=com</value> >>>>>> </param> >>>>>> <param> >>>>>> <name>*main.ldapRealm.contextFactory.url* >>>>>> </name> >>>>>> <!-- IP Address of the WINDOSWS 2012 Acive >>>>>> Directory Server which works for Ranger --> >>>>>> <value>*ldap://IP_OF_WINDOWS_AD:389*</value> >>>>>> </param> >>>>>> <param> >>>>>> <name>*main.ldapRealm.authorizationEnabled* >>>>>> </name> >>>>>> <value>true</value> >>>>>> </param> >>>>>> <param> >>>>>> <name>*main.ldapRealm.searchBase*</name> >>>>>> <value>cn=users,dc=test,dc=com</value> >>>>>> </param> >>>>>> <param> >>>>>> <param> >>>>>> <name> >>>>>> *main.ldapRealm.memberAttributeValueTemplate*</name> >>>>>> <value>cn={0},cn=users,dc=test,dc=com</value> >>>>>> <!-- also tried uid={0} --> >>>>>> </param> >>>>>> <param> >>>>>> <name> >>>>>> *main.ldapRealm.contextFactory.authenticationMechanism<*/name> >>>>>> <value>simple</value> >>>>>> </param> >>>>>> <param> >>>>>> <name>urls./**</name> >>>>>> <value>authcBasic</value> >>>>>> </param> >>>>>> </provider> >>>>>> >>>>>> <provider> >>>>>> <role>*identity-assertion*</role> >>>>>> <name>Default</name> >>>>>> <enabled>true</enabled> >>>>>> <param> >>>>>> <name>*group.principal.mapping*</name> >>>>>> <value>*=users;hdfs=admin</value> >>>>>> </param> >>>>>> </provider> >>>>>> >>>>>> <provider> >>>>>> <role>*authorization*</role> >>>>>> <name>AclsAuthz</name> >>>>>> <enabled>true</enabled> >>>>>> </provider> >>>>>> >>>>>> </gateway> >>>>>> >>>>>> >>>>>> And following is the console output while trying to access webhdfs >>>>>> using curl >>>>>> >>>>>> curl -iv -k -u knox:#123Password -X GET " >>>>>> https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"; >>>>>> >>>>>> >>>>>> *Console Output:* >>>>>> ---------------- >>>>>> >>>>>> * About to connect() to localhost port 8443 (#0) >>>>>> * Trying ::1... >>>>>> * Connected to localhost (::1) port 8443 (#0) >>>>>> * Initializing NSS with certpath: sql:/etc/pki/nssdb >>>>>> * skipping SSL peer certificate verification >>>>>> * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA >>>>>> * Server certificate: >>>>>> * subject: >>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>>>> * start date: Nov 27 20:36:22 2015 GMT >>>>>> * expire date: Nov 26 20:36:22 2016 GMT >>>>>> * common name: FQDN_OF_My_gateway_HOST >>>>>> * issuer: >>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>>>> * Server auth using Basic with user 'knox' >>>>>> > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1 >>>>>> > Authorization: Basic a25veDojMTIzUGFzc3dvcmQ= >>>>>> > User-Agent: curl/7.29.0 >>>>>> > Host: localhost:8443 >>>>>> > Accept: */* >>>>>> > >>>>>> < HTTP/1.1 401 Unauthorized >>>>>> HTTP/1.1 401 Unauthorized >>>>>> * Authentication problem. Ignoring this. >>>>>> < WWW-Authenticate: BASIC realm="application" >>>>>> WWW-Authenticate: BASIC realm="application" >>>>>> < Content-Length: 0 >>>>>> Content-Length: 0 >>>>>> < Server: Jetty(8.1.14.v20131031) >>>>>> Server: Jetty(8.1.14.v20131031) >>>>>> >>>>>> >>>>>> Please let me know if any additional information is required. >>>>>> >>>>>> Thanks, >>>>>> DP >>>>>> >>>>>> >>>>> >>>> >>> >> >
