Hi Larry, I am using the version : 0.6.0.2.3.0.0-2557 of Knox .
Checked through curl -u admin:admin-password -i -k https://localhost:8443/gateway/admin/api/v1/version On 8 December 2015 at 23:42, larry mccay <[email protected]> wrote: > In the version that I sent you the main.ldapContextFactory is set before > this entry. > Is that true in the version that you are using? > > On Tue, Dec 8, 2015 at 5:16 PM, Darpan Patel <[email protected]> wrote: > >> Well when I am keeping the param to the following value we get an error. >> >> <param> >>> <name>main.ldapRealm.contextFactory</name> >>> <value>$ldapContextFactory</value> >>> </param> >>> >>> >> >> Copying from the gateway.log. (It made me think we need to define the >> value for ldapContextFactory) >> >> 2015-12-08 22:13:58,003 ERROR env.EnvironmentLoader >> (EnvironmentLoader.java:initEnvironment(146)) - Shiro environment >> initialization failed >> org.apache.shiro.config.UnresolveableReferenceException: *The object >> with id [ldapContextFactory] has not yet been defined and therefore cannot >> be referenced. * Please ensure objects are defined in the order in which >> they should be created and made available for future reference. >> at >> org.apache.shiro.config.ReflectionBuilder.getReferencedObject(ReflectionBuilder.java:224) >> at >> org.apache.shiro.config.ReflectionBuilder.resolveReference(ReflectionBuilder.java:239) >> >> >> Regards, >> DP >> >> >>> >>> On Tue, Dec 8, 2015 at 4:59 PM, Darpan Patel <[email protected]> wrote: >>> >>>> Thanks for the merged template. I made modifications to it and >>>> >>>> I am not sure what value should I fill >>>> for main.ldapRealm.contextFactory ? >>>> We are running on windows 2008/2012 Active directory. >>>> >>>> <param> >>>> <name>main.ldapRealm.contextFactory</name> >>>> <value>$ldapContextFactory</value> >>>> </param> >>>> >>>> >>> I think that you leave it exactly like that. >>> It is some sort of shiro injection thing - it references the value >>> defined above it that way. >>> >>> >>>> I removed this parameter and I see the in the logs: >>>> >>>> 2015-12-08 21:56:51,806 ERROR hadoop.gateway >>>> (KnoxLdapRealm.java:getUserDn(574)) - Failed to get system ldap connection: >>>> javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: >>>> LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e, >>>> v1db1] >>>> >>>> >>>> ( I am happy to see new error after 3 days phew!!!) >>>> >>>> >>> Glad that you are happy but let's getting working and see how you feel. >>> :) >>> We'll also roll it into some better documentation for the AD specific >>> usecase. >>> >>> >>>> Regards, >>>> DP >>>> >>>> On 8 December 2015 at 14:52, Darpan Patel <[email protected]> wrote: >>>> >>>>> Thanks Larray. >>>>> I will check this and update you. >>>>> >>>>> Regards, >>>>> DP >>>>> >>>>> On 8 December 2015 at 12:18, larry mccay <[email protected]> wrote: >>>>> >>>>>> Hi Darpan - >>>>>> >>>>>> The following topology is probably a better starting point for you AD >>>>>> configuration - I've tried to merge yours with it as best I can: >>>>>> >>>>>> <gateway> >>>>>> <provider> >>>>>> <role>authentication</role> >>>>>> <name>ShiroProvider</name> >>>>>> <enabled>true</enabled> >>>>>> <param> >>>>>> <name>sessionTimeout</name> >>>>>> <value>30</value> >>>>>> </param> >>>>>> <param> >>>>>> <name>main.ldapRealm</name> >>>>>> >>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> <name>main.ldapContextFactory</name> >>>>>> >>>>>> >>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> <name>main.ldapRealm.contextFactory</name> >>>>>> <value>$ldapContextFactory</value> >>>>>> </param> >>>>>> <param> >>>>>> <name>main.ldapRealm.contextFactory.url</name> >>>>>> <!-- ADJUST host, port for your AD setup--> >>>>>> <value>ldap://IP_OF_WINDOWS_AD:389</value> >>>>>> </param> >>>>>> <!-- ignored due to use of >>>>>> main.ldapRealm.userSearchAttributeName --> >>>>>> <param> >>>>>> <name>main.ldapRealm.userDnTemplate</name> >>>>>> <value>cn={0},CN=users,DC=test,DC=com</value> >>>>>> <!-- also tried following values --> >>>>>> <value>uid={0},CN=users,DC=test,DC=com</value> >>>>>> <value>cn={0},DC=test,DC=com</value> >>>>>> </param> >>>>>> >>>>>> <!-- Param above is ignored sAMAccount is usually used for AD >>>>>> --> >>>>>> <param> >>>>>> <name>main.ldapRealm.userSearchAttributeName</name> >>>>>> <value>sAMAccountName</value> >>>>>> </param> >>>>>> >>>>>> <!-- adjust as appropriate --> >>>>>> <param> >>>>>> <name>main.ldapRealm.userObjectClass</name> >>>>>> <value>person</value> >>>>>> </param> >>>>>> >>>>>> <!-- adjust the dn below to match your environment --> >>>>>> <param> >>>>>> <name>main.ldapRealm.contextFactory.systemUsername</name> >>>>>> <value>cn={systemuser},ou=process,ou=accounts,dc=test,dc=com</value> >>>>>> </param> >>>>>> >>>>>> <!-- should be moved to the credential store for the gateway to be >>>>>> more secure --> >>>>>> <param> >>>>>> <name>main.ldapRealm.contextFactory.systemPassword</name> >>>>>> <value>{systemuser_password}/value> >>>>>> </param> >>>>>> >>>>>> <!-- let's disable for now since you have no >>>>>> authorization policies defined anyway --> >>>>>> <param> >>>>>> <name>main.ldapRealm.authorizationEnabled</name> >>>>>> <value>false</value> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> <name>main.ldapRealm.searchBase</name> >>>>>> <value>cn=users,dc=test,dc=com</value> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> >>>>>> <param> >>>>>> >>>>>> <name>main.ldapRealm.memberAttributeValueTemplate</name> >>>>>> <value>cn={0},cn=users,dc=test,dc=com</value> >>>>>> <!-- also tried uid={0} --> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> >>>>>> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> >>>>>> <value>simple</value> >>>>>> </param> >>>>>> >>>>>> <param> >>>>>> <name>urls./**</name> >>>>>> <value>authcBasic</value> >>>>>> </param> >>>>>> </provider> >>>>>> >>>>>> <!-- the group principal mapping below is not likely what >>>>>> you want >>>>>> note that mapping of the hdfs group to admin. Also, we have >>>>>> disabled authorization above so there is no need for >>>>>> groups --> >>>>>> <provider> >>>>>> <role>identity-assertion</role> >>>>>> <name>Default</name> >>>>>> <enabled>true</enabled> >>>>>> <!--param> >>>>>> <name>group.principal.mapping</name> >>>>>> <value>*=users;hdfs=admin</value> >>>>>> </param--> >>>>>> </provider> >>>>>> >>>>>> <provider> >>>>>> <role>authorization</role> >>>>>> <name>AclsAuthz</name> >>>>>> <enabled>true</enabled> >>>>>> </provider> >>>>>> >>>>>> </gateway> >>>>>> >>>>>> We need to better document the difference between LDAP and AD for >>>>>> such deployments. >>>>>> >>>>>> I've also tried to document some of the changes that I made. >>>>>> Note that you don't have any authorization ACLs defined in the >>>>>> AclsAuthz provider so I disabled group lookup. >>>>>> That will only add complexity to your config - we can re-enable once >>>>>> authentication is working. >>>>>> >>>>>> Please go through this config and ensure that DNs, host and ports and >>>>>> system usernames match your environment. >>>>>> >>>>>> Hope this helps. >>>>>> >>>>>> --larry >>>>>> >>>>>> On Tue, Dec 8, 2015 at 5:16 AM, Darpan Patel <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> For this blocker issue let more information if it can help fixing >>>>>>> the authorization problem. >>>>>>> Please let me know if more details required. >>>>>>> (+ dev list) >>>>>>> >>>>>>> */etc/krb5.conf* >>>>>>> >>>>>>> [libdefaults] >>>>>>> renew_lifetime = 7d >>>>>>> forwardable = true >>>>>>> default_realm = HORTONWORKS.COM >>>>>>> ticket_lifetime = 24h >>>>>>> dns_lookup_realm = false >>>>>>> dns_lookup_kdc = false >>>>>>> #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>>>>> #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>>>>> >>>>>>> [domain_realm] >>>>>>> .hortonworks.com = HORTONWORKS.COM >>>>>>> HORTONWORKS.COm = HORTONWORKS.COM >>>>>>> >>>>>>> [logging] >>>>>>> default = FILE:/var/log/krb5kdc.log >>>>>>> admin_server = FILE:/var/log/kadmind.log >>>>>>> kdc = FILE:/var/log/krb5kdc.log >>>>>>> >>>>>>> [realms] >>>>>>> HORTONWORKS.COM = { >>>>>>> admin_server = KDC_SERVER_HOST >>>>>>> kdc = KDC_SERVER_HOST >>>>>>> } >>>>>>> *TEST.COM <http://TEST.COM>* = { >>>>>>> admin_server = WINDOWS_12_SERVER_AD_HOST >>>>>>> kdc = WINDOWS_12_SERVER_AD_HOST >>>>>>> } >>>>>>> >>>>>>> >>>>>>> */usr/hdp/current/knox-server/conf/gateway-site.xml* >>>>>>> >>>>>>> <configuration> >>>>>>> <property> >>>>>>> <name>*gateway.gateway.conf.dir*</name> >>>>>>> <value>deployments</value> >>>>>>> </property> >>>>>>> <property> >>>>>>> <name>*gateway.hadoop.kerberos.secured*</name> >>>>>>> <value>true</value> >>>>>>> </property> >>>>>>> <property> >>>>>>> <name>*gateway.path*</name> >>>>>>> <value>gateway</value> >>>>>>> </property> >>>>>>> <property> >>>>>>> <name>*gateway.port*</name> >>>>>>> <value>8443</value> >>>>>>> </property> >>>>>>> <property> >>>>>>> <name>*java.security.auth.login.config*</name> >>>>>>> <value>/*etc/knox/conf/krb5JAASLogin.conf*</value> >>>>>>> </property> >>>>>>> <property> >>>>>>> <name>*java.security.krb5.conf*</name> >>>>>>> <value>*/etc/krb5.conf*</value> >>>>>>> </property> >>>>>>> <property> >>>>>>> <name>sun.security.krb5.debug</name> >>>>>>> <value>true</value> >>>>>>> </property> >>>>>>> </configuration> >>>>>>> >>>>>>> >>>>>>> */etc/knox/conf/krb5JAASLogin.conf* >>>>>>> >>>>>>> com.sun.security.jgss.initiate { >>>>>>> com.sun.security.auth.module.Krb5LoginModule required >>>>>>> renewTGT=true >>>>>>> doNotPrompt=true >>>>>>> useKeyTab=true >>>>>>> keyTab="/etc/security/keytabs/knox.service.keytab" >>>>>>> principal="knox/[email protected]" >>>>>>> isInitiator=true >>>>>>> storeKey=true >>>>>>> useTicketCache=true >>>>>>> client=true; >>>>>>> }; >>>>>>> >>>>>>> Regards, >>>>>>> DP >>>>>>> >>>>>>> ---------- Forwarded message ---------- >>>>>>> From: Darpan Patel <[email protected]> >>>>>>> Date: 7 December 2015 at 17:59 >>>>>>> Subject: Need help setting up Knox for A/D integrated Kerberized >>>>>>> Cluster >>>>>>> To: [email protected] >>>>>>> >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I am stuck on an issue from last two days. I would be really >>>>>>> grateful if someone can help on this. >>>>>>> >>>>>>> We have HDP 2.3 implemented over 8 node cluster and the same cluster >>>>>>> has been Kerberized and later on we have integrated it with Active >>>>>>> Directory (Which runs in the same VPN). We also verified that Windows >>>>>>> 2012 >>>>>>> A/D integration with Ranger works fine for defining policies and audit >>>>>>> log. >>>>>>> But I am stuck at Knox bit. I am trying to replicate the same >>>>>>> configuration >>>>>>> properties which I have set for Ranger LDAP-AD Integration. >>>>>>> >>>>>>> I am taking reference of the Hortonworks documentation and also >>>>>>> Apache Knox documentation. >>>>>>> >>>>>>> The A/D domain name is TEST.COM and all the users are under Users >>>>>>> >>>>>>> [image: Inline images 1] >>>>>>> >>>>>>> >>>>>>> Under the Users we have few users one of the them is knox, darpan, >>>>>>> test,etc. >>>>>>> >>>>>>> When we issue following command on the node on which Knox Server is >>>>>>> running (topology name is default) >>>>>>> >>>>>>> >>>>>>> >>>>>>> *curl -iv -k -u [email protected]:#123Password -X GET >>>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>" OR* >>>>>>> >>>>>>> *curl -iv -k -u knox:#123Password -X GET >>>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>"* >>>>>>> >>>>>>> >>>>>>> Every time I see < HTTP/1.1 401 Unauthorized HTTP/1.1 401 >>>>>>> Unauthorized on the console. >>>>>>> >>>>>>> >>>>>>> Entries in the *gateway-audit.log *are like this : >>>>>>> >>>>>>> gateway-audit.log >>>>>>> ================== >>>>>>> 15/12/07 17:11:08 >>>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>>>>> 15/12/07 17:11:09 >>>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||authentication| >>>>>>> *principal*|*[email protected] <[email protected]>*|failure|*LDAP >>>>>>> authentication failed.* >>>>>>> 15/12/07 17:11:09 >>>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>>>>> status: 401 >>>>>>> >>>>>>> >>>>>>> 15/12/07 17:05:28 >>>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>>>>> 15/12/07 17:05:29 >>>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||authentication| >>>>>>> *principal*|knox|failure|*LDAP authentication failed.* >>>>>>> 15/12/07 17:05:29 >>>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>>>>> status: 401 >>>>>>> >>>>>>> >>>>>>> *Gateway.log* >>>>>>> *===========* >>>>>>> 2015-12-07 17:05:28,620 INFO hadoop.gateway >>>>>>> (KnoxLdapRealm.java:getUserDn(550)) - Computed userDn: >>>>>>> cn=knox,CN=users,DC=test,DC=com using dnTemplate for principal: knox >>>>>>> >>>>>>> >>>>>>> Following is the part of our *default.xml *topology: >>>>>>> >>>>>>> >>>>>>> <gateway> >>>>>>> <provider> >>>>>>> <role>authentication</role> >>>>>>> <name>ShiroProvider</name> >>>>>>> <enabled>true</enabled> >>>>>>> <param> >>>>>>> <name>sessionTimeout</name> >>>>>>> <value>30</value> >>>>>>> </param> >>>>>>> <param> >>>>>>> <name>*main.ldapRealm*</name> >>>>>>> >>>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>>>>>> </param> >>>>>>> >>>>>>> <param> >>>>>>> <name>*main.ldapContextFactory*</name> >>>>>>> >>>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>>>>>> </param> >>>>>>> >>>>>>> <param> >>>>>>> <name>*main.ldapRealm.userDnTemplate*</name> >>>>>>> <value>cn={0},CN=users,DC=test,DC=com</value> >>>>>>> <!-- also tried following values --> >>>>>>> >>>>>>> <value>uid={0},CN=users,DC=test,DC=com</value> >>>>>>> <value>cn={0},DC=test,DC=com</value> >>>>>>> </param> >>>>>>> <param> >>>>>>> <name>*main.ldapRealm.contextFactory.url* >>>>>>> </name> >>>>>>> <!-- IP Address of the WINDOSWS 2012 Acive >>>>>>> Directory Server which works for Ranger --> >>>>>>> <value>*ldap://IP_OF_WINDOWS_AD:389* >>>>>>> </value> >>>>>>> </param> >>>>>>> <param> >>>>>>> <name>*main.ldapRealm.authorizationEnabled* >>>>>>> </name> >>>>>>> <value>true</value> >>>>>>> </param> >>>>>>> <param> >>>>>>> <name>*main.ldapRealm.searchBase*</name> >>>>>>> <value>cn=users,dc=test,dc=com</value> >>>>>>> </param> >>>>>>> <param> >>>>>>> <param> >>>>>>> <name> >>>>>>> *main.ldapRealm.memberAttributeValueTemplate*</name> >>>>>>> <value>cn={0},cn=users,dc=test,dc=com</value> >>>>>>> <!-- also tried uid={0} --> >>>>>>> </param> >>>>>>> <param> >>>>>>> <name> >>>>>>> *main.ldapRealm.contextFactory.authenticationMechanism<*/name> >>>>>>> <value>simple</value> >>>>>>> </param> >>>>>>> <param> >>>>>>> <name>urls./**</name> >>>>>>> <value>authcBasic</value> >>>>>>> </param> >>>>>>> </provider> >>>>>>> >>>>>>> <provider> >>>>>>> <role>*identity-assertion*</role> >>>>>>> <name>Default</name> >>>>>>> <enabled>true</enabled> >>>>>>> <param> >>>>>>> <name>*group.principal.mapping*</name> >>>>>>> <value>*=users;hdfs=admin</value> >>>>>>> </param> >>>>>>> </provider> >>>>>>> >>>>>>> <provider> >>>>>>> <role>*authorization*</role> >>>>>>> <name>AclsAuthz</name> >>>>>>> <enabled>true</enabled> >>>>>>> </provider> >>>>>>> >>>>>>> </gateway> >>>>>>> >>>>>>> >>>>>>> And following is the console output while trying to access webhdfs >>>>>>> using curl >>>>>>> >>>>>>> curl -iv -k -u knox:#123Password -X GET " >>>>>>> https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"; >>>>>>> >>>>>>> >>>>>>> *Console Output:* >>>>>>> ---------------- >>>>>>> >>>>>>> * About to connect() to localhost port 8443 (#0) >>>>>>> * Trying ::1... >>>>>>> * Connected to localhost (::1) port 8443 (#0) >>>>>>> * Initializing NSS with certpath: sql:/etc/pki/nssdb >>>>>>> * skipping SSL peer certificate verification >>>>>>> * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA >>>>>>> * Server certificate: >>>>>>> * subject: >>>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>>>>> * start date: Nov 27 20:36:22 2015 GMT >>>>>>> * expire date: Nov 26 20:36:22 2016 GMT >>>>>>> * common name: FQDN_OF_My_gateway_HOST >>>>>>> * issuer: >>>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>>>>> * Server auth using Basic with user 'knox' >>>>>>> > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1 >>>>>>> > Authorization: Basic a25veDojMTIzUGFzc3dvcmQ= >>>>>>> > User-Agent: curl/7.29.0 >>>>>>> > Host: localhost:8443 >>>>>>> > Accept: */* >>>>>>> > >>>>>>> < HTTP/1.1 401 Unauthorized >>>>>>> HTTP/1.1 401 Unauthorized >>>>>>> * Authentication problem. Ignoring this. >>>>>>> < WWW-Authenticate: BASIC realm="application" >>>>>>> WWW-Authenticate: BASIC realm="application" >>>>>>> < Content-Length: 0 >>>>>>> Content-Length: 0 >>>>>>> < Server: Jetty(8.1.14.v20131031) >>>>>>> Server: Jetty(8.1.14.v20131031) >>>>>>> >>>>>>> >>>>>>> Please let me know if any additional information is required. >>>>>>> >>>>>>> Thanks, >>>>>>> DP >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
