Try: curl -iv -k -u knox:#123Password -X GET " https://gateway:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS";
The above assumes that there is a knox user in your AD. On Wed, Dec 9, 2015 at 8:50 AM, Darpan Patel <[email protected]> wrote: > HI Larry, > > Thanks for quick response. the value of contextFactory I missed somehow.. > Now I don't see the contextFactory undefined error but .... When I try to > curl the default gateway for webhdfs still I am seeing the same console. > > I tried issuing the following curl command with valid TGT in the cache and > after kdestroy and removing the TGT for both I am seeing the same output. > > curl -iv -k -u [email protected]:#123Password -X GET " > https://gateway:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"; > also tried > curl -iv -k -X GET " > https://gateway:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"; > > > I am attaching the default gateway topology file with the email to avoid > lot of texts. > > > In the *gateway.log *I don't see any entry while hitting the curl > > In the *gateway-audit *I see following : > > 15/12/09 13:44:47 > ||d96572dd-a988-4392-b7c8-fcf7e1d154f7|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| > 15/12/09 13:44:48 > ||d96572dd-a988-4392-b7c8-fcf7e1d154f7|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response > status: 401 > > I am not sure what I am missing!!! > > *Thank you very much for the help.* > > Regards, > DP > > > *Console Output:* > > [root@gateway knox-server]# curl -iv -k -u [email protected]:KnoxPassword@123 > -X GET "https://gateway:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"; > * About to connect() to gateway port 8443 (#0) > * Trying 192.168.197.8... > * Connected to gateway (192.168.197.8) port 8443 (#0) > * Initializing NSS with certpath: sql:/etc/pki/nssdb > * skipping SSL peer certificate verification > * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA > * Server certificate: > * subject: > CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US > * start date: Nov 27 20:36:22 2015 GMT > * expire date: Nov 26 20:36:22 2016 GMT > * common name: FQDN_OF_My_gateway_HOST > * issuer: > CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US > * Server auth using Basic with user '[email protected]' > > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1 > > Authorization: Basic a25veEB0ZXN0LmNvbTojMTIzUGFzc3dvcmQ= > > User-Agent: curl/7.29.0 > > Host: gateway:8443 > > Accept: */* > > > < HTTP/1.1 401 Unauthorized > HTTP/1.1 401 Unauthorized > * Authentication problem. Ignoring this. > < WWW-Authenticate: BASIC realm="application" > WWW-Authenticate: BASIC realm="application" > < Content-Length: 0 > Content-Length: 0 > < Server: Jetty(8.1.14.v20131031) > Server: Jetty(8.1.14.v20131031) > > < > * Connection #0 to host gateway left intact > > > > > > On 9 December 2015 at 13:24, larry mccay <[email protected]> wrote: > >> I meant the version of the topology that I sent you. >> Note the order of the following to config items: >> >> <param> >> <name>main.ldapContextFactory</name> >> >> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >> </param> >> >> <param> >> <name>main.ldapRealm.contextFactory</name> >> <value>$ldapContextFactory</value> >> </param> >> >> Do you have them in that order in the topology that you are using? >> >> On Wed, Dec 9, 2015 at 8:06 AM, Darpan Patel <[email protected]> wrote: >> >>> When we keep : >>> >>> <param> >>> <name>main.ldapRealm.contextFactory</name> >>> <value>$ldapContextFactory</value> >>> </param> >>> >>> in the log I see that the context Factory object is not defined >>> previously and hence could not be referred. Any idea for AD 2008/2012 >>> Windows Server what should be the value? >>> >>> I am knox : 0.6.0.2 version. >>> >>> 2015-12-09 12:39:45,185 ERROR env.EnvironmentLoader >>> (EnvironmentLoader.java:initEnvironment(146)) - Shiro environment >>> initialization failed >>> org.apache.shiro.config.UnresolveableReferenceException: The object with >>> id [ldapContextFactory] has not yet been defined and therefore cannot >>> be referenced. Please ensure objects are defined in the order in which >>> they should be created and made available for future reference. >>> >>> Many thanks, >>> DP >>> >>> >>> >>> On 9 December 2015 at 07:58, Darpan Patel <[email protected]> wrote: >>> >>>> Hi Larry, >>>> >>>> I am using the version : 0.6.0.2.3.0.0-2557 of Knox . >>>> >>>> >>>> Checked through curl -u admin:admin-password -i -k >>>> https://localhost:8443/gateway/admin/api/v1/version >>>> >>>> >>>> >>>> >>>> On 8 December 2015 at 23:42, larry mccay <[email protected]> wrote: >>>> >>>>> In the version that I sent you the main.ldapContextFactory is set >>>>> before this entry. >>>>> Is that true in the version that you are using? >>>>> >>>>> On Tue, Dec 8, 2015 at 5:16 PM, Darpan Patel <[email protected]> >>>>> wrote: >>>>> >>>>>> Well when I am keeping the param to the following value we get an >>>>>> error. >>>>>> >>>>>> <param> >>>>>>> <name>main.ldapRealm.contextFactory</name> >>>>>>> <value>$ldapContextFactory</value> >>>>>>> </param> >>>>>>> >>>>>>> >>>>>> >>>>>> Copying from the gateway.log. (It made me think we need to define the >>>>>> value for ldapContextFactory) >>>>>> >>>>>> 2015-12-08 22:13:58,003 ERROR env.EnvironmentLoader >>>>>> (EnvironmentLoader.java:initEnvironment(146)) - Shiro environment >>>>>> initialization failed >>>>>> org.apache.shiro.config.UnresolveableReferenceException: *The object >>>>>> with id [ldapContextFactory] has not yet been defined and therefore >>>>>> cannot >>>>>> be referenced. * Please ensure objects are defined in the order in >>>>>> which they should be created and made available for future reference. >>>>>> at >>>>>> org.apache.shiro.config.ReflectionBuilder.getReferencedObject(ReflectionBuilder.java:224) >>>>>> at >>>>>> org.apache.shiro.config.ReflectionBuilder.resolveReference(ReflectionBuilder.java:239) >>>>>> >>>>>> >>>>>> Regards, >>>>>> DP >>>>>> >>>>>> >>>>>>> >>>>>>> On Tue, Dec 8, 2015 at 4:59 PM, Darpan Patel <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for the merged template. I made modifications to it and >>>>>>>> >>>>>>>> I am not sure what value should I fill >>>>>>>> for main.ldapRealm.contextFactory ? >>>>>>>> We are running on windows 2008/2012 Active directory. >>>>>>>> >>>>>>>> <param> >>>>>>>> <name>main.ldapRealm.contextFactory</name> >>>>>>>> <value>$ldapContextFactory</value> >>>>>>>> </param> >>>>>>>> >>>>>>>> >>>>>>> I think that you leave it exactly like that. >>>>>>> It is some sort of shiro injection thing - it references the value >>>>>>> defined above it that way. >>>>>>> >>>>>>> >>>>>>>> I removed this parameter and I see the in the logs: >>>>>>>> >>>>>>>> 2015-12-08 21:56:51,806 ERROR hadoop.gateway >>>>>>>> (KnoxLdapRealm.java:getUserDn(574)) - Failed to get system ldap >>>>>>>> connection: >>>>>>>> javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: >>>>>>>> LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e, >>>>>>>> v1db1] >>>>>>>> >>>>>>>> >>>>>>>> ( I am happy to see new error after 3 days phew!!!) >>>>>>>> >>>>>>>> >>>>>>> Glad that you are happy but let's getting working and see how you >>>>>>> feel. :) >>>>>>> We'll also roll it into some better documentation for the AD >>>>>>> specific usecase. >>>>>>> >>>>>>> >>>>>>>> Regards, >>>>>>>> DP >>>>>>>> >>>>>>>> On 8 December 2015 at 14:52, Darpan Patel <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks Larray. >>>>>>>>> I will check this and update you. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> DP >>>>>>>>> >>>>>>>>> On 8 December 2015 at 12:18, larry mccay <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Darpan - >>>>>>>>>> >>>>>>>>>> The following topology is probably a better starting point for >>>>>>>>>> you AD configuration - I've tried to merge yours with it as best I >>>>>>>>>> can: >>>>>>>>>> >>>>>>>>>> <gateway> >>>>>>>>>> <provider> >>>>>>>>>> <role>authentication</role> >>>>>>>>>> <name>ShiroProvider</name> >>>>>>>>>> <enabled>true</enabled> >>>>>>>>>> <param> >>>>>>>>>> <name>sessionTimeout</name> >>>>>>>>>> <value>30</value> >>>>>>>>>> </param> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm</name> >>>>>>>>>> >>>>>>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapContextFactory</name> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.contextFactory</name> >>>>>>>>>> <value>$ldapContextFactory</value> >>>>>>>>>> </param> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.contextFactory.url</name> >>>>>>>>>> <!-- ADJUST host, port for your AD setup--> >>>>>>>>>> <value>ldap://IP_OF_WINDOWS_AD:389</value> >>>>>>>>>> </param> >>>>>>>>>> <!-- ignored due to use of >>>>>>>>>> main.ldapRealm.userSearchAttributeName --> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.userDnTemplate</name> >>>>>>>>>> <value>cn={0},CN=users,DC=test,DC=com</value> >>>>>>>>>> <!-- also tried following values --> >>>>>>>>>> <value>uid={0},CN=users,DC=test,DC=com</value> >>>>>>>>>> <value>cn={0},DC=test,DC=com</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <!-- Param above is ignored sAMAccount is usually used for >>>>>>>>>> AD --> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.userSearchAttributeName</name> >>>>>>>>>> <value>sAMAccountName</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <!-- adjust as appropriate --> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.userObjectClass</name> >>>>>>>>>> <value>person</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <!-- adjust the dn below to match your environment --> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.contextFactory.systemUsername</name> >>>>>>>>>> >>>>>>>>>> <value>cn={systemuser},ou=process,ou=accounts,dc=test,dc=com</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <!-- should be moved to the credential store for the gateway to >>>>>>>>>> be more secure --> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.contextFactory.systemPassword</name> >>>>>>>>>> <value>{systemuser_password}/value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <!-- let's disable for now since you have no >>>>>>>>>> authorization policies defined anyway --> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.authorizationEnabled</name> >>>>>>>>>> <value>false</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <param> >>>>>>>>>> <name>main.ldapRealm.searchBase</name> >>>>>>>>>> <value>cn=users,dc=test,dc=com</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <param> >>>>>>>>>> >>>>>>>>>> <param> >>>>>>>>>> >>>>>>>>>> <name>main.ldapRealm.memberAttributeValueTemplate</name> >>>>>>>>>> <value>cn={0},cn=users,dc=test,dc=com</value> >>>>>>>>>> <!-- also tried uid={0} --> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <param> >>>>>>>>>> >>>>>>>>>> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> >>>>>>>>>> <value>simple</value> >>>>>>>>>> </param> >>>>>>>>>> >>>>>>>>>> <param> >>>>>>>>>> <name>urls./**</name> >>>>>>>>>> <value>authcBasic</value> >>>>>>>>>> </param> >>>>>>>>>> </provider> >>>>>>>>>> >>>>>>>>>> <!-- the group principal mapping below is not likely >>>>>>>>>> what you want >>>>>>>>>> note that mapping of the hdfs group to admin. Also, we have >>>>>>>>>> disabled authorization above so there is no need >>>>>>>>>> for groups --> >>>>>>>>>> <provider> >>>>>>>>>> <role>identity-assertion</role> >>>>>>>>>> <name>Default</name> >>>>>>>>>> <enabled>true</enabled> >>>>>>>>>> <!--param> >>>>>>>>>> <name>group.principal.mapping</name> >>>>>>>>>> <value>*=users;hdfs=admin</value> >>>>>>>>>> </param--> >>>>>>>>>> </provider> >>>>>>>>>> >>>>>>>>>> <provider> >>>>>>>>>> <role>authorization</role> >>>>>>>>>> <name>AclsAuthz</name> >>>>>>>>>> <enabled>true</enabled> >>>>>>>>>> </provider> >>>>>>>>>> >>>>>>>>>> </gateway> >>>>>>>>>> >>>>>>>>>> We need to better document the difference between LDAP and AD for >>>>>>>>>> such deployments. >>>>>>>>>> >>>>>>>>>> I've also tried to document some of the changes that I made. >>>>>>>>>> Note that you don't have any authorization ACLs defined in the >>>>>>>>>> AclsAuthz provider so I disabled group lookup. >>>>>>>>>> That will only add complexity to your config - we can re-enable >>>>>>>>>> once authentication is working. >>>>>>>>>> >>>>>>>>>> Please go through this config and ensure that DNs, host and ports >>>>>>>>>> and system usernames match your environment. >>>>>>>>>> >>>>>>>>>> Hope this helps. >>>>>>>>>> >>>>>>>>>> --larry >>>>>>>>>> >>>>>>>>>> On Tue, Dec 8, 2015 at 5:16 AM, Darpan Patel <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> For this blocker issue let more information if it can help >>>>>>>>>>> fixing the authorization problem. >>>>>>>>>>> Please let me know if more details required. >>>>>>>>>>> (+ dev list) >>>>>>>>>>> >>>>>>>>>>> */etc/krb5.conf* >>>>>>>>>>> >>>>>>>>>>> [libdefaults] >>>>>>>>>>> renew_lifetime = 7d >>>>>>>>>>> forwardable = true >>>>>>>>>>> default_realm = HORTONWORKS.COM >>>>>>>>>>> ticket_lifetime = 24h >>>>>>>>>>> dns_lookup_realm = false >>>>>>>>>>> dns_lookup_kdc = false >>>>>>>>>>> #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>>>>>>>>> #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>>>>>>>>> >>>>>>>>>>> [domain_realm] >>>>>>>>>>> .hortonworks.com = HORTONWORKS.COM >>>>>>>>>>> HORTONWORKS.COm = HORTONWORKS.COM >>>>>>>>>>> >>>>>>>>>>> [logging] >>>>>>>>>>> default = FILE:/var/log/krb5kdc.log >>>>>>>>>>> admin_server = FILE:/var/log/kadmind.log >>>>>>>>>>> kdc = FILE:/var/log/krb5kdc.log >>>>>>>>>>> >>>>>>>>>>> [realms] >>>>>>>>>>> HORTONWORKS.COM = { >>>>>>>>>>> admin_server = KDC_SERVER_HOST >>>>>>>>>>> kdc = KDC_SERVER_HOST >>>>>>>>>>> } >>>>>>>>>>> *TEST.COM <http://TEST.COM>* = { >>>>>>>>>>> admin_server = WINDOWS_12_SERVER_AD_HOST >>>>>>>>>>> kdc = WINDOWS_12_SERVER_AD_HOST >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> */usr/hdp/current/knox-server/conf/gateway-site.xml* >>>>>>>>>>> >>>>>>>>>>> <configuration> >>>>>>>>>>> <property> >>>>>>>>>>> <name>*gateway.gateway.conf.dir*</name> >>>>>>>>>>> <value>deployments</value> >>>>>>>>>>> </property> >>>>>>>>>>> <property> >>>>>>>>>>> <name>*gateway.hadoop.kerberos.secured*</name> >>>>>>>>>>> <value>true</value> >>>>>>>>>>> </property> >>>>>>>>>>> <property> >>>>>>>>>>> <name>*gateway.path*</name> >>>>>>>>>>> <value>gateway</value> >>>>>>>>>>> </property> >>>>>>>>>>> <property> >>>>>>>>>>> <name>*gateway.port*</name> >>>>>>>>>>> <value>8443</value> >>>>>>>>>>> </property> >>>>>>>>>>> <property> >>>>>>>>>>> <name>*java.security.auth.login.config*</name> >>>>>>>>>>> <value>/*etc/knox/conf/krb5JAASLogin.conf*</value> >>>>>>>>>>> </property> >>>>>>>>>>> <property> >>>>>>>>>>> <name>*java.security.krb5.conf*</name> >>>>>>>>>>> <value>*/etc/krb5.conf*</value> >>>>>>>>>>> </property> >>>>>>>>>>> <property> >>>>>>>>>>> <name>sun.security.krb5.debug</name> >>>>>>>>>>> <value>true</value> >>>>>>>>>>> </property> >>>>>>>>>>> </configuration> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> */etc/knox/conf/krb5JAASLogin.conf* >>>>>>>>>>> >>>>>>>>>>> com.sun.security.jgss.initiate { >>>>>>>>>>> com.sun.security.auth.module.Krb5LoginModule required >>>>>>>>>>> renewTGT=true >>>>>>>>>>> doNotPrompt=true >>>>>>>>>>> useKeyTab=true >>>>>>>>>>> keyTab="/etc/security/keytabs/knox.service.keytab" >>>>>>>>>>> principal="knox/[email protected]" >>>>>>>>>>> isInitiator=true >>>>>>>>>>> storeKey=true >>>>>>>>>>> useTicketCache=true >>>>>>>>>>> client=true; >>>>>>>>>>> }; >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> DP >>>>>>>>>>> >>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>> From: Darpan Patel <[email protected]> >>>>>>>>>>> Date: 7 December 2015 at 17:59 >>>>>>>>>>> Subject: Need help setting up Knox for A/D integrated Kerberized >>>>>>>>>>> Cluster >>>>>>>>>>> To: [email protected] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I am stuck on an issue from last two days. I would be really >>>>>>>>>>> grateful if someone can help on this. >>>>>>>>>>> >>>>>>>>>>> We have HDP 2.3 implemented over 8 node cluster and the same >>>>>>>>>>> cluster has been Kerberized and later on we have integrated it with >>>>>>>>>>> Active >>>>>>>>>>> Directory (Which runs in the same VPN). We also verified that >>>>>>>>>>> Windows 2012 >>>>>>>>>>> A/D integration with Ranger works fine for defining policies and >>>>>>>>>>> audit log. >>>>>>>>>>> But I am stuck at Knox bit. I am trying to replicate the same >>>>>>>>>>> configuration >>>>>>>>>>> properties which I have set for Ranger LDAP-AD Integration. >>>>>>>>>>> >>>>>>>>>>> I am taking reference of the Hortonworks documentation and also >>>>>>>>>>> Apache Knox documentation. >>>>>>>>>>> >>>>>>>>>>> The A/D domain name is TEST.COM and all the users are under >>>>>>>>>>> Users >>>>>>>>>>> >>>>>>>>>>> [image: Inline images 1] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Under the Users we have few users one of the them is knox, >>>>>>>>>>> darpan, test,etc. >>>>>>>>>>> >>>>>>>>>>> When we issue following command on the node on which Knox Server >>>>>>>>>>> is running (topology name is default) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *curl -iv -k -u [email protected]:#123Password -X GET >>>>>>>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>>>>>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>" >>>>>>>>>>> OR* >>>>>>>>>>> >>>>>>>>>>> *curl -iv -k -u knox:#123Password -X GET >>>>>>>>>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>>>>>>>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>"* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Every time I see < HTTP/1.1 401 Unauthorized HTTP/1.1 401 >>>>>>>>>>> Unauthorized on the console. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Entries in the *gateway-audit.log *are like this : >>>>>>>>>>> >>>>>>>>>>> gateway-audit.log >>>>>>>>>>> ================== >>>>>>>>>>> 15/12/07 17:11:08 >>>>>>>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>>>>>>>>> 15/12/07 17:11:09 >>>>>>>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||authentication| >>>>>>>>>>> *principal*|*[email protected] <[email protected]>*|failure|*LDAP >>>>>>>>>>> authentication failed.* >>>>>>>>>>> 15/12/07 17:11:09 >>>>>>>>>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>>>>>>>>> status: 401 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 15/12/07 17:05:28 >>>>>>>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>>>>>>>>> 15/12/07 17:05:29 >>>>>>>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||authentication| >>>>>>>>>>> *principal*|knox|failure|*LDAP authentication failed.* >>>>>>>>>>> 15/12/07 17:05:29 >>>>>>>>>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>>>>>>>>> status: 401 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Gateway.log* >>>>>>>>>>> *===========* >>>>>>>>>>> 2015-12-07 17:05:28,620 INFO hadoop.gateway >>>>>>>>>>> (KnoxLdapRealm.java:getUserDn(550)) - Computed userDn: >>>>>>>>>>> cn=knox,CN=users,DC=test,DC=com using dnTemplate for principal: knox >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Following is the part of our *default.xml *topology: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> <gateway> >>>>>>>>>>> <provider> >>>>>>>>>>> <role>authentication</role> >>>>>>>>>>> <name>ShiroProvider</name> >>>>>>>>>>> <enabled>true</enabled> >>>>>>>>>>> <param> >>>>>>>>>>> <name>sessionTimeout</name> >>>>>>>>>>> <value>30</value> >>>>>>>>>>> </param> >>>>>>>>>>> <param> >>>>>>>>>>> <name>*main.ldapRealm*</name> >>>>>>>>>>> >>>>>>>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>>>>>>>>>> </param> >>>>>>>>>>> >>>>>>>>>>> <param> >>>>>>>>>>> <name>*main.ldapContextFactory*</name> >>>>>>>>>>> >>>>>>>>>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>>>>>>>>>> </param> >>>>>>>>>>> >>>>>>>>>>> <param> >>>>>>>>>>> <name>*main.ldapRealm.userDnTemplate* >>>>>>>>>>> </name> >>>>>>>>>>> >>>>>>>>>>> <value>cn={0},CN=users,DC=test,DC=com</value> >>>>>>>>>>> <!-- also tried following values --> >>>>>>>>>>> >>>>>>>>>>> <value>uid={0},CN=users,DC=test,DC=com</value> >>>>>>>>>>> <value>cn={0},DC=test,DC=com</value> >>>>>>>>>>> </param> >>>>>>>>>>> <param> >>>>>>>>>>> <name> >>>>>>>>>>> *main.ldapRealm.contextFactory.url*</name> >>>>>>>>>>> <!-- IP Address of the WINDOSWS 2012 >>>>>>>>>>> Acive Directory Server which works for Ranger --> >>>>>>>>>>> <value>*ldap://IP_OF_WINDOWS_AD:389* >>>>>>>>>>> </value> >>>>>>>>>>> </param> >>>>>>>>>>> <param> >>>>>>>>>>> <name> >>>>>>>>>>> *main.ldapRealm.authorizationEnabled*</name> >>>>>>>>>>> <value>true</value> >>>>>>>>>>> </param> >>>>>>>>>>> <param> >>>>>>>>>>> <name>*main.ldapRealm.searchBase*</name> >>>>>>>>>>> <value>cn=users,dc=test,dc=com</value> >>>>>>>>>>> </param> >>>>>>>>>>> <param> >>>>>>>>>>> <param> >>>>>>>>>>> <name> >>>>>>>>>>> *main.ldapRealm.memberAttributeValueTemplate*</name> >>>>>>>>>>> >>>>>>>>>>> <value>cn={0},cn=users,dc=test,dc=com</value> >>>>>>>>>>> <!-- also tried uid={0} --> >>>>>>>>>>> </param> >>>>>>>>>>> <param> >>>>>>>>>>> <name> >>>>>>>>>>> *main.ldapRealm.contextFactory.authenticationMechanism<*/name> >>>>>>>>>>> <value>simple</value> >>>>>>>>>>> </param> >>>>>>>>>>> <param> >>>>>>>>>>> <name>urls./**</name> >>>>>>>>>>> <value>authcBasic</value> >>>>>>>>>>> </param> >>>>>>>>>>> </provider> >>>>>>>>>>> >>>>>>>>>>> <provider> >>>>>>>>>>> <role>*identity-assertion*</role> >>>>>>>>>>> <name>Default</name> >>>>>>>>>>> <enabled>true</enabled> >>>>>>>>>>> <param> >>>>>>>>>>> <name>*group.principal.mapping*</name> >>>>>>>>>>> <value>*=users;hdfs=admin</value> >>>>>>>>>>> </param> >>>>>>>>>>> </provider> >>>>>>>>>>> >>>>>>>>>>> <provider> >>>>>>>>>>> <role>*authorization*</role> >>>>>>>>>>> <name>AclsAuthz</name> >>>>>>>>>>> <enabled>true</enabled> >>>>>>>>>>> </provider> >>>>>>>>>>> >>>>>>>>>>> </gateway> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> And following is the console output while trying to access >>>>>>>>>>> webhdfs using curl >>>>>>>>>>> >>>>>>>>>>> curl -iv -k -u knox:#123Password -X GET " >>>>>>>>>>> https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>>>>>>>>> " >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Console Output:* >>>>>>>>>>> ---------------- >>>>>>>>>>> >>>>>>>>>>> * About to connect() to localhost port 8443 (#0) >>>>>>>>>>> * Trying ::1... >>>>>>>>>>> * Connected to localhost (::1) port 8443 (#0) >>>>>>>>>>> * Initializing NSS with certpath: sql:/etc/pki/nssdb >>>>>>>>>>> * skipping SSL peer certificate verification >>>>>>>>>>> * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA >>>>>>>>>>> * Server certificate: >>>>>>>>>>> * subject: >>>>>>>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>>>>>>>>> * start date: Nov 27 20:36:22 2015 GMT >>>>>>>>>>> * expire date: Nov 26 20:36:22 2016 GMT >>>>>>>>>>> * common name: FQDN_OF_My_gateway_HOST >>>>>>>>>>> * issuer: >>>>>>>>>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>>>>>>>>> * Server auth using Basic with user 'knox' >>>>>>>>>>> > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1 >>>>>>>>>>> > Authorization: Basic a25veDojMTIzUGFzc3dvcmQ= >>>>>>>>>>> > User-Agent: curl/7.29.0 >>>>>>>>>>> > Host: localhost:8443 >>>>>>>>>>> > Accept: */* >>>>>>>>>>> > >>>>>>>>>>> < HTTP/1.1 401 Unauthorized >>>>>>>>>>> HTTP/1.1 401 Unauthorized >>>>>>>>>>> * Authentication problem. Ignoring this. >>>>>>>>>>> < WWW-Authenticate: BASIC realm="application" >>>>>>>>>>> WWW-Authenticate: BASIC realm="application" >>>>>>>>>>> < Content-Length: 0 >>>>>>>>>>> Content-Length: 0 >>>>>>>>>>> < Server: Jetty(8.1.14.v20131031) >>>>>>>>>>> Server: Jetty(8.1.14.v20131031) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Please let me know if any additional information is required. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> DP >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
