Well, if you can be more specific about what is confusing then we can try and make it more clear. Please try and follow my instructions and see how far you get.
On Thu, Jul 2, 2015 at 6:34 PM, Aneela Saleem <[email protected]> wrote: > Thank you so much Larry. Thanks for your help. > > Actually i feel the documentation and logging is very confusing. > > On Fri, Jul 3, 2015 at 3:13 AM, larry mccay <[email protected]> wrote: > >> Oops - hit send too early... >> >> At that point, you should be good to try the curl command - you have to >> make sure that you replace the "sandbox" path element with "cluster1" or >> whatever your topology is. >> >> Hope that gets you going. >> >> --larry >> >> On Thu, Jul 2, 2015 at 6:11 PM, larry mccay <[email protected]> >> wrote: >> >>> You aren't asking any foolish questions. >>> If you don't have proper instructions to follow then you will find >>> yourself in a jam. >>> >>> You have actually already contributed to Knox by finding a bug that we >>> will fix in https://issues.apache.org/jira/browse/KNOX-564 - all >>> contributions from the community are valuable. >>> >>> Now.... >>> >>> Let's step back and ensure that you have a solid foundation for using >>> Knox in your environment. >>> >>> 1. You need a Hadoop cluster install that you can point your instance of >>> Knox at - this is available from distro vendors as VMs that you can >>> download, etc >>> 2. You need to have Knox installed in the VMs or external to the VMs but >>> somewhere that it can see the Hadoop cluster in the VMs. >>> 3. Since you have had Knox running already - you don't need to create >>> the master secret but if it were a brand new install then you would and you >>> could follow the instructions in the apache docs for QuickStart >>> 4. The sandbox.xml topology is intended to be used with such VMs - you >>> can copy it to cluster1 or whatever you like >>> 5. The authentication provider in sandbox.xml is the Shiro provider with >>> configuration for HTTP Basic authentication against an LDAP server >>> - since it is expecting to authenticate against the demo ldap >>> server, you will need to start the Demo LDAP Server - see step 6 in the >>> QuickStart section of the apache docs. >>> >>> cd {GATEWAY_HOME}bin/ldap.sh start >>> >>> >>> - since it is expecting HTTP Basic authentication, you need to >>> provide username and password using the -u username:password switch in curl >>> - see examples in the apache docs for the exact command line. >>> >>> curl -i -k *-u guest:guest-password* -X GET >>> 'https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS' >>> >>> 6. >>> >>> >>> On Thu, Jul 2, 2015 at 5:51 PM, Aneela Saleem <[email protected]> >>> wrote: >>> >>>> I'm very much confused with the documentation and could not getting >>>> points related to configuration. >>>> I'm so sorry if I am posting foolish questions >>>> >>>> >>>> On Fri, Jul 3, 2015 at 2:49 AM, Aneela Saleem <[email protected]> >>>> wrote: >>>> >>>>> Thanks Larry. >>>>> >>>>> On Fri, Jul 3, 2015 at 2:41 AM, larry mccay <[email protected]> >>>>> wrote: >>>>> >>>>>> We need a separate jira for the NullPointerException. >>>>>> What you have in KNOX-563 should be discussed on the user@ list. >>>>>> As I indicated in the jira - you need to supply credentials and make >>>>>> sure that the LDAP server is running. >>>>>> >>>>>> I will close KNOX-563 and we can continue to discuss your problems >>>>>> here on the mailing list. >>>>>> >>>>>> On Thu, Jul 2, 2015 at 4:49 PM, Aneela Saleem <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hi Kevin, >>>>>>> >>>>>>> Please follow this URL >>>>>>> >>>>>>> https://issues.apache.org/jira/browse/KNOX-563 >>>>>>> >>>>>>> On Fri, Jul 3, 2015 at 1:22 AM, Kevin Minder < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Anella or Larry, >>>>>>>> Can one of you please file a jira about the NPE. Knox should >>>>>>>> certainly handle that much more gracefully. That would be very much >>>>>>>> appreciated. Please attache the topology file (e.g. Sandbox.xml) that >>>>>>>> resulted in this error if possible. >>>>>>>> Kevin. >>>>>>>> >>>>>>>> From: larry mccay <[email protected]> >>>>>>>> Reply-To: "[email protected]" <[email protected]> >>>>>>>> Date: Thursday, July 2, 2015 at 3:37 PM >>>>>>>> To: "[email protected]" <[email protected]> >>>>>>>> Subject: Re: Knox - Hostname was NOT found >>>>>>>> >>>>>>>> You are probably getting far enough now to be hitting errors due >>>>>>>> to your lack of providers. Start with sandbox.xml config and start the >>>>>>>> ldap >>>>>>>> server. See the apache docs for details. >>>>>>>> On Jul 2, 2015 3:14 PM, "Aneela Saleem" <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hey Larry, >>>>>>>>> >>>>>>>>> i got the following error >>>>>>>>> >>>>>>>>> <title>Error 500 Server Error</title> >>>>>>>>> </head> >>>>>>>>> <body><h2>HTTP ERROR 500</h2> >>>>>>>>> <p>Problem accessing /gateway/cluster1/webhdfs/v1/. >>>>>>>>> >>>>>>>>> And the gateway.log file contains: >>>>>>>>> >>>>>>>>> 2015-07-03 00:09:04,560 ERROR hadoop.gateway >>>>>>>>> (AbstractGatewayFilter.java:doFilter(69)) - Failed to execute filter: >>>>>>>>> java.lang.NullPointerException >>>>>>>>> 2015-07-03 00:09:04,560 ERROR hadoop.gateway >>>>>>>>> (GatewayFilter.java:doFilter(135)) - Gateway processing failed: >>>>>>>>> javax.servlet.ServletException: java.lang.NullPointerException >>>>>>>>> javax.servlet.ServletException: java.lang.NullPointerException >>>>>>>>> at >>>>>>>>> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:70) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:129) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:121) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >>>>>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:370) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) >>>>>>>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >>>>>>>>> at >>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> Caused by: java.lang.NullPointerException >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jul 3, 2015 at 12:07 AM, larry mccay < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Try lower case webhdfs as in: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 'https://localhost:8443/gateway/cluster1/webhdfs/v1/?op=GETHOMEDIRECTORY' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jul 2, 2015 at 2:59 PM, Aneela Saleem < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Dear larry, >>>>>>>>>>> >>>>>>>>>>> The gateway.log file contains the following logs: >>>>>>>>>>> >>>>>>>>>>> 2015-07-02 23:43:16,391 INFO hadoop.gateway >>>>>>>>>>> (GatewayServer.java:handleCreateDeployment(427)) - Deploying >>>>>>>>>>> topology >>>>>>>>>>> cluster1 to >>>>>>>>>>> /home/hduser/knox-0.6.0/bin/../data/deployments/cluster1.war.14e501479a0 >>>>>>>>>>> 2015-07-02 23:43:16,392 INFO hadoop.gateway >>>>>>>>>>> (DeploymentFactory.java:createDeployment(82)) - Configured services >>>>>>>>>>> directory is /home/hduser/knox-0.6.0/bin/../data/services >>>>>>>>>>> 2015-07-02 23:43:16,428 INFO hadoop.gateway >>>>>>>>>>> (DefaultGatewayServices.java:initializeContribution(180)) - >>>>>>>>>>> Credential >>>>>>>>>>> store found for the cluster: cluster1 - no need to create one. >>>>>>>>>>> 2015-07-02 23:44:40,453 WARN hadoop.gateway >>>>>>>>>>> (GatewayFilter.java:doFilter(152)) - Failed to match path >>>>>>>>>>> /WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>>> 2015-07-02 23:44:51,722 WARN hadoop.gateway >>>>>>>>>>> (GatewayFilter.java:doFilter(152)) - Failed to match path >>>>>>>>>>> /WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>>> 2015-07-02 23:45:30,783 WARN hadoop.gateway >>>>>>>>>>> (GatewayFilter.java:doFilter(152)) - Failed to match path >>>>>>>>>>> /WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>>> >>>>>>>>>>> And i'm just a beginner. I'm trying to configure Knox Topology >>>>>>>>>>> to connect to HDFS, using default services provided. And i'm >>>>>>>>>>> validating >>>>>>>>>>> connection to services. >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 2, 2015 at 11:50 PM, larry mccay < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Please check the {GATEWAY_HOME}/logs/gateway.log file for >>>>>>>>>>>> errors during deployment. >>>>>>>>>>>> >>>>>>>>>>>> I notice that you have no providers described in cluster1.xml >>>>>>>>>>>> - this may be the root of your problem. >>>>>>>>>>>> I don't think that I have ever even tried that. >>>>>>>>>>>> >>>>>>>>>>>> What are you expecting the authentication behavior to be >>>>>>>>>>>> there? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 2, 2015 at 2:27 PM, Aneela Saleem < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> ${KNOX_HOME}/knox/conf/topologies has the following >>>>>>>>>>>>> contents: >>>>>>>>>>>>> >>>>>>>>>>>>> admin.xml cluster1.xml README sandbox.xml >>>>>>>>>>>>> >>>>>>>>>>>>> cluster1.xml is the topology descriptor file created by me. >>>>>>>>>>>>> >>>>>>>>>>>>> cluster1.xml has following contents: >>>>>>>>>>>>> >>>>>>>>>>>>> <topology> >>>>>>>>>>>>> <gateway> >>>>>>>>>>>>> </gateway> >>>>>>>>>>>>> <service> >>>>>>>>>>>>> <role>NAMENODE</role> >>>>>>>>>>>>> <url>hdfs:// namenode-host :8020</url> >>>>>>>>>>>>> </service> >>>>>>>>>>>>> >>>>>>>>>>>>> <service> >>>>>>>>>>>>> <role>JOBTRACKER</role> >>>>>>>>>>>>> <url>rpc:// jobtracker-host :8050</url> >>>>>>>>>>>>> </service> >>>>>>>>>>>>> >>>>>>>>>>>>> <service> >>>>>>>>>>>>> <role>RESOURCEMANAGER</role> >>>>>>>>>>>>> <url>http://red3:8088/ws</url> >>>>>>>>>>>>> </service> >>>>>>>>>>>>> >>>>>>>>>>>>> <service> >>>>>>>>>>>>> <role>WEBHDFS</role> >>>>>>>>>>>>> <url>http://localhost:50070/webhdfs</url> >>>>>>>>>>>>> </service> >>>>>>>>>>>>> >>>>>>>>>>>>> <service> >>>>>>>>>>>>> <role>WEBHCAT</role> >>>>>>>>>>>>> <url>http://webcat-host :50111/templeton</url> >>>>>>>>>>>>> </service> >>>>>>>>>>>>> >>>>>>>>>>>>> <service> >>>>>>>>>>>>> <role>OOZIE</role> >>>>>>>>>>>>> <url>http://oozie-host :11000/oozie</url> >>>>>>>>>>>>> </service> >>>>>>>>>>>>> >>>>>>>>>>>>> <service> >>>>>>>>>>>>> <role>WEBHBASE</role> >>>>>>>>>>>>> <url>http://webhbase-host :60080</url> >>>>>>>>>>>>> </service> >>>>>>>>>>>>> </topology> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> admin.xml has default contents. I did not make any changes. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 2, 2015 at 11:15 PM, Steve Howard < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> To be clear, the error isn't hostname not found (that just >>>>>>>>>>>>>> means it wasn't in the DNS cache on your computer), it's that >>>>>>>>>>>>>> the URL can't >>>>>>>>>>>>>> be found by the knox server. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you post the contents of your >>>>>>>>>>>>>> ${KNOX_HOME}/knox/conf/topologies directory? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jul 2, 2015 at 1:50 PM, Aneela Saleem < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm trying to connect to HDFS through Knox gateway. I run >>>>>>>>>>>>>>> the following command: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> curl -vk >>>>>>>>>>>>>>> https://localhost:8443/gateway/cluster1/WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> when i run this command i get the following error: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * Hostname was NOT found in DNS cache >>>>>>>>>>>>>>> * Trying 127.0.0.1... >>>>>>>>>>>>>>> * Connected to localhost (127.0.0.1) port 8443 (#0) >>>>>>>>>>>>>>> * successfully set certificate verify locations: >>>>>>>>>>>>>>> * CAfile: none >>>>>>>>>>>>>>> CApath: /etc/ssl/certs >>>>>>>>>>>>>>> * SSLv3, TLS handshake, Client hello (1): >>>>>>>>>>>>>>> * SSLv3, TLS handshake, Server hello (2): >>>>>>>>>>>>>>> * SSLv3, TLS handshake, CERT (11): >>>>>>>>>>>>>>> * SSLv3, TLS handshake, Server key exchange (12): >>>>>>>>>>>>>>> * SSLv3, TLS handshake, Server finished (14): >>>>>>>>>>>>>>> * SSLv3, TLS handshake, Client key exchange (16): >>>>>>>>>>>>>>> * SSLv3, TLS change cipher, Client hello (1): >>>>>>>>>>>>>>> * SSLv3, TLS handshake, Finished (20): >>>>>>>>>>>>>>> * SSLv3, TLS change cipher, Client hello (1): >>>>>>>>>>>>>>> * SSLv3, TLS handshake, Finished (20): >>>>>>>>>>>>>>> * SSL connection using ECDHE-RSA-DES-CBC3-SHA >>>>>>>>>>>>>>> * Server certificate: >>>>>>>>>>>>>>> * subject: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; >>>>>>>>>>>>>>> CN=localhost >>>>>>>>>>>>>>> * start date: 2015-06-29 21:39:18 GMT >>>>>>>>>>>>>>> * expire date: 2016-06-28 21:39:18 GMT >>>>>>>>>>>>>>> * issuer: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; >>>>>>>>>>>>>>> CN=localhost >>>>>>>>>>>>>>> * SSL certificate verify result: self signed certificate >>>>>>>>>>>>>>> (18), continuing anyway. >>>>>>>>>>>>>>> > GET /gateway/cluster1/WEBHDFS/v1?op=GETHOMEDIRECTORY >>>>>>>>>>>>>>> HTTP/1.1 >>>>>>>>>>>>>>> > User-Agent: curl/7.35.0 >>>>>>>>>>>>>>> > Host: localhost:8443 >>>>>>>>>>>>>>> > Accept: */* >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> < HTTP/1.1 404 Not Found >>>>>>>>>>>>>>> < Content-Length: 0 >>>>>>>>>>>>>>> * Server Jetty(8.1.14.v20131031) is not blacklisted >>>>>>>>>>>>>>> < Server: Jetty(8.1.14.v20131031) >>>>>>>>>>>>>>> < >>>>>>>>>>>>>>> * Connection #0 to host localhost left intact >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can anyone please help me in tackling this issue? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
