FYI, I'm getting the exact same error after I set fs.defaultFS to file:/// in core-site.xml in the hadoop installation and restarted yarn.
Wrong FS: hdfs://localhost:9000/user/dtadmin/datatorrent, expected: file:/// For now, dtgateway only works if fs.defaultFS is the same FS as the "DFS location" in the dtgateway configuration. I'll log this as a bug. Please double check the correct value is picked up for fs.defaultFS in your installation. I think that value should be in core-site.xml, not hdfs-site.xml, and if not specified in core-site.xml, it would default to file:/// David On Wed, Sep 28, 2016 at 4:02 PM, <[email protected]> wrote: > On 2016-09-28 14:56, Vlad Rozov wrote: > >> Check namenode logs. Are there any exceptions/errors raised at the >> same time as the gateway exception? >> > > Nope. Nothing interesting there. > > > > >> On 9/28/16 14:43, [email protected] wrote: >> >>> On 2016-09-28 14:09, David Yan wrote: >>> >>>> dtgateway does not use the native libraries. >>>> >>>> I just tried a clean install of RTS 3.5.0 community edition on plain >>>> vanilla hadoop 2.7.3 on ubuntu 14.04 and was able to launch the pi >>>> demo from the web interface. I was able to do the same even with the >>>> native libraries removed. >>>> >>> >>> Thanks. I thought it was a long shot. >>> >>> What is your fs.defaultFS property in hadoop? Can you verify that it >>>> points to your hdfs (hdfs://namenode:9000/) ? >>>> >>> >>> In $HADOOP_CONF_DIR/hdfs-site.xml, fs.defaultFS is hdfs://namenode:9000 >>> >>> Note also that if I try to give a URL to the dtgateway Installation >>> Wizard >>> that isn't the namenode host/port, it complains immediately, instead of >>> proceeding to the next page. >>> >>> Unfortunately I don't have access to an ARM64 so any finding you post >>>> here would be helpful. >>>> >>> >>> I'm not surprised. A classic chicken-and-egg problem. There aren't a >>> lot of machines around, so the software doesn't get a lot of attention, so >>> people don't buy machines. >>> >>> >>> >>>> David >>>> >>>> On Wed, Sep 28, 2016 at 11:04 AM, <[email protected]> wrote: >>>> >>>> I had a thought of what might be the problem: >>>>> the hadoop native library doesn't build on SUSE/SLES ARM64, >>>>> so I'm running with the pure Java library. >>>>> >>>>> Is it possible that the dtgateway is trying to use the native >>>>> library >>>>> directly? >>>>> >>>>> Thanks, >>>>> -david >>>>> >>>>> On 2016-09-26 14:45, David Yan wrote: >>>>> >>>>> Do you see any exception stacktrace in the log when this error >>>>> occurred: >>>>> >>>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, expected: >>>>> file:/// >>>>> >>>>> David >>>>> >>>>> On Mon, Sep 26, 2016 at 2:39 PM, <[email protected]> wrote: >>>>> >>>>> From David Yan <[email protected]> >>>>> Date Sat 00:12 >>>>> >>>>> Can you please provide any exception stacktrace in the >>>>> dtgateway.log file when that happens? >>>>> >>>>> I reran the dtgateway installation wizard, which failed (same as >>>>> before) with error msg: >>>>> >>>>> | DFS directory cannot be written to with error message "Mkdirs >>>>> failed to create /user/dtadmin/datatorrent (exists=false, >>>>> cwd=file:/opt/datatorrent/releases/3.4.0)" >>>>> >>>>> The corresponding exception in dtgateway.log is: >>>>> >>>>> | 2016-09-26 13:43:54,767 ERROR com.datatorrent.gateway.I: DFS >>>>> Directory cannot be written to with exception: >>>>> | java.io.IOException: Mkdirs failed to create >>>>> /user/dtadmin/datatorrent (exists=false, >>>>> cwd=file:/opt/datatorrent/releases/3.4.0) >>>>> | at >>>>> >>>> >>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileS >>>> ystem.java:455) >>>> >>>>> | at >>>>> >>>> >>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileS >>>> ystem.java:440) >>>> >>>>> | at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) >>>>> | at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) >>>>> | at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) >>>>> | at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778) >>>>> | at >>>>> com.datatorrent.stram.client.FSAgent.createFile(FSAgent.java:77) >>>>> | at com.datatorrent.gateway.I.h(gc:324) >>>>> | at com.datatorrent.gateway.I.h(gc:284) >>>>> | at >>>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:136) >>>>> | at >>>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:171) >>>>> | at >>>>> >>>> >>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.setCo >>>> nfigProperty(fc:34) >>>> >>>>> | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> | at >>>>> >>>> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>>> ssorImpl.java:62) >>>> >>>>> | at >>>>> >>>> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>>> thodAccessorImpl.java:43) >>>> >>>>> | at java.lang.reflect.Method.invoke(Method.java:498) >>>>> | at >>>>> >>>> >>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invo >>>> ke(JavaMethodInvokerFactory.java:60) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractRes >>>> ourceMethodDispatchProvider$ResponseOutInvoker._dispatch(A >>>> bstractResourceMethodDispatchProvider.java:205) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJav >>>> aMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept( >>>> HttpMethodRule.java:288) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>>> t(RightHandPathRule.java:147) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept( >>>> SubLocatorRule.java:134) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>>> t(RightHandPathRule.java:147) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept( >>>> SubLocatorRule.java:134) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>>> t(RightHandPathRule.java:147) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accep >>>> t(ResourceClassRule.java:108) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>>> t(RightHandPathRule.java:147) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule >>>> .accept(RootResourceClassesRule.java:84) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl._h >>>> andleRequest(WebApplicationImpl.java:1469) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl._h >>>> andleRequest(WebApplicationImpl.java:1400) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha >>>> ndleRequest(WebApplicationImpl.java:1349) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha >>>> ndleRequest(WebApplicationImpl.java:1339) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.spi.container.servlet.WebComponent.service(We >>>> bComponent.java:416) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.spi.container.servlet.ServletContainer.servic >>>> e(ServletContainer.java:537) >>>> >>>>> | at >>>>> >>>> >>>> com.sun.jersey.spi.container.servlet.ServletContainer.servic >>>> e(ServletContainer.java:699) >>>> >>>>> | at >>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>>> | at >>>>> >>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669) >>>> >>>> | at >>>>> >>>> >>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan >>>> dler.java:457) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.session.SessionHandler.doHandle( >>>> SessionHandler.java:229) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle( >>>> ContextHandler.java:1075) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand >>>> ler.java:384) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.session.SessionHandler.doScope( >>>> SessionHandler.java:193) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.handler.ContextHandler.doScope( >>>> ContextHandler.java:1009) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped >>>> Handler.java:135) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.handler.HandlerCollection.handle( >>>> HandlerCollection.java:154) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl >>>> erWrapper.java:116) >>>> >>>>> | at org.eclipse.jetty.server.Server.handle(Server.java:368) >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.AbstractHttpConnection.handleReques >>>> t(AbstractHttpConnection.java:489) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.AbstractHttpConnection.content(Abst >>>> ractHttpConnection.java:953) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandl >>>> er.content(AbstractHttpConnection.java:1014) >>>> >>>>> | at >>>>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) >>>>> | at >>>>> >>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) >>>> >>>> | at >>>>> >>>> >>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHtt >>>> pConnection.java:82) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(Select >>>> ChannelEndPoint.java:628) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectC >>>> hannelEndPoint.java:52) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued >>>> ThreadPool.java:608) >>>> >>>>> | at >>>>> >>>> >>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT >>>> hreadPool.java:543) >>>> >>>>> | at java.lang.Thread.run(Thread.java:745) >>>>> | 2016-09-26 13:43:54,772 INFO >>>>> com.datatorrent.gateway.resources.ws.v2.WSResource: Caught exception >>>>> in processing web service: >>>>> com.datatorrent.stram.client.DTConfiguration$ConfigException: DFS >>>>> directory cannot be written to with error message "Mkdirs failed to >>>>> create /user/dtadmin/datatorrent (exists=false, >>>>> cwd=file:/opt/datatorrent/releases/3.4.0)" >>>>> >>>>> Also, when I shutdown the gateway, I get a different exception in >>>>> dtgateway.log: >>>>> >>>>> | 2016-09-26 13:40:44,366 INFO com.datatorrent.gateway.DTGateway: >>>>> Shutting down >>>>> | 2016-09-26 13:40:44,427 ERROR com.datatorrent.gateway.I: DFS >>>>> Directory cannot be written to with exception: >>>>> | java.io.IOException: Filesystem closed >>>>> | at >>>>> org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) >>>>> | at >>>>> org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2041) >>>>> | at >>>>> >>>> >>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall( >>>> DistributedFileSystem.java:707) >>>> >>>>> | at >>>>> >>>> >>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall( >>>> DistributedFileSystem.java:703) >>>> >>>>> | at >>>>> >>>> >>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst >>>> emLinkResolver.java:81) >>>> >>>>> | at >>>>> >>>> >>>> org.apache.hadoop.hdfs.DistributedFileSystem.delete(Distribu >>>> tedFileSystem.java:714) >>>> >>>>> | at >>>>> com.datatorrent.stram.client.FSAgent.deleteFile(FSAgent.java:94) >>>>> | at com.datatorrent.gateway.I.h(gc:30) >>>>> | at com.datatorrent.gateway.l.K(pc:158) >>>>> | at com.datatorrent.gateway.l.h(pc:65) >>>>> | at com.datatorrent.gateway.K.run(yc:156) >>>>> | at java.lang.Thread.run(Thread.java:745) >>>>> | 2016-09-26 13:40:44,444 INFO com.datatorrent.gateway.DTGateway: >>>>> Shutdown complete >>>>> >>>>> Thanks much. >>>>> -david >>>>> >>>>> On Tue, Sep 13, 2016 at 4:06 PM, <[email protected]> wrote: >>>>> >>>>> I have the dtgateway running (community edition) with >>>>> apex/malhar 3.4 and hadoop 2.7.4 on SuSE SLES 12.1 on an ARM64 >>>>> server (aarch64). >>>>> >>>>> In the Installation Wizard, I set the DFS location to: >>>>> >>>>> hdfs://namenode:9000/user/dtadmin/datatorrent >>>>> >>>>> and the gateway saves the hadoop configuration and restarts >>>>> successfully, but I get an error: >>>>> >>>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, >>>>> expected: file:/// >>>>> >>>>> the 'service dtgateway status' command says it's running. >>>>> The log file has the same error as above, but nothing else >>>>> useful. >>>>> >>>>> Is there a manual way to move the demo apps into the >>>>> gateway? >>>>> >>>>> If I ignore this error and try to upload or import a demo >>>>> app, I get the same error. >>>>> >>>>> What's odd is that the gateway does write into the HDFS >>>>> directory: >>>>> >>>>> Found 6 items >>>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>>> /user/dtadmin/datatorrent/appPackages >>>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>>> /user/dtadmin/datatorrent/apps >>>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>>> /user/dtadmin/datatorrent/conf >>>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>>> /user/dtadmin/datatorrent/dashboards >>>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>>> /user/dtadmin/datatorrent/licenses >>>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>>> /user/dtadmin/datatorrent/systemAlerts >>>>> >>>>> I've tried playing with different ways to configure the DFS >>>>> location, but nothing works. >>>>> If I use an actual local filesystem directory, I get a >>>>> different errro from the installation wizard (Non-DFS file system >>>>> is used: org.apache.hadoop.fs.LocalFileSystem), which makes sense. >>>>> But if I ignore this error, the app wizard successsfully writes >>>>> packages into the local directory, but I can't launch them (no >>>>> surprise there). >>>>> >>>>> Anybody have any ideas what I can do? >>>>> >>>>> thanks much. >>>>> -david >>>>> >>>> >>> >
