From David Yan <[email protected]>
Date Sat 00:12
Can you please provide any exception stacktrace in the dtgateway.log
file when that happens?
I reran the dtgateway installation wizard, which failed (same as before)
with error msg:
| DFS directory cannot be written to with error message "Mkdirs failed
to create /user/dtadmin/datatorrent (exists=false,
cwd=file:/opt/datatorrent/releases/3.4.0)"
The corresponding exception in dtgateway.log is:
| 2016-09-26 13:43:54,767 ERROR com.datatorrent.gateway.I: DFS Directory
cannot be written to with exception:
| java.io.IOException: Mkdirs failed to create /user/dtadmin/datatorrent
(exists=false, cwd=file:/opt/datatorrent/releases/3.4.0)
| at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)
| at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
| at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
| at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
| at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
| at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
| at
com.datatorrent.stram.client.FSAgent.createFile(FSAgent.java:77)
| at com.datatorrent.gateway.I.h(gc:324)
| at com.datatorrent.gateway.I.h(gc:284)
| at
com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:136)
| at
com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:171)
| at
com.datatorrent.gateway.resources.ws.v2.ConfigResource.setConfigProperty(fc:34)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| at java.lang.reflect.Method.invoke(Method.java:498)
| at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
| at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
| at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
| at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
| at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
| at
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)
| at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
| at
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)
| at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
| at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
| at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
| at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
| at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
| at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
| at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
| at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
| at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
| at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
| at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
| at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
| at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
| at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:457)
| at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
| at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
| at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
| at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
| at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
| at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
| at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
| at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
| at org.eclipse.jetty.server.Server.handle(Server.java:368)
| at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
| at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
| at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
| at
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
| at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
| at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
| at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
| at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
| at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
| at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
| at java.lang.Thread.run(Thread.java:745)
| 2016-09-26 13:43:54,772 INFO
com.datatorrent.gateway.resources.ws.v2.WSResource: Caught exception in
processing web service:
com.datatorrent.stram.client.DTConfiguration$ConfigException: DFS
directory cannot be written to with error message "Mkdirs failed to
create /user/dtadmin/datatorrent (exists=false,
cwd=file:/opt/datatorrent/releases/3.4.0)"
Also, when I shutdown the gateway, I get a different exception in
dtgateway.log:
| 2016-09-26 13:40:44,366 INFO com.datatorrent.gateway.DTGateway:
Shutting down
| 2016-09-26 13:40:44,427 ERROR com.datatorrent.gateway.I: DFS Directory
cannot be written to with exception:
| java.io.IOException: Filesystem closed
| at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
| at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2041)
| at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707)
| at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703)
| at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
| at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714)
| at
com.datatorrent.stram.client.FSAgent.deleteFile(FSAgent.java:94)
| at com.datatorrent.gateway.I.h(gc:30)
| at com.datatorrent.gateway.l.K(pc:158)
| at com.datatorrent.gateway.l.h(pc:65)
| at com.datatorrent.gateway.K.run(yc:156)
| at java.lang.Thread.run(Thread.java:745)
| 2016-09-26 13:40:44,444 INFO com.datatorrent.gateway.DTGateway:
Shutdown complete
Thanks much.
-david
On Tue, Sep 13, 2016 at 4:06 PM, <[email protected]> wrote:
> I have the dtgateway running (community edition) with apex/malhar 3.4 and
hadoop 2.7.4 on SuSE SLES 12.1 on an ARM64 server (aarch64).
>
> In the Installation Wizard, I set the DFS location to:
>
> hdfs://namenode:9000/user/dtadmin/datatorrent
>
> and the gateway saves the hadoop configuration and restarts successfully,
but I get an error:
>
> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, expected:
file:///
>
> the 'service dtgateway status' command says it's running.
> The log file has the same error as above, but nothing else useful.
>
> Is there a manual way to move the demo apps into the gateway?
>
> If I ignore this error and try to upload or import a demo app, I get the
same error.
>
> What's odd is that the gateway does write into the HDFS directory:
>
> Found 6 items
> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35
/user/dtadmin/datatorrent/appPackages
> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35
/user/dtadmin/datatorrent/apps
> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35
/user/dtadmin/datatorrent/conf
> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35
/user/dtadmin/datatorrent/dashboards
> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35
/user/dtadmin/datatorrent/licenses
> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35
/user/dtadmin/datatorrent/systemAlerts
>
>
> I've tried playing with different ways to configure the DFS location, but
nothing works.
> If I use an actual local filesystem directory, I get a different errro
from the installation wizard (Non-DFS file system is used:
org.apache.hadoop.fs.LocalFileSystem), which makes sense. But if I ignore this
error, the app wizard successsfully writes packages into the local directory, but
I can't launch them (no surprise there).
>
> Anybody have any ideas what I can do?
>
>
> thanks much.
> -david
