Added back users@ to the thread. On Mon, Sep 26, 2016 at 5:03 PM, David Yan <[email protected]> wrote:
> Can you try one of the released hadoop versions, like 2.7.3 or 2.7.2 or > 2.6.4? > We will check that sha commit on our side as well. > > David > > On Mon, Sep 26, 2016 at 4:40 PM, <[email protected]> wrote: > >> On 2016-09-26 16:02, David Yan wrote: >> >>> From your first email, you said you're using hadoop 2.7.4, but that >>> hadoop version has not been released. >>> >> >> Did you build it yourself? If so, can you provide a github tag or >>> something similar so we can build that ourselves so that we can >>> reproduce the issue? >>> >> >> Yes. Built from apache git repo, branch branch-2.7, date Aug29, sha is: >> >> 32a86f199cd8e7f32c264af55e3459e4b4751963 >> >> I changed 1 file: >> >> diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml >> index 1765363..c35df02 100644 >> --- a/hadoop-project/pom.xml >> +++ b/hadoop-project/pom.xml >> @@ -74,3 +74,3 @@ >> <!-- define the protobuf JAR version >> --> >> - <protobuf.version>2.5.0</protobuf.version> >> + <protobuf.version>2.6.1</protobuf.version> >> <protoc.path>${env.HADOOP_PROTOC_PATH}</protoc.path> >> @@ -95,3 +95,3 @@ >> <!-- Plugin versions and config --> >> - <maven-surefire-plugin.argLine>-Xmx4096m -XX:MaxPermSize=768m >> -XX:+HeapDumpOnOutOfMemoryError</maven-surefire-plugin.argLine> >> + <maven-surefire-plugin.argLine>-Xmx4096m >> -XX:+HeapDumpOnOutOfMemoryError</maven-surefire-plugin.argLine> >> <maven-surefire-plugin.version>2.17</maven-surefire-plugin.version> >> >> >>> Also, what Apex or DT RTS version are you using? >>> >> >> 3.4.0, also built from Apache source, using tar file: >> apex-3.4.0-source-release.tar.gz >> >> Likewise for malhar, using apache-apex-malhar-3.4.0-source-release.tar.gz >> >> I had to change one thing to get it to build. >> This disables the maven check against deploying from a snapshot. >> >> apache-apex-malhar-3.4.0> diff -p -C1 contrib/pom.xml contrib/pom.xml.orig >> *** contrib/pom.xml Tue Sep 13 12:02:27 2016 >> --- contrib/pom.xml.orig Fri May 20 00:19:42 2016 >> *************** >> *** 199,201 **** >> </plugin> >> - <!-- SOFTIRON patch >> <plugin> >> --- 199,200 ---- >> *************** >> *** 214,216 **** >> </plugin> >> ! --> >> <plugin> >> --- 213,215 ---- >> </plugin> >> ! >> <plugin> >> >> >> Note: I also tried malhar v3.5 on top of apex3.4, and got the same >> results. >> >> I think that covers everything. >> >> thanks again, >> -david >> >> >> >> >>> Thanks! >>> >>> David >>> >>> On Mon, Sep 26, 2016 at 3:15 PM, <[email protected]> wrote: >>> >>> On 2016-09-26 14:52, David Yan wrote: >>>> >>>> Also, one of the first steps in the installation wizard is to >>>>> enter >>>>> the hadoop location (the same screen as the DFS directory). >>>>> Can you please double check the hadoop location points to the >>>>> correct >>>>> hadoop binary in your system? >>>>> >>>> >>>> That, I've already done. It's definitely correct >>>> (/opt/hadoop/bin/hadoop) >>>> >>>> -dbs >>>> >>>> David >>>> >>>> On Mon, Sep 26, 2016 at 2:45 PM, David Yan <[email protected]> >>>> wrote: >>>> >>>> Do you see any exception stacktrace in the log when this error >>>> occurred: >>>> >>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, expected: >>>> file:/// >>>> >>>> David >>>> >>>> On Mon, Sep 26, 2016 at 2:39 PM, <[email protected]> wrote: >>>> From David Yan <[email protected]> >>>> Date Sat 00:12 >>>> >>>> Can you please provide any exception stacktrace in the dtgateway.log >>>> file when that happens? >>>> >>>> I reran the dtgateway installation wizard, which failed (same as >>>> before) with error msg: >>>> >>>> | DFS directory cannot be written to with error message "Mkdirs >>>> failed to create /user/dtadmin/datatorrent (exists=false, >>>> cwd=file:/opt/datatorrent/releases/3.4.0)" >>>> >>>> The corresponding exception in dtgateway.log is: >>>> >>>> | 2016-09-26 13:43:54,767 ERROR com.datatorrent.gateway.I: DFS >>>> Directory cannot be written to with exception: >>>> | java.io.IOException: Mkdirs failed to create >>>> /user/dtadmin/datatorrent (exists=false, >>>> cwd=file:/opt/datatorrent/releases/3.4.0) >>>> | at >>>> >>>> >>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileS >>> ystem.java:455) >>> >>>> | at >>>> >>>> >>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileS >>> ystem.java:440) >>> >>>> | at >>>> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) >>>> | at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) >>>> | at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789) >>>> | at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778) >>>> | at >>>> com.datatorrent.stram.client.FSAgent.createFile(FSAgent.java:77) >>>> | at com.datatorrent.gateway.I.h(gc:324) >>>> | at com.datatorrent.gateway.I.h(gc:284) >>>> | at >>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:136) >>>> | at >>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:171) >>>> | at >>>> >>>> >>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.setCo >>> nfigProperty(fc:34) >>> >>>> | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>> Method) >>>> | at >>>> >>>> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>> ssorImpl.java:62) >>> >>>> | at >>>> >>>> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> >>>> | at java.lang.reflect.Method.invoke(Method.java:498) >>>> | at >>>> >>>> >>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invo >>> ke(JavaMethodInvokerFactory.java:60) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractRes >>> ourceMethodDispatchProvider$ResponseOutInvoker._dispatch(Abs >>> tractResourceMethodDispatchProvider.java:205) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJav >>> aMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(H >>> ttpMethodRule.java:288) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>> t(RightHandPathRule.java:147) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(S >>> ubLocatorRule.java:134) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>> t(RightHandPathRule.java:147) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(S >>> ubLocatorRule.java:134) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>> t(RightHandPathRule.java:147) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accep >>> t(ResourceClassRule.java:108) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep >>> t(RightHandPathRule.java:147) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule >>> .accept(RootResourceClassesRule.java:84) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl._h >>> andleRequest(WebApplicationImpl.java:1469) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl._h >>> andleRequest(WebApplicationImpl.java:1400) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha >>> ndleRequest(WebApplicationImpl.java:1349) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha >>> ndleRequest(WebApplicationImpl.java:1339) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.spi.container.servlet.WebComponent.service(We >>> bComponent.java:416) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.spi.container.servlet.ServletContainer.servic >>> e(ServletContainer.java:537) >>> >>>> | at >>>> >>>> >>>> com.sun.jersey.spi.container.servlet.ServletContainer.servic >>> e(ServletContainer.java:699) >>> >>>> | at >>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>> | at >>>> >>>> >>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan >>> dler.java:457) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(Ses >>> sionHandler.java:229) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(Con >>> textHandler.java:1075) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand >>> ler.java:384) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.session.SessionHandler.doScope(Sess >>> ionHandler.java:193) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(Cont >>> extHandler.java:1009) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped >>> Handler.java:135) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(Ha >>> ndlerCollection.java:154) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl >>> erWrapper.java:116) >>> >>>> | at org.eclipse.jetty.server.Server.handle(Server.java:368) >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.AbstractHttpConnection.handleReques >>> t(AbstractHttpConnection.java:489) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.AbstractHttpConnection.content(Abst >>> ractHttpConnection.java:953) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandl >>> er.content(AbstractHttpConnection.java:1014) >>> >>>> | at >>>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) >>>> | at >>>> >>>> >>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHtt >>> pConnection.java:82) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(Select >>> ChannelEndPoint.java:628) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectC >>> hannelEndPoint.java:52) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued >>> ThreadPool.java:608) >>> >>>> | at >>>> >>>> >>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT >>> hreadPool.java:543) >>> >>>> | at java.lang.Thread.run(Thread.java:745) >>>> | 2016-09-26 13:43:54,772 INFO >>>> com.datatorrent.gateway.resources.ws.v2.WSResource: Caught exception >>>> in processing web service: >>>> com.datatorrent.stram.client.DTConfiguration$ConfigException: DFS >>>> directory cannot be written to with error message "Mkdirs failed to >>>> create /user/dtadmin/datatorrent (exists=false, >>>> cwd=file:/opt/datatorrent/releases/3.4.0)" >>>> >>>> Also, when I shutdown the gateway, I get a different exception in >>>> dtgateway.log: >>>> >>>> | 2016-09-26 13:40:44,366 INFO com.datatorrent.gateway.DTGateway: >>>> Shutting down >>>> | 2016-09-26 13:40:44,427 ERROR com.datatorrent.gateway.I: DFS >>>> Directory cannot be written to with exception: >>>> | java.io.IOException: Filesystem closed >>>> | at >>>> org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) >>>> | at >>>> org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2041) >>>> | at >>>> >>>> >>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(Distr >>> ibutedFileSystem.java:707) >>> >>>> | at >>>> >>>> >>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(Distr >>> ibutedFileSystem.java:703) >>> >>>> | at >>>> >>>> >>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst >>> emLinkResolver.java:81) >>> >>>> | at >>>> >>>> >>>> org.apache.hadoop.hdfs.DistributedFileSystem.delete(Distribu >>> tedFileSystem.java:714) >>> >>>> | at >>>> com.datatorrent.stram.client.FSAgent.deleteFile(FSAgent.java:94) >>>> | at com.datatorrent.gateway.I.h(gc:30) >>>> | at com.datatorrent.gateway.l.K(pc:158) >>>> | at com.datatorrent.gateway.l.h(pc:65) >>>> | at com.datatorrent.gateway.K.run(yc:156) >>>> | at java.lang.Thread.run(Thread.java:745) >>>> | 2016-09-26 13:40:44,444 INFO com.datatorrent.gateway.DTGateway: >>>> Shutdown complete >>>> >>>> Thanks much. >>>> -david >>>> >>>> On Tue, Sep 13, 2016 at 4:06 PM, <[email protected]> wrote: >>>> >>>> I have the dtgateway running (community edition) with >>>> apex/malhar 3.4 and hadoop 2.7.4 on SuSE SLES 12.1 on an ARM64 >>>> server (aarch64). >>>> >>>> In the Installation Wizard, I set the DFS location to: >>>> >>>> hdfs://namenode:9000/user/dtadmin/datatorrent >>>> >>>> and the gateway saves the hadoop configuration and restarts >>>> successfully, but I get an error: >>>> >>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, >>>> expected: file:/// >>>> >>>> the 'service dtgateway status' command says it's running. >>>> The log file has the same error as above, but nothing else >>>> useful. >>>> >>>> Is there a manual way to move the demo apps into the gateway? >>>> >>>> If I ignore this error and try to upload or import a demo app, >>>> I get the same error. >>>> >>>> What's odd is that the gateway does write into the HDFS >>>> directory: >>>> >>>> Found 6 items >>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>> >>> /user/dtadmin/datatorrent/appPackages >>> >>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>> /user/dtadmin/datatorrent/apps >>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>> /user/dtadmin/datatorrent/conf >>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>> /user/dtadmin/datatorrent/dashboards >>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>> /user/dtadmin/datatorrent/licenses >>>> drwxrwxrwt - dtadmin hadoop 0 2016-09-13 15:35 >>>> /user/dtadmin/datatorrent/systemAlerts >>>> >>>> I've tried playing with different ways to configure the DFS >>>> location, but nothing works. >>>> If I use an actual local filesystem directory, I get a >>>> different errro from the installation wizard (Non-DFS file system >>>> is >>>> used: org.apache.hadoop.fs.LocalFileSystem), which makes sense. But >>>> if I ignore this error, the app wizard successsfully writes packages >>>> into the local directory, but I can't launch them (no surprise >>>> there). >>>> >>>> Anybody have any ideas what I can do? >>>> >>>> thanks much. >>>> -david >>>> >>> >> >
