Spark 1.5.1 Build Failure
Hi, I am trying to build spark 1.5.1 for hadoop 2.5 but I get the following error. *build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.0-cdh5.3.2 -DskipTests clean package* [INFO] Spark Project Parent POM ... SUCCESS [ 9.812 s] [INFO] Spark Project Launcher . SUCCESS [ 27.701 s] [INFO] Spark Project Networking ... SUCCESS [ 16.721 s] [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 8.617 s] [INFO] Spark Project Unsafe ... SUCCESS [ 27.124 s] [INFO] Spark Project Core . FAILURE [09:08 min] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-core_2.10: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. CompileFailed -> [Help 1] -- Regards, Raghuveer Chanda
Re: Spark 1.5.1 Build Failure
t) @ spark-launcher_2.10 ---* *[INFO] * *[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-launcher_2.10 ---* *[INFO] Using 'UTF-8' encoding to copy filtered resources.* *[INFO] skip non existing resourceDirectory /home/rchanda/Downloads/spark-1.5.1/launcher/src/main/resources* *[INFO] Copying 3 resources* *[INFO] * *[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-launcher_2.10 ---* *[INFO] Using zinc server for incremental compilation* *[info] Compiling 8 Java sources to /home/rchanda/Downloads/spark-1.5.1/launcher/target/scala-2.10/classes...* *[info] Error occurred during initialization of VM* *[info] java.lang.Error: Properties init: Could not determine current working directory.* *[info] at java.lang.System.initProperties(Native Method)* *[info] at java.lang.System.initializeSystemClass(System.java:1119)* *[info] * *[error] Compile failed at Oct 29, 2015 2:11:26 PM [0.110s]* *[INFO] * *[INFO] Reactor Summary:* *[INFO] * *[INFO] Spark Project Parent POM ... SUCCESS [ 9.628 s]* *[INFO] Spark Project Launcher . FAILURE [ 7.573 s]* *[INFO] Spark Project Networking ... SKIPPED* *[INFO] Spark Project Shuffle Streaming Service SKIPPED* *[INFO] Spark Project Unsafe ... SKIPPED* *[INFO] Spark Project Core . SKIPPED* *[INFO] Spark Project Bagel SKIPPED* *[INFO] Spark Project GraphX ... SKIPPED* *[INFO] Spark Project Streaming SKIPPED* *[INFO] Spark Project Catalyst . SKIPPED* *[INFO] Spark Project SQL .. SKIPPED* *[INFO] Spark Project ML Library ... SKIPPED* *[INFO] Spark Project Tools SKIPPED* *[INFO] Spark Project Hive . SKIPPED* *[INFO] Spark Project REPL . SKIPPED* *[INFO] Spark Project Assembly . SKIPPED* *[INFO] Spark Project External Twitter . SKIPPED* *[INFO] Spark Project External Flume Sink .. SKIPPED* *[INFO] Spark Project External Flume ... SKIPPED* *[INFO] Spark Project External Flume Assembly .. SKIPPED* *[INFO] Spark Project External MQTT SKIPPED* *[INFO] Spark Project External MQTT Assembly ... SKIPPED* *[INFO] Spark Project External ZeroMQ .. SKIPPED* *[INFO] Spark Project External Kafka ... SKIPPED* *[INFO] Spark Project Examples . SKIPPED* *[INFO] Spark Project External Kafka Assembly .. SKIPPED* *[INFO] * *[INFO] BUILD FAILURE* *[INFO] * *[INFO] Total time: 19.258 s* *[INFO] Finished at: 2015-10-29T14:11:26-07:00* *[INFO] Final Memory: 39M/94M* *[INFO] * *[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-launcher_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> [Help 1]* *[ERROR] * *[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.* *[ERROR] Re-run Maven using the -X switch to enable full debug logging.* *[ERROR] * *[ERROR] For more information about the errors and possible solutions, please read the following articles:* *[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException <http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException>* *[ERROR] * *[ERROR] After correcting the problems, you can resume the build with the command* *[ERROR] mvn -rf :spark-launcher_2.10* On Fri, Oct 30, 2015 at 12:55 PM, Jia Zhan <zhanjia...@gmail.com> wrote: > Hi, > > Have tried tried building it successfully without hadoop? > > $build/mnv -DskiptTests clean package > > Can you check it build/mvn was started successfully, or it's using your > own mvn? Let us know your jdk version as well. > > On Thu, Oct 29, 2015 at 11:34 PM, Raghuveer Chanda < > raghuveer.cha...@gmail.com> wrote: > >> Hi, >> >> I am trying to build spark 1.5.1 for hadoop 2.5 but I get the following >> error. >> >> >> *build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.0-cdh5.3.2 >> -DskipTests clean package* >> >> >> [INFO] Spark Project Parent POM ... SUCCESS [ >> 9.812 s] >> [INFO] Spark Project Launcher ..
Re: Spark 1.5.1 Build Failure
There seems to be a error at the zinc server, how can I shut down the zinc server completely *build/zinc-0.3.5.3/bin/zinc -shutdown *will shutdown but it again restarts with the mvn/build command ? *Error in Debug mode :* *[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-core_2.10: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. CompileFailed -> [Help 1]* *org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-core_2.10: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed.* * at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)* * at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)* * at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)* * at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)* * at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)* * at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)* * at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)* * at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)* * at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)* * at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)* * at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)* * at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)* * at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)* * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* * at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)* * at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)* * at java.lang.reflect.Method.invoke(Method.java:606)* * at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)* * at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)* * at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)* * at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)* *Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed.* * at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)* * at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)* * ... 20 more* *Caused by: Compile failed via zinc server* * at sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java:136)* * at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86)* * at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303)* * at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119)* * at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99)* * at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)* * at scala_maven.ScalaTestCompileMojo.execute(ScalaTestCompileMojo.java:48)* * at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)* * ... 21 more* *[ERROR] * *[ERROR] * *[ERROR] For more information about the errors and possible solutions, please read the following articles:* *[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException <http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException>* *[ERROR] * *[ERROR] After correcting the problems, you can resume the build with the command* *[ERROR] mvn -rf :spark-core_2.10* *Regards* *Raghuveer* On Fri, Oct 30, 2015 at 1:18 PM, Raghuveer Chanda < raghuveer.cha...@gmail.com> wrote: > Thanks for the reply. > > I am using the mvn and scala from the source code build/mvn only and I get > the same error without hadoop also after clean package. > > > *Java Version:* > > *rchanda@ubuntu:~/Downloads/spark-1.5.1$ java -version* > *java version "1.7.0_85"* > *OpenJDK Runtime Environment (IcedTea 2.6.1) (7u85-2.6.1-5ubuntu0.14.04.1)* > *OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)* > > *Complete Error:* > > *rchanda@ubuntu:~/Downloads/spark-1.5.1$ build/mvn -DskiptTests clean > package* > *Using `mvn` from path: > /home/rchanda/Downloads/spark-1.5.1/build/apache-maven-3.3.3/bin/mvn* > *[INFO] Scanning for projects...* > *[INFO] > * >
Spark on Yarn
Hi all, I am trying to run spark on yarn in quickstart cloudera vm.It already has spark 1.3 and Hadoop 2.6.0-cdh5.4.0 installed.(I am not using spark-submit since I want to run a different version of spark). I am able to run spark 1.3 on yarn but get the below error for spark 1.4. The log shows its running on spark 1.4 but still gives a error on a method which is present in 1.4 and not 1.3. Even the fat jar contains the class files of 1.4. As far as running in yarn the installed spark version shouldnt matter, but still its running on the other version. *Hadoop Version:* Hadoop 2.6.0-cdh5.4.0 Subversion http://github.com/cloudera/hadoop -r c788a14a5de9ecd968d1e2666e8765c5f018c271 Compiled by jenkins on 2015-04-21T19:18Z Compiled with protoc 2.5.0 >From source with checksum cd78f139c66c13ab5cee96e15a629025 This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.4.0.jar *Error:* LogType:stderr Log Upload Time:Tue Oct 20 21:58:56 -0700 2015 LogLength:2334 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/filecache/10/simple-yarn-app-1.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/10/20 21:58:50 INFO spark.SparkContext: *Running Spark version 1.4.0* 15/10/20 21:58:53 INFO spark.SecurityManager: Changing view acls to: yarn 15/10/20 21:58:53 INFO spark.SecurityManager: Changing modify acls to: yarn 15/10/20 21:58:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn); users with modify permissions: Set(yarn) *Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.network.util.JavaUtils.timeStringAsSec(Ljava/lang/String;)J* at org.apache.spark.util.Utils$.timeStringAsSeconds(Utils.scala:1027) at org.apache.spark.SparkConf.getTimeAsSeconds(SparkConf.scala:194) at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:68) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982) at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:245) at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:52) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:247) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:188) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267) at org.apache.spark.SparkContext.(SparkContext.scala:424) at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:61) at com.hortonworks.simpleyarnapp.HelloWorld.main(HelloWorld.java:50) 15/10/20 21:58:53 INFO util.Utils: Shutdown hook called Please help :) -- Regards and Thanks, Raghuveer Chanda
Re: Spark on Yarn
Hi, So does this mean I can't run spark 1.4 fat jar on yarn without installing spark 1.4. I am including spark 1.4 in my pom.xml so doesn't this mean its compiling in 1.4. On Wed, Oct 21, 2015 at 4:38 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi > > The compiled version (master side) and client version diverge on spark > network JavaUtils. You should use the same/aligned version. > > Regards > JB > > > > Sent from my Samsung device > > > ---- Original message > From: Raghuveer Chanda <raghuveer.cha...@gmail.com> > Date: 21/10/2015 12:33 (GMT+01:00) > To: user@spark.apache.org > Subject: Spark on Yarn > > Hi all, > > I am trying to run spark on yarn in quickstart cloudera vm.It already has > spark 1.3 and Hadoop 2.6.0-cdh5.4.0 installed.(I am not using > spark-submit since I want to run a different version of spark). > > I am able to run spark 1.3 on yarn but get the below error for spark 1.4. > > The log shows its running on spark 1.4 but still gives a error on a method > which is present in 1.4 and not 1.3. Even the fat jar contains the class > files of 1.4. > > As far as running in yarn the installed spark version shouldnt matter, but > still its running on the other version. > > > *Hadoop Version:* > Hadoop 2.6.0-cdh5.4.0 > Subversion http://github.com/cloudera/hadoop -r > c788a14a5de9ecd968d1e2666e8765c5f018c271 > Compiled by jenkins on 2015-04-21T19:18Z > Compiled with protoc 2.5.0 > From source with checksum cd78f139c66c13ab5cee96e15a629025 > This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.4.0.jar > > *Error:* > LogType:stderr > Log Upload Time:Tue Oct 20 21:58:56 -0700 2015 > LogLength:2334 > Log Contents: > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/filecache/10/simple-yarn-app-1.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 15/10/20 21:58:50 INFO spark.SparkContext: *Running Spark version 1.4.0* > 15/10/20 21:58:53 INFO spark.SecurityManager: Changing view acls to: yarn > 15/10/20 21:58:53 INFO spark.SecurityManager: Changing modify acls to: yarn > 15/10/20 21:58:53 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(yarn); users with modify permissions: Set(yarn) > *Exception in thread "main" java.lang.NoSuchMethodError: > org.apache.spark.network.util.JavaUtils.timeStringAsSec(Ljava/lang/String;)J* > at org.apache.spark.util.Utils$.timeStringAsSeconds(Utils.scala:1027) > at org.apache.spark.SparkConf.getTimeAsSeconds(SparkConf.scala:194) > at > org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:68) > at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) > at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) > at > org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) > at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982) > at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) > at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:245) > at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:52) > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:247) > at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:188) > at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267) > at org.apache.spark.SparkContext.(SparkContext.scala:424) > at > org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:61) > at com.hortonworks.simpleyarnapp.HelloWorld.main(HelloWorld.java:50) > 15/10/20 21:58:53 INFO util.Utils: Shutdown hook called > > Please help :) > > -- > Regards and Thanks, > Raghuveer Chanda > -- Regards, Raghuveer Chanda Computer Science and Engineering IIT Kharagpur +91-9475470374
Re: Spark on Yarn
Please find the attached pom.xml. I am using maven to build the fat jar and trying to run it in yarn using *hadoop jar simple-yarn-app-master/target/simple-yarn-app-1.1.0-shaded.jar com.hortonworks.simpleyarnapp.Client hdfs://quickstart.cloudera:8020/simple-yarn-app-1.1.0-shaded.jar* Basically I am following the below code and changed the Application Master to run a Spark application class. https://github.com/hortonworks/simple-yarn-app It works for 1.3 the installed version in cdh but throws error for 1.4. When I am bundling the spark within the jar it shouldn't be the case right ? On Wed, Oct 21, 2015 at 5:11 PM, Adrian Tanase <atan...@adobe.com> wrote: > The question is the spark dependency is marked as provided or is included > in the fat jar. > > For example, we are compiling the spark distro separately for java 8 + > scala 2.11 + hadoop 2.6 (with maven) and marking it as provided in sbt. > > -adrian > > From: Raghuveer Chanda > Date: Wednesday, October 21, 2015 at 2:14 PM > To: Jean-Baptiste Onofré > Cc: "user@spark.apache.org" > Subject: Re: Spark on Yarn > > Hi, > > So does this mean I can't run spark 1.4 fat jar on yarn without installing > spark 1.4. > > I am including spark 1.4 in my pom.xml so doesn't this mean its compiling > in 1.4. > > > On Wed, Oct 21, 2015 at 4:38 PM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi >> >> The compiled version (master side) and client version diverge on spark >> network JavaUtils. You should use the same/aligned version. >> >> Regards >> JB >> >> >> >> Sent from my Samsung device >> >> >> Original message >> From: Raghuveer Chanda <raghuveer.cha...@gmail.com> >> Date: 21/10/2015 12:33 (GMT+01:00) >> To: user@spark.apache.org >> Subject: Spark on Yarn >> >> Hi all, >> >> I am trying to run spark on yarn in quickstart cloudera vm.It already >> has spark 1.3 and Hadoop 2.6.0-cdh5.4.0 installed.(I am not using >> spark-submit since I want to run a different version of spark). >> >> I am able to run spark 1.3 on yarn but get the below error for spark 1.4. >> >> The log shows its running on spark 1.4 but still gives a error on a >> method which is present in 1.4 and not 1.3. Even the fat jar contains the >> class files of 1.4. >> >> As far as running in yarn the installed spark version shouldnt matter, >> but still its running on the other version. >> >> >> *Hadoop Version:* >> Hadoop 2.6.0-cdh5.4.0 >> Subversion http://github.com/cloudera/hadoop -r >> c788a14a5de9ecd968d1e2666e8765c5f018c271 >> Compiled by jenkins on 2015-04-21T19:18Z >> Compiled with protoc 2.5.0 >> From source with checksum cd78f139c66c13ab5cee96e15a629025 >> This command was run using >> /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.4.0.jar >> >> *Error:* >> LogType:stderr >> Log Upload Time:Tue Oct 20 21:58:56 -0700 2015 >> LogLength:2334 >> Log Contents: >> SLF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in >> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/filecache/10/simple-yarn-app-1.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> explanation. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> 15/10/20 21:58:50 INFO spark.SparkContext: *Running Spark version 1.4.0* >> 15/10/20 21:58:53 INFO spark.SecurityManager: Changing view acls to: yarn >> 15/10/20 21:58:53 INFO spark.SecurityManager: Changing modify acls to: >> yarn >> 15/10/20 21:58:53 INFO spark.SecurityManager: SecurityManager: >> authentication disabled; ui acls disabled; users with view permissions: >> Set(yarn); users with modify permissions: Set(yarn) >> *Exception in thread "main" java.lang.NoSuchMethodError: >> org.apache.spark.network.util.JavaUtils.timeStringAsSec(Ljava/lang/String;)J* >> at org.apache.spark.util.Utils$.timeStringAsSeconds(Utils.scala:1027) >> at org.apache.spark.SparkConf.getTimeAsSeconds(SparkConf.scala:194) >> at >> org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:68) >> at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) >> at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) >> at >> org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$
Mosek Solver with Apache Spark
Hi, Has anyone tried Mosek http://www.mosek.com/ Solver in Spark? I getting weird serialization errors. I came to know that Mosek uses shared libraries which may not be serialized. Is this the reason that they are not serialized or Is it working for anyone. -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
Re: YARN ResourceManager and Hadoop NameNode Web UI not visible in port 8088, port 50070
The problem is solved, the web interfaces are not opening in local network connecting to server with proxy its opening only in the servers without proxy .. On Thu, Sep 25, 2014 at 1:12 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi Raghuveer, This might be a better question for the cdh-user list or the Hadoop user list. The Hadoop web interfaces for both the NameNode and ResourceManager are enabled by default. Is it possible you have a firewall blocking those ports? -Sandy On Wed, Sep 24, 2014 at 9:00 PM, Raghuveer Chanda raghuveer.cha...@gmail.com wrote: Hi, Im running a spark job in YARN cluster ..but im not able to see the Web Interface of the YARN ResourceManager and Hadoop NameNode Web UI in port 8088, port 50070 and spark stages. Only the spark UI in port 18080 is visible. I got the URL's from cloudera but may be due to some default option for security the Web Interface is disabled. How can i enable the web interface i.e is there any option in cloudera or is the server firewall is blocking it .. Please help .. -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
Re: Spark with YARN
Thanks for the reply, I have doubt as to which path to set for YARN_CONF_DIR My /etc/hadoop folder has the following sub folders conf conf.cloudera.hdfs conf.cloudera.mapreduce conf.cloudera.yarn and both conf and conf.cloudera.yarn folders have yarn-site.xml. As of now I set the variable as export YARN_CONF_DIR=/etc/hadoop/conf But no change in the error .. only driver is added no executors for it, do i need to make any change in conf files or what is the correct path for variable. On Wed, Sep 24, 2014 at 11:18 PM, Greg Hill greg.h...@rackspace.com wrote: Do you have YARN_CONF_DIR set in your environment to point Spark to where your yarn configs are? Greg From: Raghuveer Chanda raghuveer.cha...@gmail.com Date: Wednesday, September 24, 2014 12:25 PM To: u...@spark.incubator.apache.org u...@spark.incubator.apache.org Subject: Spark with YARN Hi, I'm new to spark and facing problem with running a job in cluster using YARN. Initially i ran jobs using spark master as --master spark://dml2:7077 and it is running fine on 3 workers. But now im shifting to YARN, so installed YARN in cloud era on 3 node cluster and changed the master to yarn-cluster but it is not working I attached the screenshots of UI which are not progressing and just hanging on. Output on terminal : This error is repeating ./spark-submit --class class-name --master yarn-cluster --num-executors 3 --executor-cores 3 jar-with-dependencies.jar Do i need to configure YARN or why it is not getting all the workers .. please help ... 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ appUser: chanda 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
Re: Spark with YARN
The screenshot executors.8080.png is of the executors tab itself and only driver is added without workers even if I kept the master as yarn-cluster. On Wed, Sep 24, 2014 at 11:18 PM, Matt Narrell matt.narr...@gmail.com wrote: This just shows the driver. Click the Executors tab in the Spark UI mn On Sep 24, 2014, at 11:25 AM, Raghuveer Chanda raghuveer.cha...@gmail.com wrote: Hi, I'm new to spark and facing problem with running a job in cluster using YARN. Initially i ran jobs using spark master as --master spark://dml2:7077 and it is running fine on 3 workers. But now im shifting to YARN, so installed YARN in cloud era on 3 node cluster and changed the master to yarn-cluster but it is not working I attached the screenshots of UI which are not progressing and just hanging on. Output on terminal : This error is repeating ./spark-submit --class class-name --master yarn-cluster --num-executors 3 --executor-cores 3 jar-with-dependencies.jar Do i need to configure YARN or why it is not getting all the workers .. please help ... 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ appUser: chanda 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur executors.8088.PNGspark.8088.PNGyarn.8088.PNG - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
Re: Spark with YARN
Thanks for the reply .. This is the error in the logs obtained from UI at http://dml3:8042/node/containerlogs/container_1411578463780_0001_02_01/chanda So now how to set the Log Server url .. Failed while trying to construct the redirect url to the log server. Log Server url may not be configuredContainer does not exist. On Wed, Sep 24, 2014 at 11:37 PM, Marcelo Vanzin van...@cloudera.com wrote: You'll need to look at the driver output to have a better idea of what's going on. You can use yarn logs --applicationId blah after your app is finished (e.g. by killing it) to look at it. My guess is that your cluster doesn't have enough resources available to service the container request you're making. That will show up in the driver as periodic messages that no containers have been allocated yet. On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda raghuveer.cha...@gmail.com wrote: Hi, I'm new to spark and facing problem with running a job in cluster using YARN. Initially i ran jobs using spark master as --master spark://dml2:7077 and it is running fine on 3 workers. But now im shifting to YARN, so installed YARN in cloud era on 3 node cluster and changed the master to yarn-cluster but it is not working I attached the screenshots of UI which are not progressing and just hanging on. Output on terminal : This error is repeating ./spark-submit --class class-name --master yarn-cluster --num-executors 3 --executor-cores 3 jar-with-dependencies.jar Do i need to configure YARN or why it is not getting all the workers .. please help ... 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ appUser: chanda 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Marcelo -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
Re: Spark with YARN
Yeah I got the logs and its reporting about the memory. 14/09/25 00:08:26 WARN YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory Now I shifted to big cluster with more memory but here im not able to view the UI while the job is running .. I need to check the status and spark UI of job. cse-hadoop-xx:18080 doesnt have running job it only had the jobs with master as spark://cse-hadoop-xx:7077 and not yarn-cluster . Im not ableto view the following links .. http://cse-hadoop-xx:50070/ http://cse-hadoop-xx:8088/ Is this due to some security option im not able to view the UI how can change it in cloudera.? On Thu, Sep 25, 2014 at 12:04 AM, Marcelo Vanzin van...@cloudera.com wrote: You need to use the command line yarn application that I mentioned (yarn logs). You can't look at the logs through the UI after the app stops. On Wed, Sep 24, 2014 at 11:16 AM, Raghuveer Chanda raghuveer.cha...@gmail.com wrote: Thanks for the reply .. This is the error in the logs obtained from UI at http://dml3:8042/node/containerlogs/container_1411578463780_0001_02_01/chanda So now how to set the Log Server url .. Failed while trying to construct the redirect url to the log server. Log Server url may not be configured Container does not exist. On Wed, Sep 24, 2014 at 11:37 PM, Marcelo Vanzin van...@cloudera.com wrote: You'll need to look at the driver output to have a better idea of what's going on. You can use yarn logs --applicationId blah after your app is finished (e.g. by killing it) to look at it. My guess is that your cluster doesn't have enough resources available to service the container request you're making. That will show up in the driver as periodic messages that no containers have been allocated yet. On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda raghuveer.cha...@gmail.com wrote: Hi, I'm new to spark and facing problem with running a job in cluster using YARN. Initially i ran jobs using spark master as --master spark://dml2:7077 and it is running fine on 3 workers. But now im shifting to YARN, so installed YARN in cloud era on 3 node cluster and changed the master to yarn-cluster but it is not working I attached the screenshots of UI which are not progressing and just hanging on. Output on terminal : This error is repeating ./spark-submit --class class-name --master yarn-cluster --num-executors 3 --executor-cores 3 jar-with-dependencies.jar Do i need to configure YARN or why it is not getting all the workers .. please help ... 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ appUser: chanda 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM: application identifier: application_1411578463780_0001 appId: 1 clientToAMToken: null appDiagnostics: appMasterHost: dml3 appQueue: root.chanda appMasterRpcPort: 0 appStartTime: 1411578513545 yarnAppState: RUNNING distributedFinalState: UNDEFINED appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/ -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Marcelo -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur -- Marcelo -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
YARN ResourceManager and Hadoop NameNode Web UI not visible in port 8088, port 50070
Hi, Im running a spark job in YARN cluster ..but im not able to see the Web Interface of the YARN ResourceManager and Hadoop NameNode Web UI in port 8088, port 50070 and spark stages. Only the spark UI in port 18080 is visible. I got the URL's from cloudera but may be due to some default option for security the Web Interface is disabled. How can i enable the web interface i.e is there any option in cloudera or is the server firewall is blocking it .. Please help .. -- Regards, Raghuveer Chanda 4th year Undergraduate Student Computer Science and Engineering IIT Kharagpur
