Thanks Zhang, that solved the error. This is probably not documented anywhere so I missed it.
Thanks again, Udit On Fri, Apr 17, 2015 at 3:24 PM, Zhan Zhang <zzh...@hortonworks.com> wrote: > Besides the hdp.version in spark-defaults.conf, I think you probably > forget to put the file* java-opts* under $SPARK_HOME/conf with following > contents. > > [root@c6402 conf]# pwd > /usr/hdp/current/spark-client/conf > [root@c6402 conf]# ls > fairscheduler.xml.template * java-opts * log4j.properties.template > metrics.properties.template spark-defaults.conf spark-env.sh > hive-site.xml log4j.properties metrics.properties > slaves.template spark-defaults.conf.template > spark-env.sh.template > *[root@c6402 conf]# more java-opts* > * -Dhdp.version=2.2.0.0-2041* > [root@c6402 conf]# > > > Thanks. > > Zhan Zhang > > > On Apr 17, 2015, at 3:09 PM, Udit Mehta <ume...@groupon.com> wrote: > > Hi, > > This is the log trace: > https://gist.github.com/uditmehta27/511eac0b76e6d61f8b47 > > On the yarn RM UI, I see : > > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ExecutorLauncher > > > The command I run is: bin/spark-shell --master yarn-client > > The spark defaults I use is: > spark.yarn.jar > hdfs://namenode1-dev.snc1:8020/spark/spark-assembly-1.3.0-hadoop2.4.0.jar > spark.yarn.access.namenodes hdfs://namenode1-dev.snc1:8032 > spark.dynamicAllocation.enabled false > spark.scheduler.mode FAIR > spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041 > spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041 > > Is there anything wrong in what I am trying to do? > > thanks again! > > > On Fri, Apr 17, 2015 at 2:56 PM, Zhan Zhang <zzh...@hortonworks.com> > wrote: > >> Hi Udit, >> >> By the way, do you mind to share the whole log trace? >> >> Thanks. >> >> Zhan Zhang >> >> On Apr 17, 2015, at 2:26 PM, Udit Mehta <ume...@groupon.com> wrote: >> >> I am just trying to launch a spark shell and not do anything fancy. I >> got the binary distribution from apache and put the spark assembly on hdfs. >> I then specified the yarn.jars option in spark defaults to point to the >> assembly in hdfs. I still got the same error so though I had to build it >> for hdp. I am using hdp 2.2 with hadoop 2.6/ >> >> On Fri, Apr 17, 2015 at 2:21 PM, Udit Mehta <ume...@groupon.com> wrote: >> >>> Thanks. Would that distribution work for hdp 2.2? >>> >>> On Fri, Apr 17, 2015 at 2:19 PM, Zhan Zhang <zzh...@hortonworks.com> >>> wrote: >>> >>>> You don’t need to put any yarn assembly in hdfs. The spark assembly >>>> jar will include everything. It looks like your package does not include >>>> yarn module, although I didn’t find anything wrong in your mvn command. Can >>>> you check whether the ExecutorLauncher class is in your jar file or >>>> not? >>>> >>>> BTW: For spark-1.3, you can use the binary distribution from apache. >>>> >>>> Thanks. >>>> >>>> Zhan Zhang >>>> >>>> >>>> >>>> On Apr 17, 2015, at 2:01 PM, Udit Mehta <ume...@groupon.com> wrote: >>>> >>>> I followed the steps described above and I still get this error: >>>> >>>> >>>> Error: Could not find or load main class >>>> org.apache.spark.deploy.yarn.ExecutorLauncher >>>> >>>> >>>> I am trying to build spark 1.3 on hdp 2.2. >>>> I built spark from source using: >>>> build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive >>>> -Phive-thriftserver -DskipTests package >>>> >>>> Maybe I am not putting the correct yarn assembly on hdfs or some other >>>> issue? >>>> >>>> Thanks, >>>> Udit >>>> >>>> On Mon, Mar 30, 2015 at 10:18 AM, Zhan Zhang <zzh...@hortonworks.com> >>>> wrote: >>>> >>>>> Hi Folks, >>>>> >>>>> Just to summarize it to run SPARK on HDP distribution. >>>>> >>>>> 1. The spark version has to be 1.3.0 and above if you are using >>>>> upstream distribution. This configuration is mainly for HDP rolling >>>>> upgrade purpose, and the patch only went into spark upstream from 1.3.0. >>>>> >>>>> 2. In $SPARK_HOME/conf/sp[ark-defaults.conf, adding following >>>>> settings. >>>>> spark.driver.extraJavaOptions -Dhdp.version=xxxxx >>>>> >>>>> spark.yarn.am.extraJavaOptions -Dhdp.version=xxxxx >>>>> >>>>> 3. In $SPARK_HOME/java-opts, add following options. >>>>> -Dhdp.version=xxxxx >>>>> >>>>> Thanks. >>>>> >>>>> Zhan Zhang >>>>> >>>>> >>>>> >>>>> On Mar 30, 2015, at 6:56 AM, Doug Balog <doug.sparku...@dugos.com> >>>>> wrote: >>>>> >>>>> The “best” solution to spark-shell’s problem is creating a file >>>>> $SPARK_HOME/conf/java-opts >>>>> with “-Dhdp.version=2.2.0.0-2014” >>>>> >>>>> Cheers, >>>>> >>>>> Doug >>>>> >>>>> On Mar 28, 2015, at 1:25 PM, Michael Stone <mst...@mathom.us> wrote: >>>>> >>>>> I've also been having trouble running 1.3.0 on HDP. The >>>>> spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041 >>>>> configuration directive seems to work with pyspark, but not propagate >>>>> when using spark-shell. (That is, everything works find with pyspark, and >>>>> spark-shell fails with the "bad substitution" message.) >>>>> >>>>> Mike Stone >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>>> >>>> >>>> >>> >> >> > >
