HI guys I have documented the steps involved in getting Spark 1.5.1 run on CDH 5.4.0 here, let me know if it works for you as well https://www.linkedin.com/pulse/running-spark-151-cdh-deenar-toraskar-cfa?trk=hp-feed-article-title-publish
Looking forward to CDH 5.5 which supports Spark 1.5.x out of the box. Regards Deenar *Think Reactive Ltd* deenar.toras...@thinkreactive.co.uk 07714140812 On 23 October 2015 at 17:31, Deenar Toraskar <deenar.toras...@gmail.com> wrote: > I got this working. For others trying this It turns out in Spark 1.3/CDH5.4 > > spark.yarn.jar=local:/opt/cloudera/parcels/.... > > I had changed this to reflect the 1.5.1 version of spark assembly jar > > spark.yarn.jar=/opt/spark-1.5.1-bin/... > > and this didn't work, I had to drop the "local:" prefix > > spark.yarn.jar=/opt/spark-1.5.1-bin/... > > Regards > Deenar > > On 23 October 2015 at 17:30, Deenar Toraskar < > deenar.toras...@thinkreactive.co.uk> wrote: > >> I got this working. For others trying this It turns out in Spark >> 1.3/CDH5.4 >> >> spark.yarn.jar=local:/opt/cloudera/parcels/.... >> >> I had changed this to reflect the 1.5.1 version of spark assembly jar >> >> spark.yarn.jar=/opt/spark-1.5.1-bin/... >> >> and this didn't work, I had to drop the "local:" prefix >> >> spark.yarn.jar=/opt/spark-1.5.1-bin/... >> >> Regards >> Deenar >> >> >> >> >> *Think Reactive Ltd* >> deenar.toras...@thinkreactive.co.uk >> 07714140812 >> >> >> >> On 23 October 2015 at 13:34, Deenar Toraskar <deenar.toras...@gmail.com> >> wrote: >> >>> Sandy >>> >>> The assembly jar does contain org.apache.spark.deploy.yarn.ExecutorLauncher. >>> I am trying to find out how i can increase the logging level, so I know the >>> exact classpath used by Yarn ContainerLaunch. >>> >>> Deenar >>> >>> On 23 October 2015 at 03:30, Sandy Ryza <sandy.r...@cloudera.com> wrote: >>> >>>> Hi Deenar, >>>> >>>> The version of Spark you have may not be compiled with YARN support. >>>> If you inspect the contents of the assembly jar, does >>>> org.apache.spark.deploy.yarn.ExecutorLauncher exist? If not, you'll >>>> need to find a version that does have the YARN classes. You can also build >>>> your own using the -Pyarn flag. >>>> >>>> -Sandy >>>> >>>> On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar < >>>> deenar.toras...@gmail.com> wrote: >>>> >>>>> Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 ( >>>>> http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz) >>>>> working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works >>>>> well including connecting to the Hive metastore. I am facing an issue >>>>> running spark jobs in yarn-client/yarn-cluster mode. The executors fail to >>>>> start as java cannot find ExecutorLauncher. Error: Could not find or >>>>> load main class org.apache.spark.deploy.yarn.ExecutorLauncher client >>>>> token: N/Adiagnostics: Application application_1443531450011_13437 >>>>> failed 2 times due to AM Container for >>>>> appattempt_1443531450011_13437_000002 exited with exitCode: 1Stack >>>>> trace: ExitCodeException exitCode=1:at >>>>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at >>>>> org.apache.hadoop.util.Shell.run(Shell.java:455)at >>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at >>>>> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at >>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at >>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at >>>>> java.lang.Thread.run(Thread.java:745) Any ideas as to what might be >>>>> going wrong. Also how can I turn on more detailed logging to see what >>>>> command line is being run by Yarn to launch containers? RegardsDeenar >>>>> >>>> >>>> >>> >> >
