I got this working. For others trying this It turns out in Spark 1.3/CDH5.4
spark.yarn.jar=local:/opt/cloudera/parcels/.... I had changed this to reflect the 1.5.1 version of spark assembly jar spark.yarn.jar=/opt/spark-1.5.1-bin/... and this didn't work, I had to drop the "local:" prefix spark.yarn.jar=/opt/spark-1.5.1-bin/... Regards Deenar On 23 October 2015 at 17:30, Deenar Toraskar < deenar.toras...@thinkreactive.co.uk> wrote: > I got this working. For others trying this It turns out in Spark 1.3/CDH5.4 > > spark.yarn.jar=local:/opt/cloudera/parcels/.... > > I had changed this to reflect the 1.5.1 version of spark assembly jar > > spark.yarn.jar=/opt/spark-1.5.1-bin/... > > and this didn't work, I had to drop the "local:" prefix > > spark.yarn.jar=/opt/spark-1.5.1-bin/... > > Regards > Deenar > > > > > *Think Reactive Ltd* > deenar.toras...@thinkreactive.co.uk > 07714140812 > > > > On 23 October 2015 at 13:34, Deenar Toraskar <deenar.toras...@gmail.com> > wrote: > >> Sandy >> >> The assembly jar does contain org.apache.spark.deploy.yarn.ExecutorLauncher. >> I am trying to find out how i can increase the logging level, so I know the >> exact classpath used by Yarn ContainerLaunch. >> >> Deenar >> >> On 23 October 2015 at 03:30, Sandy Ryza <sandy.r...@cloudera.com> wrote: >> >>> Hi Deenar, >>> >>> The version of Spark you have may not be compiled with YARN support. If >>> you inspect the contents of the assembly jar, does >>> org.apache.spark.deploy.yarn.ExecutorLauncher exist? If not, you'll >>> need to find a version that does have the YARN classes. You can also build >>> your own using the -Pyarn flag. >>> >>> -Sandy >>> >>> On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar < >>> deenar.toras...@gmail.com> wrote: >>> >>>> Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 ( >>>> http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz) >>>> working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works >>>> well including connecting to the Hive metastore. I am facing an issue >>>> running spark jobs in yarn-client/yarn-cluster mode. The executors fail to >>>> start as java cannot find ExecutorLauncher. Error: Could not find or >>>> load main class org.apache.spark.deploy.yarn.ExecutorLauncher client >>>> token: N/Adiagnostics: Application application_1443531450011_13437 >>>> failed 2 times due to AM Container for >>>> appattempt_1443531450011_13437_000002 exited with exitCode: 1Stack >>>> trace: ExitCodeException exitCode=1:at >>>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at >>>> org.apache.hadoop.util.Shell.run(Shell.java:455)at >>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at >>>> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at >>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at >>>> java.lang.Thread.run(Thread.java:745) Any ideas as to what might be >>>> going wrong. Also how can I turn on more detailed logging to see what >>>> command line is being run by Yarn to launch containers? RegardsDeenar >>>> >>> >>> >> >
