Re: Spark 1.5 on CDH 5.4.0

2015-11-01 Thread Deenar Toraskar 
HI guys

I have documented the steps involved in getting Spark 1.5.1 run on CDH
5.4.0 here, let me know if it works for you as well
https://www.linkedin.com/pulse/running-spark-151-cdh-deenar-toraskar-cfa?trk=hp-feed-article-title-publish

Looking forward to CDH 5.5 which supports Spark 1.5.x out of the box.

Regards
Deenar




*Think Reactive Ltd*
deenar.toras...@thinkreactive.co.uk
07714140812


On 23 October 2015 at 17:31, Deenar Toraskar 
wrote:

> I got this working. For others trying this It turns out in Spark 1.3/CDH5.4
>
> spark.yarn.jar=local:/opt/cloudera/parcels/
>
> I had changed this to reflect the 1.5.1 version of spark assembly jar
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> and this didn't work, I had to drop the "local:" prefix
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> Regards
> Deenar
>
> On 23 October 2015 at 17:30, Deenar Toraskar <
> deenar.toras...@thinkreactive.co.uk> wrote:
>
>> I got this working. For others trying this It turns out in Spark
>> 1.3/CDH5.4
>>
>> spark.yarn.jar=local:/opt/cloudera/parcels/
>>
>> I had changed this to reflect the 1.5.1 version of spark assembly jar
>>
>> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>>
>> and this didn't work, I had to drop the "local:" prefix
>>
>> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>>
>> Regards
>> Deenar
>>
>>
>>
>>
>> *Think Reactive Ltd*
>> deenar.toras...@thinkreactive.co.uk
>> 07714140812
>>
>>
>>
>> On 23 October 2015 at 13:34, Deenar Toraskar 
>> wrote:
>>
>>> Sandy
>>>
>>> The assembly jar does contain org.apache.spark.deploy.yarn.ExecutorLauncher.
>>> I am trying to find out how i can increase the logging level, so I know the
>>> exact classpath used by Yarn ContainerLaunch.
>>>
>>> Deenar
>>>
>>> On 23 October 2015 at 03:30, Sandy Ryza  wrote:
>>>
 Hi Deenar,

 The version of Spark you have may not be compiled with YARN support.
 If you inspect the contents of the assembly jar, does
 org.apache.spark.deploy.yarn.ExecutorLauncher exist?  If not, you'll
 need to find a version that does have the YARN classes.  You can also build
 your own using the -Pyarn flag.

 -Sandy

 On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar <
 deenar.toras...@gmail.com> wrote:

> Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 (
> http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz)
> working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works
> well including connecting to the Hive metastore. I am facing an issue
> running spark jobs in yarn-client/yarn-cluster mode. The executors fail to
> start as java cannot find ExecutorLauncher. Error: Could not find or
> load main class org.apache.spark.deploy.yarn.ExecutorLauncher client
> token: N/Adiagnostics: Application application_1443531450011_13437
> failed 2 times due to AM Container for
> appattempt_1443531450011_13437_02 exited with exitCode: 1Stack
> trace: ExitCodeException exitCode=1:at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at
> org.apache.hadoop.util.Shell.run(Shell.java:455)at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at
> java.lang.Thread.run(Thread.java:745) Any ideas as to what might be
> going wrong. Also how can I turn on more detailed logging to see what
> command line is being run by Yarn to launch containers? RegardsDeenar
>


>>>
>>
>

Re: Spark 1.5 on CDH 5.4.0

2015-10-23 Thread Deenar Toraskar 
Sandy

The assembly jar does contain org.apache.spark.deploy.yarn.ExecutorLauncher.
I am trying to find out how i can increase the logging level, so I know the
exact classpath used by Yarn ContainerLaunch.

Deenar

On 23 October 2015 at 03:30, Sandy Ryza  wrote:

> Hi Deenar,
>
> The version of Spark you have may not be compiled with YARN support.  If
> you inspect the contents of the assembly jar, does
> org.apache.spark.deploy.yarn.ExecutorLauncher exist?  If not, you'll need
> to find a version that does have the YARN classes.  You can also build your
> own using the -Pyarn flag.
>
> -Sandy
>
> On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar <
> deenar.toras...@gmail.com> wrote:
>
>> Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 (
>> http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz)
>> working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works
>> well including connecting to the Hive metastore. I am facing an issue
>> running spark jobs in yarn-client/yarn-cluster mode. The executors fail to
>> start as java cannot find ExecutorLauncher. Error: Could not find or
>> load main class org.apache.spark.deploy.yarn.ExecutorLauncher client
>> token: N/Adiagnostics: Application application_1443531450011_13437
>> failed 2 times due to AM Container for
>> appattempt_1443531450011_13437_02 exited with exitCode: 1Stack
>> trace: ExitCodeException exitCode=1:at
>> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at
>> org.apache.hadoop.util.Shell.run(Shell.java:455)at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at
>> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at
>> java.util.concurrent.FutureTask.run(FutureTask.java:262)at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at
>> java.lang.Thread.run(Thread.java:745) Any ideas as to what might be
>> going wrong. Also how can I turn on more detailed logging to see what
>> command line is being run by Yarn to launch containers? RegardsDeenar
>>
>
>

Re: Spark 1.5 on CDH 5.4.0

2015-10-23 Thread Deenar Toraskar 
I got this working. For others trying this It turns out in Spark 1.3/CDH5.4

spark.yarn.jar=local:/opt/cloudera/parcels/

I had changed this to reflect the 1.5.1 version of spark assembly jar

spark.yarn.jar=/opt/spark-1.5.1-bin/...

and this didn't work, I had to drop the "local:" prefix

spark.yarn.jar=/opt/spark-1.5.1-bin/...

Regards
Deenar

On 23 October 2015 at 17:30, Deenar Toraskar <
deenar.toras...@thinkreactive.co.uk> wrote:

> I got this working. For others trying this It turns out in Spark 1.3/CDH5.4
>
> spark.yarn.jar=local:/opt/cloudera/parcels/
>
> I had changed this to reflect the 1.5.1 version of spark assembly jar
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> and this didn't work, I had to drop the "local:" prefix
>
> spark.yarn.jar=/opt/spark-1.5.1-bin/...
>
> Regards
> Deenar
>
>
>
>
> *Think Reactive Ltd*
> deenar.toras...@thinkreactive.co.uk
> 07714140812
>
>
>
> On 23 October 2015 at 13:34, Deenar Toraskar 
> wrote:
>
>> Sandy
>>
>> The assembly jar does contain org.apache.spark.deploy.yarn.ExecutorLauncher.
>> I am trying to find out how i can increase the logging level, so I know the
>> exact classpath used by Yarn ContainerLaunch.
>>
>> Deenar
>>
>> On 23 October 2015 at 03:30, Sandy Ryza  wrote:
>>
>>> Hi Deenar,
>>>
>>> The version of Spark you have may not be compiled with YARN support.  If
>>> you inspect the contents of the assembly jar, does
>>> org.apache.spark.deploy.yarn.ExecutorLauncher exist?  If not, you'll
>>> need to find a version that does have the YARN classes.  You can also build
>>> your own using the -Pyarn flag.
>>>
>>> -Sandy
>>>
>>> On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar <
>>> deenar.toras...@gmail.com> wrote:
>>>
 Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 (
 http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz)
 working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works
 well including connecting to the Hive metastore. I am facing an issue
 running spark jobs in yarn-client/yarn-cluster mode. The executors fail to
 start as java cannot find ExecutorLauncher. Error: Could not find or
 load main class org.apache.spark.deploy.yarn.ExecutorLauncher client
 token: N/Adiagnostics: Application application_1443531450011_13437
 failed 2 times due to AM Container for
 appattempt_1443531450011_13437_02 exited with exitCode: 1Stack
 trace: ExitCodeException exitCode=1:at
 org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at
 org.apache.hadoop.util.Shell.run(Shell.java:455)at
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at
 org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at
 java.util.concurrent.FutureTask.run(FutureTask.java:262)at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at
 java.lang.Thread.run(Thread.java:745) Any ideas as to what might be
 going wrong. Also how can I turn on more detailed logging to see what
 command line is being run by Yarn to launch containers? RegardsDeenar

>>>
>>>
>>
>

Re: Spark 1.5 on CDH 5.4.0

2015-10-22 Thread Sandy Ryza 
Hi Deenar,

The version of Spark you have may not be compiled with YARN support.  If
you inspect the contents of the assembly jar, does
org.apache.spark.deploy.yarn.ExecutorLauncher exist?  If not, you'll need
to find a version that does have the YARN classes.  You can also build your
own using the -Pyarn flag.

-Sandy

On Thu, Oct 22, 2015 at 9:04 AM, Deenar Toraskar 
wrote:

> Hi I have got the prebuilt version of Spark 1.5 for Hadoop 2.6 (
> http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz)
> working with CDH 5.4.0 in local mode on a cluster with Kerberos. It works
> well including connecting to the Hive metastore. I am facing an issue
> running spark jobs in yarn-client/yarn-cluster mode. The executors fail to
> start as java cannot find ExecutorLauncher. Error: Could not find or load
> main class org.apache.spark.deploy.yarn.ExecutorLauncher client token: 
> N/Adiagnostics:
> Application application_1443531450011_13437 failed 2 times due to AM
> Container for appattempt_1443531450011_13437_02 exited with exitCode: 
> 1Stack
> trace: ExitCodeException exitCode=1:at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:538)at
> org.apache.hadoop.util.Shell.run(Shell.java:455)at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at
> java.lang.Thread.run(Thread.java:745) Any ideas as to what might be going
> wrong. Also how can I turn on more detailed logging to see what command
> line is being run by Yarn to launch containers? RegardsDeenar
>