Re: Spark 1.5.2 - Programmatically launching spark on yarn-client mode

2016-01-30 Thread Nirav Patel 
Thanks Ted. In my application jar there was no spark 1.3.1 artifacts.
Anyhow I got it working via Oozie spark action.

On Thu, Jan 28, 2016 at 7:42 PM, Ted Yu  wrote:

> Looks like '--properties-file' is no longer supported.
>
> Was it possible that Spark 1.3.1 artifact / dependency leaked into your
> app ?
>
> Cheers
>
> On Thu, Jan 28, 2016 at 7:36 PM, Nirav Patel 
> wrote:
>
>> Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client
>> mode programmatically via creating a sparkConf and sparkContext object
>> manually. It was inspired from spark self-contained application example
>> here:
>>
>>
>> https://spark.apache.org/docs/1.5.2/quick-start.html#self-contained-applications\
>>
>> Only additional configuration we would provide would be all related to
>> yarn like executor instance, cores, memory, extraJavaOptions etc.
>>
>> However after upgrading to spark 1.5.2 above application breaks on a line
>> `val sparkContext = new SparkContext(sparkConf)`
>>
>> 16/01/28 17:38:35 ERROR util.Utils: Uncaught exception in thread main
>>
>> java.lang.NullPointerException
>>
>> at
>> org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
>>
>> at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1228)
>>
>> at org.apache.spark.SparkEnv.stop(SparkEnv.scala:100)
>>
>> at
>> org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1749)
>>
>> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)
>>
>> at org.apache.spark.SparkContext.stop(SparkContext.scala:1748)
>>
>> at org.apache.spark.SparkContext.(SparkContext.scala:593)
>>
>>
>> *In yarn container logs I see following:*
>>
>> 16/01/28 17:38:29 INFO yarn.ApplicationMaster: Registered signal handlers 
>> for [TERM, HUP, INT]*Unknown/unsupported param List*(--properties-file, 
>> /tmp/hadoop-xactly/nm-local-dir/usercache/xactly/appcache/application_1453752281504_3427/container_1453752281504_3427_01_02/__spark_conf__/__spark_conf__.properties)
>>
>> Usage: org.apache.spark.deploy.yarn.ApplicationMaster [options]
>> Options:
>>   --jar JAR_PATH   Path to your application's JAR file
>>   --class CLASS_NAME   Name of your application's main class
>>   --primary-py-fileA main Python file
>>   --py-files PY_FILES  Comma-separated list of .zip, .egg, or .py files to
>>place on the PYTHONPATH for Python apps.
>>   --args ARGS  Arguments to be passed to your application's main 
>> class.
>>Multiple invocations are possible, each will be 
>> passed in order.
>>   --num-executors NUMNumber of executors to start (Default: 2)
>>   --executor-cores NUM   Number of cores for the executors (Default: 1)
>>   --executor-memory MEM  Memory per executor (e.g. 1000M, 2G) (Default: 1G)
>>
>>
>>
>> So is this approach still supposed to work? Or do I must use
>> SparkLauncher class with spark 1.5.2?
>>
>> Thanks
>>
>> Nirav
>>
>>
>>
>>
>> [image: What's New with Xactly] 
>>
>>   [image: LinkedIn]
>>   [image: Twitter]
>>   [image: Facebook]
>>   [image: YouTube]
>> 
>
>
>

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube]

Re: Spark 1.5.2 - Programmatically launching spark on yarn-client mode

2016-01-28 Thread Ted Yu 
Looks like '--properties-file' is no longer supported.

Was it possible that Spark 1.3.1 artifact / dependency leaked into your app
?

Cheers

On Thu, Jan 28, 2016 at 7:36 PM, Nirav Patel  wrote:

> Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client
> mode programmatically via creating a sparkConf and sparkContext object
> manually. It was inspired from spark self-contained application example
> here:
>
>
> https://spark.apache.org/docs/1.5.2/quick-start.html#self-contained-applications\
>
> Only additional configuration we would provide would be all related to
> yarn like executor instance, cores, memory, extraJavaOptions etc.
>
> However after upgrading to spark 1.5.2 above application breaks on a line
> `val sparkContext = new SparkContext(sparkConf)`
>
> 16/01/28 17:38:35 ERROR util.Utils: Uncaught exception in thread main
>
> java.lang.NullPointerException
>
> at
> org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
>
> at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1228)
>
> at org.apache.spark.SparkEnv.stop(SparkEnv.scala:100)
>
> at
> org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1749)
>
> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)
>
> at org.apache.spark.SparkContext.stop(SparkContext.scala:1748)
>
> at org.apache.spark.SparkContext.(SparkContext.scala:593)
>
>
> *In yarn container logs I see following:*
>
> 16/01/28 17:38:29 INFO yarn.ApplicationMaster: Registered signal handlers for 
> [TERM, HUP, INT]*Unknown/unsupported param List*(--properties-file, 
> /tmp/hadoop-xactly/nm-local-dir/usercache/xactly/appcache/application_1453752281504_3427/container_1453752281504_3427_01_02/__spark_conf__/__spark_conf__.properties)
>
> Usage: org.apache.spark.deploy.yarn.ApplicationMaster [options]
> Options:
>   --jar JAR_PATH   Path to your application's JAR file
>   --class CLASS_NAME   Name of your application's main class
>   --primary-py-fileA main Python file
>   --py-files PY_FILES  Comma-separated list of .zip, .egg, or .py files to
>place on the PYTHONPATH for Python apps.
>   --args ARGS  Arguments to be passed to your application's main 
> class.
>Multiple invocations are possible, each will be passed 
> in order.
>   --num-executors NUMNumber of executors to start (Default: 2)
>   --executor-cores NUM   Number of cores for the executors (Default: 1)
>   --executor-memory MEM  Memory per executor (e.g. 1000M, 2G) (Default: 1G)
>
>
>
> So is this approach still supposed to work? Or do I must use SparkLauncher
> class with spark 1.5.2?
>
> Thanks
>
> Nirav
>
>
>
>
> [image: What's New with Xactly] 
>
>   [image: LinkedIn]
>   [image: Twitter]
>   [image: Facebook]
>   [image: YouTube]
>

Spark 1.5.2 - Programmatically launching spark on yarn-client mode

2016-01-28 Thread Nirav Patel 
Hi, we were using spark 1.3.1 and launching our spark jobs on yarn-client
mode programmatically via creating a sparkConf and sparkContext object
manually. It was inspired from spark self-contained application example
here:

https://spark.apache.org/docs/1.5.2/quick-start.html#self-contained-applications\

Only additional configuration we would provide would be all related to yarn
like executor instance, cores, memory, extraJavaOptions etc.

However after upgrading to spark 1.5.2 above application breaks on a line
`val sparkContext = new SparkContext(sparkConf)`

16/01/28 17:38:35 ERROR util.Utils: Uncaught exception in thread main

java.lang.NullPointerException

at
org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)

at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1228)

at org.apache.spark.SparkEnv.stop(SparkEnv.scala:100)

at
org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1749)

at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)

at org.apache.spark.SparkContext.stop(SparkContext.scala:1748)

at org.apache.spark.SparkContext.(SparkContext.scala:593)


*In yarn container logs I see following:*

16/01/28 17:38:29 INFO yarn.ApplicationMaster: Registered signal
handlers for [TERM, HUP, INT]*Unknown/unsupported param
List*(--properties-file,
/tmp/hadoop-xactly/nm-local-dir/usercache/xactly/appcache/application_1453752281504_3427/container_1453752281504_3427_01_02/__spark_conf__/__spark_conf__.properties)

Usage: org.apache.spark.deploy.yarn.ApplicationMaster [options]
Options:
  --jar JAR_PATH   Path to your application's JAR file
  --class CLASS_NAME   Name of your application's main class
  --primary-py-fileA main Python file
  --py-files PY_FILES  Comma-separated list of .zip, .egg, or .py files to
   place on the PYTHONPATH for Python apps.
  --args ARGS  Arguments to be passed to your application's main class.
   Multiple invocations are possible, each will be
passed in order.
  --num-executors NUMNumber of executors to start (Default: 2)
  --executor-cores NUM   Number of cores for the executors (Default: 1)
  --executor-memory MEM  Memory per executor (e.g. 1000M, 2G) (Default: 1G)



So is this approach still supposed to work? Or do I must use SparkLauncher
class with spark 1.5.2?

Thanks

Nirav

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube]