Try adding these properties:
spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950
spark.yarn.am.extraJavaOptions -Dhdp.version=2.3.2.0-2950
There was a similar discussion with Spark 1.3.0 over here
http://stackoverflow.com/questions/29470542/spark-1-3-0-running-pi-example-on-yarn-fails
Thanks
Best Regards
On Fri, Dec 18, 2015 at 1:33 AM, syepes wrote:
> Hello,
>
> This week I have been testing 1.6 (#d509194b) in our HDP 2.3 platform and
> its been working pretty ok, at the exception of the YARN cluster deployment
> mode.
> Note that with 1.5 using the same "spark-props.conf" and "spark-env.sh"
> config files the cluster mode works as expected.
>
> Has anyone else also tried the cluster mode in 1.6?
>
>
> Problem reproduction:
>
> # spark-submit --master yarn --deploy-mode cluster --num-executors 1
> --properties-file $PWD/spark-props.conf --class
> org.apache.spark.examples.SparkPi
> /opt/spark/lib/spark-examples-1.6.0-SNAPSHOT-hadoop2.7.1.jar
>
> Error: Could not find or load main class
> org.apache.spark.deploy.yarn.ApplicationMaster
>
> spark-props.conf
> -
> spark.driver.extraJavaOptions-Dhdp.version=2.3.2.0-2950
> spark.driver.extraLibraryPath
>
> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> spark.executor.extraJavaOptions -Dhdp.version=2.3.2.0-2950
> spark.executor.extraLibraryPath
>
> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
> -
>
> I will try to do some more debugging on this issue.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-YARN-Cluster-Mode-tp25729.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
