[
https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734082#comment-15734082
]
liyunzhang_intel commented on SPARK-13955:
------------------------------------------
test pi in yarn-client mode by using "spark.yarn.archive"
[~jerryshao]: following is the detail steps when i use "spark.yarn.archive"
1. zip all jars: zip spark-archive.zip $SPARK_HOME/jars/*
2. upload the zip to hdfs: hadoop fs -copyFromLocal spark-archive.zip
hdfs://bdpe42:8020/
3. modify the spark-defaults.conf
spark.yarn.archive=hdfs://bdpe42:8020/spark-archive.zip
4. run pi in yarn client mode
{code}
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
yarn-client --num-executors 3 --driver-memory 1g --executor-memory 1g
--executor-cores 1 $spark_example_jar > sparkPi.log 2>&1
{code}
The exception in container log is
{code}
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
{code}
The spark version is 2.0.2.
> Spark in yarn mode fails
> ------------------------
>
> Key: SPARK-13955
> URL: https://issues.apache.org/jira/browse/SPARK-13955
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.0.0
> Reporter: Jeff Zhang
> Assignee: Marcelo Vanzin
> Fix For: 2.0.0
>
>
> I ran spark-shell in yarn client, but from the logs seems the spark assembly
> jar is not uploaded to HDFS. This may be known issue in the process of
> SPARK-11157, create this ticket to track this issue. [~vanzin]
> {noformat}
> 16/03/17 17:57:48 INFO Client: Will allocate AM container, with 896 MB memory
> including 384 MB overhead
> 16/03/17 17:57:48 INFO Client: Setting up container launch context for our AM
> 16/03/17 17:57:48 INFO Client: Setting up the launch environment for our AM
> container
> 16/03/17 17:57:48 INFO Client: Preparing resources for our AM container
> 16/03/17 17:57:48 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
> is set, falling back to uploading libraries under SPARK_HOME.
> 16/03/17 17:57:48 INFO Client: Uploading resource
> file:/Users/jzhang/github/spark/lib/apache-rat-0.10.jar ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.10.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource
> file:/Users/jzhang/github/spark/lib/apache-rat-0.11.jar ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.11.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource
> file:/private/var/folders/dp/hmchg5dd3vbcvds26q91spdw0000gp/T/spark-abed04bf-6ac2-448b-91a9-dcc1c401a18f/__spark_conf__4163776487351314654.zip
> ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/__spark_conf__4163776487351314654.zip
> 16/03/17 17:57:49 INFO SecurityManager: Changing view acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: Changing modify acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(jzhang); users
> with modify permissions: Set(jzhang)
> 16/03/17 17:57:49 INFO Client: Submitting application 6 to ResourceManager
> {noformat}
> message in AM container
> {noformat}
> Error: Could not find or load main class
> org.apache.spark.deploy.yarn.ExecutorLauncher
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
