[
https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731574#comment-15731574
]
liyunzhang_intel edited comment on SPARK-13955 at 12/8/16 8:54 AM:
-------------------------------------------------------------------
[~jerryshao] &[~tzachz]:
when i try to use "spark.yarn.jars", i found following ways work
append following in your conf/spark-defaults.conf, you need specified the
location of every jar in $SPARK_HOME/jars in spark.yarn.jars like(here i don't
paste all, too long. The seperator between jars is ",")
{code}
spark.yarn.jars=/home/zly/spark-2.0.0-bin-hadoop2-without-hive/jars/activation-1.1.1.jar,
{code}
The [document|http://spark.apache.org/docs/latest/running-on-yarn.html]
describes "spark.yarn.jars":
{code}
List of libraries containing Spark code to distribute to YARN containers. By
default, Spark on YARN will use Spark jars installed locally, but the Spark
jars can also be in a world-readable location on HDFS. This allows YARN to
cache it on nodes so that it doesn't need to be distributed each time an
application runs. To point to jars on HDFS, for example, set this configuration
to hdfs:///some/path. Globs are allowed.
{code}
but when i try to use "spark.yarn.archive" like above, it fails.
was (Author: kellyzly):
[~jerryshao] &[~tzachz]:
when i try to use "spark.yarn.jars", i found following ways work
append following in your conf/spark-defaults.conf, you need specified the
location of every jar in $SPARK_HOME/jars in spark.yarn.jars like(here i don't
paste all, too long. The seperator between jars is ",")
{code}
spark.yarn.jars=/home/zly/spark-2.0.0-bin-hadoop2-without-hive/jars/activation-1.1.1.jar,
{code}
when i try to use "spark.yarn.archive" like above, it fails.
> Spark in yarn mode fails
> ------------------------
>
> Key: SPARK-13955
> URL: https://issues.apache.org/jira/browse/SPARK-13955
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.0.0
> Reporter: Jeff Zhang
> Assignee: Marcelo Vanzin
> Fix For: 2.0.0
>
>
> I ran spark-shell in yarn client, but from the logs seems the spark assembly
> jar is not uploaded to HDFS. This may be known issue in the process of
> SPARK-11157, create this ticket to track this issue. [~vanzin]
> {noformat}
> 16/03/17 17:57:48 INFO Client: Will allocate AM container, with 896 MB memory
> including 384 MB overhead
> 16/03/17 17:57:48 INFO Client: Setting up container launch context for our AM
> 16/03/17 17:57:48 INFO Client: Setting up the launch environment for our AM
> container
> 16/03/17 17:57:48 INFO Client: Preparing resources for our AM container
> 16/03/17 17:57:48 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
> is set, falling back to uploading libraries under SPARK_HOME.
> 16/03/17 17:57:48 INFO Client: Uploading resource
> file:/Users/jzhang/github/spark/lib/apache-rat-0.10.jar ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.10.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource
> file:/Users/jzhang/github/spark/lib/apache-rat-0.11.jar ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.11.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource
> file:/private/var/folders/dp/hmchg5dd3vbcvds26q91spdw0000gp/T/spark-abed04bf-6ac2-448b-91a9-dcc1c401a18f/__spark_conf__4163776487351314654.zip
> ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/__spark_conf__4163776487351314654.zip
> 16/03/17 17:57:49 INFO SecurityManager: Changing view acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: Changing modify acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(jzhang); users
> with modify permissions: Set(jzhang)
> 16/03/17 17:57:49 INFO Client: Submitting application 6 to ResourceManager
> {noformat}
> message in AM container
> {noformat}
> Error: Could not find or load main class
> org.apache.spark.deploy.yarn.ExecutorLauncher
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
