[
https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated SPARK-13955:
-------------------------------
Description:
I ran spark-shell in yarn client, but from the logs seems the spark assembly
jar is not uploaded to HDFS. This may be known issue in the process of
SPARK-11157, create this ticket to track this issue. [~vanzin]
{noformat}
16/03/17 17:57:48 INFO Client: Will allocate AM container, with 896 MB memory
including 384 MB overhead
16/03/17 17:57:48 INFO Client: Setting up container launch context for our AM
16/03/17 17:57:48 INFO Client: Setting up the launch environment for our AM
container
16/03/17 17:57:48 INFO Client: Preparing resources for our AM container
16/03/17 17:57:48 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
is set, falling back to uploading libraries under SPARK_HOME.
16/03/17 17:57:48 INFO Client: Uploading resource
file:/Users/jzhang/github/spark/lib/apache-rat-0.10.jar ->
hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.10.jar
16/03/17 17:57:49 INFO Client: Uploading resource
file:/Users/jzhang/github/spark/lib/apache-rat-0.11.jar ->
hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.11.jar
16/03/17 17:57:49 INFO Client: Uploading resource
file:/private/var/folders/dp/hmchg5dd3vbcvds26q91spdw0000gp/T/spark-abed04bf-6ac2-448b-91a9-dcc1c401a18f/__spark_conf__4163776487351314654.zip
->
hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/__spark_conf__4163776487351314654.zip
16/03/17 17:57:49 INFO SecurityManager: Changing view acls to: jzhang
16/03/17 17:57:49 INFO SecurityManager: Changing modify acls to: jzhang
16/03/17 17:57:49 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(jzhang); users
with modify permissions: Set(jzhang)
16/03/17 17:57:49 INFO Client: Submitting application 6 to ResourceManager
{noformat}
message in AM container
{noformat}
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
{noformat}
was:
I ran spark-shell in yarn client, but from the logs seems the spark assembly
jar is not uploaded to HDFS. This may be known issue in the process of
SPARK-11157, create this ticket to track this issue. [~vanzin]
{noformat}
16/03/17 11:58:59 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
is set, falling back to uploading libraries under SPARK_HOME.
16/03/17 11:58:59 INFO Client: Uploading resource
file:/Users/jzhang/github/spark/lib/apache-rat-0.10.jar ->
hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0004/apache-rat-0.10.jar
16/03/17 11:58:59 INFO Client: Uploading resource
file:/Users/jzhang/github/spark/lib/apache-rat-0.11.jar ->
hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0004/apache-rat-0.11.jar
16/03/17 11:59:00 INFO Client: Uploading resource
file:/private/var/folders/dp/hmchg5dd3vbcvds26q91spdw0000gp/T/spark-36cacbad-ca5b-482b-8ca8-607499acaaba/__spark_conf__4427292248554277597.zip
->
hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0004/__spark_conf__4427292248554277597.zip
{noformat}
message in AM container
{noformat}
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
{noformat}
> Spark in yarn mode fails
> ------------------------
>
> Key: SPARK-13955
> URL: https://issues.apache.org/jira/browse/SPARK-13955
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Jeff Zhang
>
> I ran spark-shell in yarn client, but from the logs seems the spark assembly
> jar is not uploaded to HDFS. This may be known issue in the process of
> SPARK-11157, create this ticket to track this issue. [~vanzin]
> {noformat}
> 16/03/17 17:57:48 INFO Client: Will allocate AM container, with 896 MB memory
> including 384 MB overhead
> 16/03/17 17:57:48 INFO Client: Setting up container launch context for our AM
> 16/03/17 17:57:48 INFO Client: Setting up the launch environment for our AM
> container
> 16/03/17 17:57:48 INFO Client: Preparing resources for our AM container
> 16/03/17 17:57:48 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
> is set, falling back to uploading libraries under SPARK_HOME.
> 16/03/17 17:57:48 INFO Client: Uploading resource
> file:/Users/jzhang/github/spark/lib/apache-rat-0.10.jar ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.10.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource
> file:/Users/jzhang/github/spark/lib/apache-rat-0.11.jar ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.11.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource
> file:/private/var/folders/dp/hmchg5dd3vbcvds26q91spdw0000gp/T/spark-abed04bf-6ac2-448b-91a9-dcc1c401a18f/__spark_conf__4163776487351314654.zip
> ->
> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/__spark_conf__4163776487351314654.zip
> 16/03/17 17:57:49 INFO SecurityManager: Changing view acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: Changing modify acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(jzhang); users
> with modify permissions: Set(jzhang)
> 16/03/17 17:57:49 INFO Client: Submitting application 6 to ResourceManager
> {noformat}
> message in AM container
> {noformat}
> Error: Could not find or load main class
> org.apache.spark.deploy.yarn.ExecutorLauncher
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
