[ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736236#comment-15736236 ]
liyunzhang_intel commented on SPARK-13955: ------------------------------------------ [~jerryshao]: After testing, i can use "spark.yarn.archive" in following steps: 1.cd $SPARK_HOME/jars 2. zip spark-archive.zip ./* #enter the directory, then zip 3. when you test the spark-archive.zip by "unzip -t spark-archive.zip, you will see {code} unzip -t spark-archive.zip Archive: spark-archive.zip testing: activation-1.1.1.jar OK testing: antlr4-runtime-4.5.3.jar OK testing: aopalliance-1.0.jar OK testing: aopalliance-repackaged-2.4.0-b34.jar OK testing: apacheds-i18n-2.0.0-M15.jar OK testing: apacheds-kerberos-codec-2.0.0-M15.jar OK testing: api-asn1-api-1.0.0-M20.jar OK testing: api-util-1.0.0-M20.jar OK testing: arpack_combined_all-0.1.jar OK {code} 4. copy spark-archive.zip to hdfs like "hadoop fs -copyFromLocal spark-archive.zip hdfs://bdpe42:8020/" 5. append "spark.yarn.archive=hdfs://bdpe42:8020/spark-archive.zip" to conf/spark-defaults.conf > Spark in yarn mode fails > ------------------------ > > Key: SPARK-13955 > URL: https://issues.apache.org/jira/browse/SPARK-13955 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 2.0.0 > Reporter: Jeff Zhang > Assignee: Marcelo Vanzin > Fix For: 2.0.0 > > > I ran spark-shell in yarn client, but from the logs seems the spark assembly > jar is not uploaded to HDFS. This may be known issue in the process of > SPARK-11157, create this ticket to track this issue. [~vanzin] > {noformat} > 16/03/17 17:57:48 INFO Client: Will allocate AM container, with 896 MB memory > including 384 MB overhead > 16/03/17 17:57:48 INFO Client: Setting up container launch context for our AM > 16/03/17 17:57:48 INFO Client: Setting up the launch environment for our AM > container > 16/03/17 17:57:48 INFO Client: Preparing resources for our AM container > 16/03/17 17:57:48 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive > is set, falling back to uploading libraries under SPARK_HOME. > 16/03/17 17:57:48 INFO Client: Uploading resource > file:/Users/jzhang/github/spark/lib/apache-rat-0.10.jar -> > hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.10.jar > 16/03/17 17:57:49 INFO Client: Uploading resource > file:/Users/jzhang/github/spark/lib/apache-rat-0.11.jar -> > hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.11.jar > 16/03/17 17:57:49 INFO Client: Uploading resource > file:/private/var/folders/dp/hmchg5dd3vbcvds26q91spdw0000gp/T/spark-abed04bf-6ac2-448b-91a9-dcc1c401a18f/__spark_conf__4163776487351314654.zip > -> > hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/__spark_conf__4163776487351314654.zip > 16/03/17 17:57:49 INFO SecurityManager: Changing view acls to: jzhang > 16/03/17 17:57:49 INFO SecurityManager: Changing modify acls to: jzhang > 16/03/17 17:57:49 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(jzhang); users > with modify permissions: Set(jzhang) > 16/03/17 17:57:49 INFO Client: Submitting application 6 to ResourceManager > {noformat} > message in AM container > {noformat} > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ExecutorLauncher > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org