[
https://issues.apache.org/jira/browse/SPARK-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saisai Shao updated SPARK-20365:
--------------------------------
Summary: Not so accurate classpath format for AM and Containers (was:
Inaccurate classpath format for AM and Containers)
> Not so accurate classpath format for AM and Containers
> ------------------------------------------------------
>
> Key: SPARK-20365
> URL: https://issues.apache.org/jira/browse/SPARK-20365
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.2.0
> Reporter: Saisai Shao
> Priority: Minor
>
> In Spark on YARN, when configuring "spark.yarn.jars" with local jars (jars
> started with "local" scheme), we will get inaccurate classpath for AM and
> containers. This is because we don't remove "local" scheme when concatenating
> classpath. It is OK to run because classpath is separated with ":" and java
> treat "local" as a separate jar. But we could improve it to remove the scheme.
> {code}
> java.class.path =
> /tmp/hadoop-sshao/nm-local-dir/usercache/sshao/appcache/application_1492057593145_0009/container_1492057593145_0009_01_000003:/tmp/hadoop-sshao/nm-local-dir/usercache/sshao/appcache/application_1492057593145_0009/container_1492057593145_0009_01_000003/__spark_conf__:/tmp/hadoop-sshao/nm-local-dir/usercache/sshao/appcache/application_1492057593145_0009/container_1492057593145_0009_01_000003/__spark_libs__/*:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/activation-1.1.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/antlr-2.7.7.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/antlr-runtime-3.4.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/antlr4-runtime-4.5.3.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/aopalliance-1.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/apache-log4j-extras-1.2.17.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/apacheds-i18n-2.0.0-M15.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/apacheds-kerberos-codec-2.0.0-M15.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/api-asn1-api-1.0.0-M20.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/api-util-1.0.0-M20.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/arpack_combined_all-0.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-1.7.7.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-ipc-1.7.7-tests.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-ipc-1.7.7.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/avro-mapred-1.7.7-hadoop2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/base64-2.3.8.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/bcprov-jdk15on-1.51.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/bonecp-0.8.0.RELEASE.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/breeze-macros_2.11-0.12.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/breeze_2.11-0.12.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/calcite-avatica-1.2.0-incubating.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/calcite-core-1.2.0-incubating.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/calcite-linq4j-1.2.0-incubating.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/cglib-2.2.1-v20090111.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/chill-java-0.8.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/chill_2.11-0.8.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-beanutils-1.7.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-beanutils-core-1.8.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-cli-1.2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-codec-1.10.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-collections-3.2.2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-compiler-3.0.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-compress-1.4.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-configuration-1.6.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-crypto-1.0.0.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-dbcp-1.4.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-digester-1.8.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-httpclient-3.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-io-2.4.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-lang-2.6.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-lang3-3.5.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-logging-1.2.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-math3-3.4.1.jar:local:///Users/sshao/projects/apache-spark/assembly/target/scala-2.11/jars/commons-net-3.1.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
