shanyu zhao created SPARK-30845:
-----------------------------------
Summary: spark-submit pyspark app on yarn uploads local pyspark
archives
Key: SPARK-30845
URL: https://issues.apache.org/jira/browse/SPARK-30845
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 2.4.5, 2.4.4, 2.4.3, 2.4.2, 2.4.1, 2.4.0
Reporter: shanyu zhao
Use spark-submit to submit a pyspark app on Yarn, and set this in spark-env.sh:
{code:bash}
export
PYSPARK_ARCHIVES_PATH=local:/opt/spark/python/lib/pyspark.zip,local:/opt/spark/python/lib/py4j-0.10.7-src.zip
{code}
You can see that these local archives are still uploaded to Yarn distributed
cache.
yarn.Client: Uploading resource file:/opt/spark/python/lib/pyspark.zip ->
hdfs://myhdfs/user/test1/.sparkStaging/application_1581024490249_0001/pyspark.zip
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
