Marcelo Vanzin created SPARK-26934:
--------------------------------------
Summary: python dependencies with "local:" URIs are not visible to
executors
Key: SPARK-26934
URL: https://issues.apache.org/jira/browse/SPARK-26934
Project: Spark
Issue Type: Bug
Components: PySpark, Spark Core
Affects Versions: 2.4.0
Reporter: Marcelo Vanzin
Noticed this when running tests for SPARK-24736. If you provide a file with a
"local:" URI to {{--py-files}}, spark-submit makes it visible to the driver,
but not executors. That makes code that runs in executors and references those
files fail.
The underlying issue is that when providing dependency information to
executors, when running tasks, {{SparkContext}} erases any references to files
with a "local:" scheme, so executors don't know they exist.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
