Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21426#discussion_r191010819
--- Diff: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala ---
@@ -153,4 +154,30 @@ object PythonRunner {
.map { p => formatPath(p, testWindows) }
}
+ /**
+ * Resolves the ".py" files. ".py" file should not be added as is
because PYTHONPATH does
+ * not expect a file. This method creates a temporary directory and puts
the ".py" files
+ * if exist in the given paths.
+ */
+ private def resolvePyFiles(pyFiles: Array[String]): Array[String] = {
+ val dest = Utils.createTempDir(namePrefix = "localPyFiles")
+ pyFiles.flatMap { pyFile =>
+ // In case of client with submit, the python paths should be set
before context
+ // initialization because the context initialization can be done
later.
+ // We will copy the local ".py" files because ".py" file shouldn't
be added
+ // alone but its parent directory in PYTHONPATH. See SPARK-24384.
+ if (pyFile.endsWith(".py")) {
+ val source = new File(pyFile)
+ if (source.exists() && source.canRead) {
--- End diff --
`source.isFile() && source.canRead()`
re: unreadable files, is there a check for it anywhere else? If not, that
should be added, or the app might fail with some hard to debug exception.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
