Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21426#discussion_r191039095 --- Diff: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala --- @@ -153,4 +154,30 @@ object PythonRunner { .map { p => formatPath(p, testWindows) } } + /** + * Resolves the ".py" files. ".py" file should not be added as is because PYTHONPATH does + * not expect a file. This method creates a temporary directory and puts the ".py" files + * if exist in the given paths. + */ + private def resolvePyFiles(pyFiles: Array[String]): Array[String] = { + val dest = Utils.createTempDir(namePrefix = "localPyFiles") + pyFiles.flatMap { pyFile => + // In case of client with submit, the python paths should be set before context + // initialization because the context initialization can be done later. + // We will copy the local ".py" files because ".py" file shouldn't be added + // alone but its parent directory in PYTHONPATH. See SPARK-24384. + if (pyFile.endsWith(".py")) { + val source = new File(pyFile) + if (source.exists() && source.canRead) { --- End diff -- @vanzin, do you mean that this should be checked ahead (for example in SparkSubmit) before we are in this logic? Just for clarification, this is just a sanity check. The previous behaviour was that the path is added but it's ignored and the current behaviour is that it doesn't add the path.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org