Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21426 @vanzin, for https://github.com/apache/spark/pull/21426#discussion_r191010819, mind if we proceed in a separate ticket? From my look, it needs some changes to verify this to address this comment. I think we can't simply raise an exception since we can't recognise if that file is downloaded or not in `deploy.PythonRunner`'s perspective. The most appropriate place seems to be in `SparkSubmit` and `DependencyUtils.downloadFile`. seems we should inject some codes in `DependencyUtils.downloadFile` since that's where we know the original path and where we download the file into local when needed, and I would like to avoid add such changes here. It probably needs another review iteration and the current change doesn't actually target or change the previous behaviour, really.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org