Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8959#issuecomment-144818093
Hi @tgravescs ,
This fixes the problem but I think it's actually just masking a subtle bug
elsewhere. In `getUserClasspath`, there's this code:
val mainUri = mainJar.orElse(Some(APP_JAR)).map(new URI(_))
That is not actually doing the right thing in certain cases. When invoked
from a `Client` instance, the `mainJar` argument comes from
`ClientArguments.userJar`, so it's never going to be `None` (and thus always
return the name of the original jar instead of `APP_JAR`).
The "cleanest" thing would be to have just a single version of
`getUserClasspath` that gets things from `SparkConf`, but that runs into the
problem that the conf has not yet been updated when `populateClasspath` is
called.
I think changing that `map` call to something like the following would fix
the source of the problem:
.map { path =>
val uri = new URI(path)
if (uri.getScheme == LOCAL_SCHEME) new URI(uri.getPath()) else uri
}
(Not tested.) Does that make sense? Could you try that out?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
