[
https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320552#comment-14320552
]
Marcelo Vanzin commented on SPARK-5770:
---------------------------------------
It might be possible to fix the behavior, although even then the results might
be sketchy.
Basically, when overwriting jars, you'd have to replace the executor's class
loader. That means you need to keep track of the jars added to the class
loader, and when adding a new jar, you place it in front of the others and use
Thread.currentThread().setContextClassLoader() to replace the class loader.
But that's after like 5 seconds of thinking, so there may be a lot of corner
cases in doing that. I think the best approach would be to say that overwriting
jars is not allowed, even if that doesn't cover all cases. You could still add
a different jar that tries to override already loaded classes, and that will
have the same confusing effect of the old classes being still used.
> Use addJar() to upload a new jar file to executor, it can't be added to
> classloader
> -----------------------------------------------------------------------------------
>
> Key: SPARK-5770
> URL: https://issues.apache.org/jira/browse/SPARK-5770
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: meiyoula
>
> First use addJar() to upload a jar to the executor, then change the jar
> content and upload it again. We can see the jar file in the local has be
> updated, but the classloader still load the old one. The executor log has no
> error or exception to point it.
> I use spark-shell to test it. And set "spark.files.overwrite" is true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
