[ https://issues.apache.org/jira/browse/SPARK-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell resolved SPARK-939. ----------------------------------- Resolution: Fixed > Allow user jars to take precedence over Spark jars, if desired > -------------------------------------------------------------- > > Key: SPARK-939 > URL: https://issues.apache.org/jira/browse/SPARK-939 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Reporter: Patrick Wendell > Assignee: holdenk > Priority: Blocker > Labels: starter > Fix For: 1.0.0 > > > Sometimes a user may want to include their own version of a jar that spark > itself uses. For example, if their code requires a newer version of that jar > than Spark offers. It would be good to have an option to give the users > dependencies precedence over Spark. This options should be disabled by > default, since it could lead to some odd behavior (e.g. parts of Spark not > working). But I think we should have it. > From an implementation perspective, this would require modifying the way we > do class loading inside of an Executor. The default behavior of the > URLClassLoader is to delegate to it's parent first and, if that fails, to > find a class locally. We want to have the opposite behavior. This is > sometimes referred to as "parent-last" (as opposed to "parent-first") class > loading precedence. There is an example of how to do this here: > http://stackoverflow.com/questions/5445511/how-do-i-create-a-parent-last-child-first-classloader-in-java-or-how-to-overr > We should write a similar class which can encapsulate a URL classloader and > change the delegation order. Or if possible, maybe we could find a more > elegant way to do this. See relevant discussion on the user list here: > https://groups.google.com/forum/#!topic/spark-users/b278DW3e38g > Also see the corresponding option in Hadoop: > https://issues.apache.org/jira/browse/MAPREDUCE-4521 > Some other relevant Hadoop JIRA's: > https://issues.apache.org/jira/browse/MAPREDUCE-1700 > https://issues.apache.org/jira/browse/MAPREDUCE-1938 -- This message was sent by Atlassian JIRA (v6.2#6252)