GitHub user pwendell opened a pull request:
https://github.com/apache/spark/pull/119
SPARK-1230: [WIP] Enable SparkContext.addJars() to load classes not in
CLASSPATH
Reopens and up-merges @velvia's patch previously against the old repo:
https://github.com/apache/incubator-spark/pull/299
In addition to merging I made some changes, one of which I'm not sure is
correct. The change was to avoid setting the classloader explicitly in akka.
I'm planning to add unit tests to this tomorrow to play around with the change.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwendell/spark addJars
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/119.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #119
----
commit 8854dabc0e51b809b939edd2f88000c13024ccce
Author: Evan Chan <[email protected]>
Date: 2013-12-13T17:22:56Z
Print out the class that was not found
commit ae1c199a88cb0ba5c12e49cfe200b0cee97ab1e1
Author: Evan Chan <[email protected]>
Date: 2013-12-13T23:40:14Z
Fix bug where classes in jars added via addJar() are not available to the
driver
Adds a driver-specific class loader that knows about the jars added via
addJar(), and
initialize it first thing in SparkEnv, so that all other
ThreadPool/ActorSystem
initialization will take advantage of the classloader, and user-added jars
are
made available to all Spark subsystems.
commit 5701bf4722540defc21fa3f9cb0b33883bd9497b
Author: Evan Chan <[email protected]>
Date: 2013-12-27T00:36:55Z
Merge branch 'master' into debug-class-not-found
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala
commit d3df2415eaed4da50fc1ffeef42e0140f6b49d88
Author: Evan Chan <[email protected]>
Date: 2013-12-29T08:15:17Z
Move classLoader initialization to SparkContext
SparkEnv is used by Executor as well; we want this change to affect driver
only.
commit 5c24b0bc541a61cb3551fee9386602a2472b571a
Author: Evan Chan <[email protected]>
Date: 2013-12-29T08:25:53Z
Pass the current class loader when creating Akka threadpool
commit cbabe809baf1f02c7c9097734b84a3c94ce364db
Author: Evan Chan <[email protected]>
Date: 2014-01-12T15:59:24Z
CR from pwendell
commit b0cc61d571259298025c373e636ccc563e9fee37
Author: Evan Chan <[email protected]>
Date: 2014-01-17T01:29:51Z
Merge branch 'master' into debug-class-not-found
Conflicts:
core/src/main/scala/org/apache/spark/SparkContext.scala
core/src/main/scala/org/apache/spark/SparkEnv.scala
core/src/main/scala/org/apache/spark/util/AkkaUtils.scala
commit 97a19b5f38a84376176aef8b6e35ef578b29d3c7
Author: Evan Chan <[email protected]>
Date: 2014-01-17T06:19:27Z
CR: Rename ExecutorURLClassLoader -> SparkURLClassLoader
commit 9757c6ffbc9e0e1354c19ad5fafcd7083634cb32
Author: Evan Chan <[email protected]>
Date: 2014-01-17T06:34:34Z
Use SparkURLClassLoader for driver only if spark.driver.add-dynamic-jars is
set
commit 3e52b0c4984cecdaea67412a7259c3a8ebbbfdf6
Author: Evan Chan <[email protected]>
Date: 2014-01-17T07:54:58Z
Document new config option and effect on addJar
commit 178083d09e8be45432e50e315734b40ffc8f4efa
Author: Patrick Wendell <[email protected]>
Date: 2014-03-11T04:34:46Z
Merge remote-tracking branch 'apache/master' into addJars
Conflicts:
core/src/main/scala/org/apache/spark/SparkContext.scala
core/src/main/scala/org/apache/spark/SparkEnv.scala
core/src/main/scala/org/apache/spark/util/AkkaUtils.scala
commit b132d7bd61f8ba4bd4994b8fee94099125d8ade7
Author: Patrick Wendell <[email protected]>
Date: 2014-03-11T04:56:49Z
Minor changes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
