spark has a setting to put user jars in front of classpath, which should do the trick. however i had no luck with this. see here:
https://issues.apache.org/jira/browse/SPARK-1863 On Mon, Jul 7, 2014 at 1:31 PM, Robert James <srobertja...@gmail.com> wrote: > spark-submit includes a spark-assembly uber jar, which has older > versions of many common libraries. These conflict with some of the > dependencies we need. I have been racking my brain trying to find a > solution (including experimenting with ProGuard), but haven't been > able to: when we use spark-submit, we get NoMethodErrors, even though > the code compiles fine, because the runtime classes are different than > the compile time classes! > > Can someone recommend a solution? We are using scala, sbt, and > sbt-assembly, but are happy using another tool (please provide > instructions how to). >
