the whole spark.files.userClassPathFirs never really worked for me in standalone mode, since jars were added dynamically which means they had different classloaders leading to a real classloader hell if you tried to add a newer version of jar that spark already used. see: https://issues.apache.org/jira/browse/SPARK-1863
do i understand it correctly that on yarn the the customer jars are truly placed before the yarn and spark jars on classpath? meaning at container construction time, on the same classloader? that would be great news for me. it would open up the possibility of using newer versions of many libraries. On Wed, Feb 4, 2015 at 1:12 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Hi Corey, > > When you run on Yarn, Yarn's libraries are placed in the classpath, > and they have precedence over your app's. So, with Spark 1.2, you'll > get Guava 11 in your classpath (with Spark 1.1 and earlier you'd get > Guava 14 from Spark, so still a problem for you). > > Right now, the option Markus mentioned > (spark.yarn.user.classpath.first) can be a workaround for you, since > it will place your app's jars before Yarn's on the classpath. > > > On Tue, Feb 3, 2015 at 8:20 PM, Corey Nolet <cjno...@gmail.com> wrote: > > I'm having a really bad dependency conflict right now with Guava versions > > between my Spark application in Yarn and (I believe) Hadoop's version. > > > > The problem is, my driver has the version of Guava which my application > is > > expecting (15.0) while it appears the Spark executors that are working > on my > > RDDs have a much older version (assuming it's the old version on the > Hadoop > > classpath). > > > > Is there a property like "mapreduce.job.user.classpath.first' that I can > set > > to make sure my own classpath is extablished first on the executors? > > > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
