If you can't avoid it, you need to make use of the spark.driver.userClassPathFirst and/or spark.executor.userClassPathFirst properties.
On Thu, Jul 16, 2020 at 2:03 PM Russell Spitzer <russell.spit...@gmail.com> wrote: > I believe the main issue here is that spark.jars is a bit "too late" to > actually prepend things to the class path. For most use cases this value is > not read until after the JVM has already started and the system classloader > has already loaded. > > The jar argument gets added via the dynamic class loader so it necessarily > has to come after wards :/ Driver extra classpath and it's friends, modify > the actual launch command of the driver (or executors) so they can prepend > whenever they want. > > In general you do not want to have conflicting jars at all if possible > and I would recommend looking into shading if it's really important for > your application to use a specific incompatible version of a library. Jar > (and extraClasspath) are really just > for adding additional jars and I personally would try not to rely on > classpath ordering to get the right libraries recognized. > > On Thu, Jul 16, 2020 at 1:55 PM Nupur Shukla <nupur14shu...@gmail.com> > wrote: > >> Hello, >> >> How can we use *spark.jars* to to specify conflicting jars (that is, >> jars that are already present in the spark's default classpath)? Jars >> specified in this conf gets "appended" to the classpath, and thus gets >> looked at after the default classpath. Is it not intended to be used to >> specify conflicting jars? >> Meanwhile when *spark.driver.extraClassPath* conf is specified, this >> path is "prepended" to the classpath and thus takes precedence over the >> default classpath. >> >> How can I use both to specify different jars and paths but achieve a >> precedence of spark.jars path > spark.driver.extraClassPath > spark default >> classpath (left to right precedence order)? >> >> Experiment conducted: >> >> I am using sample-project.jar which has one class in it SampleProject. >> This has a method which prints the version number of the jar. For this >> experiment I am using 3 versions of this sample-project.jar >> Sample-project-1.0.0.jar is present in the spark default classpath in my >> test cluster >> Sample-project-2.0.0.jar is present in folder /home/<user>/ClassPathConf >> on driver >> Sample-project-3.0.0.jar is present in folder /home/<user>/JarsConf on >> driver >> >> (Empty cell in img below means that conf was not specified) >> >> [image: image.png] >> >> >> Thank you, >> Nupur >> >> >>
