Interesting. Looking at SparkConf.scala :
val configs = Seq(
DeprecatedConfig("spark.files.userClassPathFirst",
"spark.executor.userClassPathFirst",
"1.3"),
DeprecatedConfig("spark.yarn.user.classpath.first", null, "1.3",
"Use spark.{driver,executor}.userClassPathFirst instead."))
It seems spark.files.userClassPathFirst and spark.yarn.user.classpath.first
are deprecated.
On Wed, Feb 25, 2015 at 12:39 AM, Sean Owen <so...@cloudera.com> wrote:
> No, we should not add fastutil back. It's up to the app to bring
> dependencies it needs, and that's how I understand this issue. The
> question is really, how to get the classloader visibility right. It
> depends on where you need these classes. Have you looked into
> spark.files.userClassPathFirst and spark.yarn.user.classpath.first ?
>
> On Wed, Feb 25, 2015 at 5:34 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > bq. depend on missing fastutil classes like Long2LongOpenHashMap
> >
> > Looks like Long2LongOpenHashMap should be added to the shaded jar.
> >
> > Cheers
> >
> > On Tue, Feb 24, 2015 at 7:36 PM, Jim Kleckner <j...@cloudphysics.com>
> wrote:
> >>
> >> Spark includes the clearspring analytics package but intentionally
> >> excludes
> >> the dependencies of the fastutil package (see below).
> >>
> >> Spark includes parquet-column which includes fastutil and relocates it
> >> under
> >> parquet/
> >> but creates a shaded jar file which is incomplete because it shades out
> >> some
> >> of
> >> the fastutil classes, notably Long2LongOpenHashMap, which is present in
> >> the
> >> fastutil jar file that parquet-column is referencing.
> >>
> >> We are using more of the clearspring classes (e.g. QDigest) and those do
> >> depend on
> >> missing fastutil classes like Long2LongOpenHashMap.
> >>
> >> Even though I add them to our assembly jar file, the class loader finds
> >> the
> >> spark assembly
> >> and we get runtime class loader errors when we try to use it.
> >>
> >> It is possible to put our jar file first, as described here:
> >> https://issues.apache.org/jira/browse/SPARK-939
> >>
> >>
> http://spark.apache.org/docs/1.2.0/configuration.html#runtime-environment
> >>
> >> which I tried with args to spark-submit:
> >> --conf spark.driver.userClassPathFirst=true --conf
> >> spark.executor.userClassPathFirst=true
> >> but we still get the class not found error.
> >>
> >> We have tried copying the source code for clearspring into our package
> and
> >> renaming the
> >> package and that makes it appear to work... Is this risky? It
> certainly
> >> is
> >> ugly.
> >>
> >> Can anyone recommend a way to deal with this "dependency **ll" ?
> >>
> >>
> >> === The spark/pom.xml file contains the following lines:
> >>
> >> <dependency>
> >> <groupId>com.clearspring.analytics</groupId>
> >> <artifactId>stream</artifactId>
> >> <version>2.7.0</version>
> >> <exclusions>
> >>
> >> <exclusion>
> >> <groupId>it.unimi.dsi</groupId>
> >> <artifactId>fastutil</artifactId>
> >> </exclusion>
> >> </exclusions>
> >> </dependency>
> >>
> >> === The parquet-column/pom.xml file contains:
> >> <artifactId>maven-shade-plugin</artifactId>
> >> <executions>
> >> <execution>
> >> <phase>package</phase>
> >> <goals>
> >> <goal>shade</goal>
> >> </goals>
> >> <configuration>
> >> <minimizeJar>true</minimizeJar>
> >> <artifactSet>
> >> <includes>
> >> <include>it.unimi.dsi:fastutil</include>
> >> </includes>
> >> </artifactSet>
> >> <relocations>
> >> <relocation>
> >> <pattern>it.unimi.dsi</pattern>
> >> <shadedPattern>parquet.it.unimi.dsi</shadedPattern>
> >> </relocation>
> >> </relocations>
> >> </configuration>
> >> </execution>
> >> </executions>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-excludes-fastutil-dependencies-we-need-tp21794.html
> >> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> >> For additional commands, e-mail: user-h...@spark.apache.org
> >>
> >
>
