I ran into a similar problem while using QDigest.
Shading the clearspring, fastutil classes solves the problem.
Snippet from pom.xml :
package
shade
, since the class loaders should all end up finding the
one copy of the class that you're adding with your app.
--
Marcelo
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Re-Spark-excludes-fastutil-dependencies-we-need-tp21849.html
Sent from the Apache
On Wed, Feb 25, 2015 at 8:42 PM, Jim Kleckner j...@cloudphysics.com wrote:
So, should the userClassPathFirst flag work and there is a bug?
Sorry for jumping in the middle of conversation (and probably missing
some of it), but note that this option applies only to executors. If
you're trying to
Interesting. Looking at SparkConf.scala :
val configs = Seq(
DeprecatedConfig(spark.files.userClassPathFirst,
spark.executor.userClassPathFirst,
1.3),
DeprecatedConfig(spark.yarn.user.classpath.first, null, 1.3,
Use spark.{driver,executor}.userClassPathFirst
Inline
On Wed, Feb 25, 2015 at 1:53 PM, Ted Yu yuzhih...@gmail.com wrote:
Interesting. Looking at SparkConf.scala :
val configs = Seq(
DeprecatedConfig(spark.files.userClassPathFirst,
spark.executor.userClassPathFirst,
1.3),
Maybe drop the exclusion for parquet-provided profile ?
Cheers
On Wed, Feb 25, 2015 at 8:42 PM, Jim Kleckner j...@cloudphysics.com wrote:
Inline
On Wed, Feb 25, 2015 at 1:53 PM, Ted Yu yuzhih...@gmail.com wrote:
Interesting. Looking at SparkConf.scala :
val configs = Seq(
No, we should not add fastutil back. It's up to the app to bring
dependencies it needs, and that's how I understand this issue. The
question is really, how to get the classloader visibility right. It
depends on where you need these classes. Have you looked into
spark.files.userClassPathFirst and
bq. depend on missing fastutil classes like Long2LongOpenHashMap
Looks like Long2LongOpenHashMap should be added to the shaded jar.
Cheers
On Tue, Feb 24, 2015 at 7:36 PM, Jim Kleckner j...@cloudphysics.com wrote:
Spark includes the clearspring analytics package but intentionally excludes
