shading at the fat jar level can work, however it means that in your unit tests where spark is a provided dependency you still can get errors because spark is using an incompatible (newer) shapeless version. the unit tests run with a single resolved shapeless after all.
for example spark ships with old guava but works fine with never guava. so in our unit tests spark uses newer guava without issues, and for fat jar we shade guava in sbt-assembly. it all works ok in this situation for us. On Tue, Jan 31, 2017 at 3:06 PM, Phil Wills <otherp...@gmail.com> wrote: > Are you not able to shade it when you're building your fat jar with > something like https://github.com/sbt/sbt-assembly#shading? I would have > thought doing the shading at the app level would be a bit less painful than > doing it at the library level. > > Phil > > On Tue, 31 Jan 2017, 04:24 Timothy Chan, <tc...@lumoslabs.com> wrote: > >> I'm using a library, https://github.com/guardian/scanamo, that uses >> shapeless 2.3.2. What are my options if I want to use this with Spark >> 2.1.0? >> >> Based on this: http://apache-spark-developers-list.1001551.n3. >> nabble.com/shapeless-in-spark-2-1-0-tt20392.html >> >> I'm guessing I would have to release my own version of scanamo with a >> shaded shapeless? >> >
