I'm using Spark 1.0 and sbt assembly plugin to create uberjar of my application. However, when I run assembly command, I get a number of errors like this:
java.lang.RuntimeException: deduplicate: different file contents found in the following: /home/username/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class /home/username/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class ... As far as I can see, Spark Core depends on both - Minlog and Kryo, and the latter includes Minlog classes itself. Classes are binary different, so assembly can't combine them. And there's a number of such conflicts - I fixed some of them manually via mergeStrategy, but list of exceptions becomes larger and larger. I can continues, but it just does't look like the right way. My questions are: 1. Is an uberjar a recommended way of running Spark applications? 2. If so, should I include Spark itself into this large jar? 3. If not, what is a recommended way to do both - development and deployment (assuming ordinary sbt project). Thanks, Andrei
