I have just upgraded to spark 1.4.0 and it seems that spark-streaming-kafka has a dependency on org.spark-project.spark unused 1.0.0 but it also embeds that jar in its artifact, causing a problem while creating a fatjar.
This is the error: [Step 1/1] (*:assembly) deduplicate: different file contents found in the > following: > > /data/system/sbt_ivy/cache/org.apache.spark/spark-streaming-kafka_2.10/jars/spark-streaming-kafka_2.10-1.4.0.jar:org/apache/spark/unused/UnusedStubClass.class > > /data/system/sbt_ivy/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
