Hi Vadim, After removing "provided" from "org.apache.spark" %% "spark-streaming-kinesis-asl" I ended up with huge number of deduplicate errors:
https://gist.github.com/trienism/3d6f8d6b7ff5b7cead6a It would be nice if you could share some pieces of your mergeStrategy code for reference. Also, after adding "provided" back to "spark-streaming-kinesis-asl" and I submit the spark job with the spark-streaming-kinesis-asl jar file sh /usr/lib/spark/bin/spark-submit --verbose --jars lib/spark-streaming-kinesis-asl_2.10-1.2.0.jar --class com.xxx.DataConsumer target/scala-2.10/xxx-assembly-0.1-SNAPSHOT.jar I still end up with the following error... Exception in thread "main" java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:379) Has anyone else run into this issue? On Mon, Apr 13, 2015 at 6:46 PM, Vadim Bichutskiy < vadim.bichuts...@gmail.com> wrote: > I don't believe the Kinesis asl should be provided. I used mergeStrategy > successfully to produce an "uber jar." > > Fyi, I've been having trouble consuming data out of Kinesis with Spark > with no success :( > Would be curious to know if you got it working. > > Vadim > > On Apr 13, 2015, at 9:36 PM, Mike Trienis <mike.trie...@orcsol.com> wrote: > > Hi All, > > I have having trouble building a fat jar file through sbt-assembly. > > [warn] Merging 'META-INF/NOTICE.txt' with strategy 'rename' > [warn] Merging 'META-INF/NOTICE' with strategy 'rename' > [warn] Merging 'META-INF/LICENSE.txt' with strategy 'rename' > [warn] Merging 'META-INF/LICENSE' with strategy 'rename' > [warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard' > [warn] Merging > 'META-INF/maven/com.thoughtworks.paranamer/paranamer/pom.properties' with > strategy 'discard' > [warn] Merging > 'META-INF/maven/com.thoughtworks.paranamer/paranamer/pom.xml' with strategy > 'discard' > [warn] Merging 'META-INF/maven/commons-dbcp/commons-dbcp/pom.properties' > with strategy 'discard' > [warn] Merging 'META-INF/maven/commons-dbcp/commons-dbcp/pom.xml' with > strategy 'discard' > [warn] Merging 'META-INF/maven/commons-pool/commons-pool/pom.properties' > with strategy 'discard' > [warn] Merging 'META-INF/maven/commons-pool/commons-pool/pom.xml' with > strategy 'discard' > [warn] Merging 'META-INF/maven/joda-time/joda-time/pom.properties' with > strategy 'discard' > [warn] Merging 'META-INF/maven/joda-time/joda-time/pom.xml' with strategy > 'discard' > [warn] Merging 'META-INF/maven/log4j/log4j/pom.properties' with strategy > 'discard' > [warn] Merging 'META-INF/maven/log4j/log4j/pom.xml' with strategy 'discard' > [warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.properties' with > strategy 'discard' > [warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.xml' with > strategy 'discard' > [warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.properties' with > strategy 'discard' > [warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.xml' with strategy > 'discard' > [warn] Merging 'META-INF/maven/org.slf4j/slf4j-log4j12/pom.properties' > with strategy 'discard' > [warn] Merging 'META-INF/maven/org.slf4j/slf4j-log4j12/pom.xml' with > strategy 'discard' > [warn] Merging 'META-INF/services/java.sql.Driver' with strategy > 'filterDistinctLines' > [warn] Merging 'rootdoc.txt' with strategy 'concat' > [warn] Strategy 'concat' was applied to a file > [warn] Strategy 'discard' was applied to 17 files > [warn] Strategy 'filterDistinctLines' was applied to a file > [warn] Strategy 'rename' was applied to 4 files > > When submitting the spark application through the command > > sh /usr/lib/spark/bin/spark-submit -class com.xxx.ExampleClassName > target/scala-2.10/xxxx-snapshot.jar > > I end up the the following error, > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/joda/time/format/DateTimeFormat > at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at java.lang.Class.newInstance(Class.java:379) > at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119) > at > com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105) > at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78) > at > com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307) > at > com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280) > at > com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.setEndpoint(AmazonKinesisClient.java:2102) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.init(AmazonKinesisClient.java:216) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:202) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:175) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:155) > at com.quickstatsengine.aws.AwsProvider$.<init>(AwsProvider.scala:20) > at com.quickstatsengine.aws.AwsProvider$.<clinit>(AwsProvider.scala) > > The snippet from my build.sbt file is: > > "org.apache.spark" %% "spark-core" % "1.2.0" % "provided", > "org.apache.spark" %% "spark-streaming" % "1.2.0" % "provided", > "com.datastax.spark" %% "spark-cassandra-connector" % > "1.2.0-alpha1" % "provided", > "org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.2.0" % > "provided", > > And the error is originating from: > > val kinesisClient = new AmazonKinesisClient(new > DefaultAWSCredentialsProviderChain()) > > Am I correct to set spark-streaming-kinesis-asl as a *provided *dependency? > Also, is there a merge strategy I need apply? > > Any help would be appreciated, Mike. > > >
