Hi Adam,
Could you try building spark with profile -Pkinesis-asl.
mvn -Pkinesis-asl -DskipTests clean package
refers to 'Running the Example' section.
https://spark.apache.org/docs/latest/streaming-kinesis-integration.html
In fact, I've seen same issue and have been able to use the AWS SDK by
trying the above.
I've also tried to use 'spark.files.userClassPathFirst' flag, and it
doesn't work.
Regards,
Yuya Urano
On 2015/03/13 3:50, Adam Lewandowski wrote:
I'm trying to use the AWS SDK (v1.9.23) to connect to DynamoDB from
within a Spark application. Spark 1.2.1 is assembled with HttpClient
4.2.6, but the AWS SDK is depending on HttpClient 4.3.4 for it's
communication with DynamoDB. The end result is an error when the app
tries to connect to DynamoDB and gets Spark's version instead:
java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpPatch
at com.amazonaws.http.AmazonHttpClient.<clinit>(AmazonHttpClient.java:129)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:359)
Caused by: java.lang.ClassNotFoundException:
org.apache.http.client.methods.HttpPatch
Including HttpClient 4.3.4 as user jars doesn't improve the situation
much:
java.lang.NoSuchMethodError:
org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
at
com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95)
I've seen the documenation regarding the
'spark.files.userClassPathFirst' flag and have tried to use it
thinking it would resolve this issue. However, when that flag is used
I get an NoClassDefFoundError on 'scala.Serializable':
java.lang.NoClassDefFoundError: scala/Serializable
...
at
org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:46)
...
Caused by: java.lang.ClassNotFoundException: scala.Serializable
This seems odd to me, since scala.Serializable is included in the
spark assembly. I thought perhaps my app was compiled against a
different scala version than spark uses, but eliminated that
possibility by using the scala compiler directly out of the spark
assembly jar with identical results.
Has anyone else seen this issue, had any success with the
"spark.files.userClassPathFirst" flag, or been able to use the AWS SDK?
I was going to submit this a Spark JIRA issue, but thought I would
check here first.
Thanks,
Adam Lewandowski
