I'm trying to use the AWS SDK (v1.9.23) to connect to DynamoDB from within a Spark application. Spark 1.2.1 is assembled with HttpClient 4.2.6, but the AWS SDK is depending on HttpClient 4.3.4 for it's communication with DynamoDB. The end result is an error when the app tries to connect to DynamoDB and gets Spark's version instead: java.lang.NoClassDefFoundError: org/apache/http/client/methods/HttpPatch at com.amazonaws.http.AmazonHttpClient.<clinit>(AmazonHttpClient.java:129) at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:359) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.methods.HttpPatch
Including HttpClient 4.3.4 as user jars doesn't improve the situation much: java.lang.NoSuchMethodError: org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95) I've seen the documenation regarding the 'spark.files.userClassPathFirst' flag and have tried to use it thinking it would resolve this issue. However, when that flag is used I get an NoClassDefFoundError on 'scala.Serializable': java.lang.NoClassDefFoundError: scala/Serializable ... at org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:46) ... Caused by: java.lang.ClassNotFoundException: scala.Serializable This seems odd to me, since scala.Serializable is included in the spark assembly. I thought perhaps my app was compiled against a different scala version than spark uses, but eliminated that possibility by using the scala compiler directly out of the spark assembly jar with identical results. Has anyone else seen this issue, had any success with the "spark.files.userClassPathFirst" flag, or been able to use the AWS SDK? I was going to submit this a Spark JIRA issue, but thought I would check here first. Thanks, Adam Lewandowski
