I ran into the same issue. What I did was use maven shade plugin to shade my version of httpcomponents libraries into another package.
On Fri, Sep 5, 2014 at 4:33 PM, Penny Espinoza < pesp...@societyconsulting.com> wrote: > Hey - I’m struggling with some dependency issues with > org.apache.httpcomponents httpcore and httpclient when using spark-submit > with YARN running Spark 1.0.2 on a Hadoop 2.2 cluster. I’ve seen several > posts about this issue, but no resolution. > > The error message is this: > > > Caused by: java.lang.NoSuchMethodError: > org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V > at > org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140) > at > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114) > at > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99) > at > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85) > at > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93) > at > com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26) > at > com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96) > at > com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155) > at > com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:118) > at > com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:102) > at > com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:332) > at > com.oncue.rna.realtime.streaming.config.package$.transferManager(package.scala:76) > at > com.oncue.rna.realtime.streaming.models.S3SchemaRegistry.<init>(SchemaRegistry.scala:27) > at > com.oncue.rna.realtime.streaming.models.S3SchemaRegistry$.schemaRegistry$lzycompute(SchemaRegistry.scala:46) > at > com.oncue.rna.realtime.streaming.models.S3SchemaRegistry$.schemaRegistry(SchemaRegistry.scala:44) > at > com.oncue.rna.realtime.streaming.coders.KafkaAvroDecoder.<init>(KafkaAvroDecoder.scala:20) > ... 17 more > > The apache httpcomponents libraries include the method above as of > version 4.2. The Spark 1.0.2 binaries seem to include version 4.1. > > I can get this to work in my driver program by adding exclusions to > force use of 4.1, but then I get the error in tasks even when using the > —jars option of the spark-submit command. How can I get both the driver > program and the individual tasks in my spark-streaming job to use the same > version of this library so my job will run all the way through? > > thanks > p >
