Re: httpclient conflict in spark

[Arvind Kandaswamy]

Hi Steve,

I was indeed using spark 2.1. I was getting this error while calling spark
via Zeppelin. Zeppelin comes with older version of httpclient apparently. I
copied the httpclient 4.5.2 and httpclient 4.2.2 into
zeppelin/interpreter/spark and this problem went away.


Thank you for your help.



On Thu, Mar 30, 2017 at 9:55 AM, Steve Loughran 
wrote:

>
> On 29 Mar 2017, at 14:42, Arvind Kandaswamy 
> wrote:
>
> Hello,
>
> I am getting the following error. I get this error when trying to use AWS
> S3. This appears to be a conflict with httpclient. AWS S3 comes with
> httplient-4.5.2.jar. I am not sure how to force spark to use this version.
> I have tried spark.driver.userClassPathFirst = true, spark.executor.
> userClassPathFirst=true. Did not help. I am using Zeppelin to call the
> spark engine in case if that is an issue.
>
>
> Spark 2.x ships with httpclient 4.5.2; there is no conflict there. if you
> are on 1.6, you could actually try bumping the httplient and httpcomponents
>  to be consistent
>
> 
> 4.5.2
> 4.4.4
>
> its important to have org.apache.httpcomponents / httpcore in sync with
> httpclient; it's probably there where your problem is arising
>
> Is there anything else that I can try?
>
> java.lang.NoSuchMethodError: org.apache.http.conn.ssl.
> SSLConnectionSocketFactory.(Ljavax/net/ssl/
> SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
> at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.(
> SdkTLSSocketFactory.java:56)
> at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory
> .getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
> at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory
> .create(ApacheConnectionManagerFactory.java:65)
> at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory
> .create(ApacheConnectionManagerFactory.java:58)
> at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(
> ApacheHttpClientFactory.java:51)
> at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(
> ApacheHttpClientFactory.java:39)
> at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:314)
> at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:298)
> at com.amazonaws.AmazonWebServiceClient.(
> AmazonWebServiceClient.java:165)
> at com.amazonaws.services.s3.AmazonS3Client.(
> AmazonS3Client.java:583)
> at com.amazonaws.services.s3.AmazonS3Client.(
> AmazonS3Client.java:563)
> at com.amazonaws.services.s3.AmazonS3Client.(
> AmazonS3Client.java:541)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(
> S3AFileSystem.java:235)
>
>
>

Re: httpclient conflict in spark

[Steve Loughran]

On 29 Mar 2017, at 14:42, Arvind Kandaswamy 
> wrote:

Hello,

I am getting the following error. I get this error when trying to use AWS S3. 
This appears to be a conflict with httpclient. AWS S3 comes with 
httplient-4.5.2.jar. I am not sure how to force spark to use this version. I 
have tried spark.driver.userClassPathFirst = true, 
spark.executor.userClassPathFirst=true. Did not help. I am using Zeppelin to 
call the spark engine in case if that is an issue.


Spark 2.x ships with httpclient 4.5.2; there is no conflict there. if you are 
on 1.6, you could actually try bumping the httplient and httpcomponents  to be 
consistent


4.5.2
4.4.4

its important to have org.apache.httpcomponents / httpcore in sync with 
httpclient; it's probably there where your problem is arising

Is there anything else that I can try?

java.lang.NoSuchMethodError: 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
at 
com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.(SdkTLSSocketFactory.java:56)
at 
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
at 
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at 
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at 
com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:51)
at 
com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:39)
at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:314)
at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:298)
at com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:165)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:583)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:563)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:541)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:235)

httpclient conflict in spark

[Arvind Kandaswamy]

Hello,

I am getting the following error. I get this error when trying to use AWS
S3. This appears to be a conflict with httpclient. AWS S3 comes with
httplient-4.5.2.jar. I am not sure how to force spark to use this version.
I have tried spark.driver.userClassPathFirst = true,
spark.executor.userClassPathFirst=true. Did not help. I am using Zeppelin
to call the spark engine in case if that is an issue.

Is there anything else that I can try?

java.lang.NoSuchMethodError:
org.apache.http.conn.ssl.SSLConnectionSocketFactory.(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
at
com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.(SdkTLSSocketFactory.java:56)
at
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
at
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at
com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at
com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:51)
at
com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:39)
at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:314)
at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:298)
at
com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:165)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:583)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:563)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:541)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:235)