Re: Quick one... AWS SDK version?

best of my knowledge, none is bundled >>> with Spark. >>> >>> >>> >>> I know, but if you are talking to s3 via the s3a client, you will need >>> the SDK version to match the hadoop-aws JAR of the same version of Hadoop >>> yo

Re: Quick one... AWS SDK version?

;> the SDK version to match the hadoop-aws JAR of the same version of Hadoop >> your JARs have. Similarly, if you were using spark-kinesis, it needs to be >> in sync there. >> >> >> *From:* Steve Loughran [mailto:ste...@hortonworks.com >> <ste...@hortonworks.c

Re: Quick one... AWS SDK version?

>>> SDK version to match the hadoop-aws JAR of the same version of Hadoop your >>> JARs have. Similarly, if you were using spark-kinesis, it needs to be in >>> sync there. >>>> >>>> From: Steve Loughran [mailto:ste...@hortonworks.com] >>

Re: Quick one... AWS SDK version?

gt;> the SDK version to match the hadoop-aws JAR of the same version of Hadoop >> your JARs have. Similarly, if you were using spark-kinesis, it needs to be >> in sync there. >> >> >> *From:* Steve Loughran [mailto:ste...@hortonworks.com >> <ste...@hortonworks.c

Re: Quick one... AWS SDK version?

ds to be in > sync there. > > > *From:* Steve Loughran [mailto:ste...@hortonworks.com > <ste...@hortonworks.com>] > *Sent:* Tuesday, October 03, 2017 2:20 PM > *To:* JG Perrin <jper...@lumeris.com> > *Cc:* user@spark.apache.org > *Subject:* Re: Quick one... AWS

Re: Quick one... AWS SDK version?

Perrin <jper...@lumeris.com<mailto:jper...@lumeris.com>> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: Quick one... AWS SDK version? On 3 Oct 2017, at 02:28, JG Perrin <jper...@lumeris.com<mailto:jper...@lumeris.com>> wrote: Hey Sparkians

RE: Quick one... AWS SDK version?

: user@spark.apache.org Subject: Re: Quick one... AWS SDK version? On 3 Oct 2017, at 02:28, JG Perrin <jper...@lumeris.com<mailto:jper...@lumeris.com>> wrote: Hey Sparkians, What version of AWS Java SDK do you use with Spark 2.2? Do you stick with the Hadoop 2.7.3 libs? You generall

RE: Quick one... AWS SDK version?

Thanks Yash… this is helpful! From: Yash Sharma [mailto:yash...@gmail.com] Sent: Tuesday, October 03, 2017 1:02 AM To: JG Perrin <jper...@lumeris.com>; user@spark.apache.org Subject: Re: Quick one... AWS SDK version? Hi JG, Here are my cluster configs if it helps. Cheers. EMR: emr

Re: Quick one... AWS SDK version?

On 3 Oct 2017, at 02:28, JG Perrin > wrote: Hey Sparkians, What version of AWS Java SDK do you use with Spark 2.2? Do you stick with the Hadoop 2.7.3 libs? You generally to have to stick with the version which hadoop was built with I'm

Re: Quick one... AWS SDK version?

Hi JG, Here are my cluster configs if it helps. Cheers. EMR: emr-5.8.0 Hadoop distribution: Amazon 2.7.3 AWS sdk: /usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.160.jar Applications: Hive 2.3.0 Spark 2.2.0 Tez 0.8.4 On Tue, 3 Oct 2017 at 12:29 JG Perrin wrote: > Hey

Quick one... AWS SDK version?

Hey Sparkians, What version of AWS Java SDK do you use with Spark 2.2? Do you stick with the Hadoop 2.7.3 libs? Thanks! jg