I recall I had similar issues running Spark on Google Dataproc. sounds like it gets Hadoop's jars on the classpath which include an older version of Guava. The solution is to shade/relocate Guava in your distribution
HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 14 Feb 2022 at 19:10, Raj ks <rajabhupati....@gmail.com> wrote: > Hi Team , > > We are trying to build a docker image using Centos and trying to connect > through S3. Same works with Hadoop 3.2.0 and spark.3.1.2 > > #Installing spark binaries > ENV SPARK_HOME /opt/spark > ENV SPARK_VERSION 3.2.1 > ENV HADOOP_VERSION 3.2.0 > ARG HADOOP_VERSION_SHORT=3.2 > ARG HADOOP_AWS_VERSION=3.3.0 > ARG AWS_SDK_VERSION=1.11.563 > > > RUN set -xe \ > && cd /tmp \ > && wget > http://mirrors.gigenet.com/apache/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION_SHORT}.tgz > \ > && tar -zxvf > spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION_SHORT}.tgz \ > && rm *.tgz \ > && mv spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION_SHORT} > ${SPARK_HOME} \ > && cp ${SPARK_HOME}/kubernetes/dockerfiles/spark/entrypoint.sh > ${SPARK_HOME} \ > && wget > https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_AWS_VERSION}/hadoop-aws-${HADOOP_AWS_VERSION}.jar > \ > && wget > https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar > \ > && wget > https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/${AWS_SDK_VERSION}/aws-java-sdk-${AWS_SDK_VERSION}.jar > \ > && mv *.jar /opt/spark/jars/ > > Error: > > Any help on this is appreciated > java.lang.NoSuchMethodError: > com/google/common/base/Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V > (loaded from file:/opt/spark/jars/guava-14.0.1.jar by > jdk.internal.loader.ClassLoaders$AppClassLoader@1e4553e) called from > class org.apache.hadoop.fs.s3a.S3AUtils (loaded from > file:/opt/spark/jars/hadoop-aws-3.3.0.jar by > jdk.internal.loader.ClassLoaders$AppClassLoader@1e4553e). > >
