[
https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070774#comment-17070774
]
Jorge Machado commented on SPARK-30272:
---------------------------------------
I failed to fix the guava stuff of course ... Today morning I tried to
replicate the problem of the missing azure-hadoop jar but It seems to be
working without any patch from my side. . I assume that I did something wrong
on build. Just for reference my steps:
{code:java}
git checkout v3.0.0-preview-rc2
./build/mvn clean package -DskipTests -Phadoop-3.2 -Pkubernetes
-Phadoop-cloud
./bin/docker-image-tool.sh -r docker.io/myrepo -t v2.3.0 -p
kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
docker run --rm -it myrepo/spark:v2.3.0 bash
185@57fb3dd68902:/opt/spark/jars$ ls -altr *azure*
-rw-r--r-- 1 root root 67314 Mar 28 17:15
hadoop-azure-datalake-3.2.0.jar
-rw-r--r-- 1 root root 480512 Mar 28 17:15 hadoop-azure-3.2.0.jar
-rw-r--r-- 1 root root 812977 Mar 28 17:15 azure-storage-7.0.0.jar
-rw-r--r-- 1 root root 10288 Mar 28 17:15 azure-keyvault-core-1.0.0.jar
-rw-r--r-- 1 root root 94061 Mar 28 17:15
azure-data-lake-store-sdk-2.2.9.jar
{code}
As you see the hadoop-azure is there but not on version 3.2.1 but I guess this
is a matter of updating the pom.
> Remove usage of Guava that breaks in Guava 27
> ---------------------------------------------
>
> Key: SPARK-30272
> URL: https://issues.apache.org/jira/browse/SPARK-30272
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
> Affects Versions: 3.0.0
> Reporter: Sean R. Owen
> Assignee: Sean R. Owen
> Priority: Major
> Fix For: 3.0.0
>
>
> Background:
> https://issues.apache.org/jira/browse/SPARK-29250
> https://github.com/apache/spark/pull/25932
> Hadoop 3.2.1 will update Guava from 11 to 27. There are a number of methods
> that changed between those releases, typically just a rename, but, means one
> set of code can't work with both, while we want to work with Hadoop 2.x and
> 3.x. Among them:
> - Objects.toStringHelper was moved to MoreObjects; we can just use the
> Commons Lang3 equivalent
> - Objects.hashCode etc were renamed; use java.util.Objects equivalents
> - MoreExecutors.sameThreadExecutor() became directExecutor(); for same-thread
> execution we can use a dummy implementation of ExecutorService / Executor
> - TypeToken.isAssignableFrom become isSupertypeOf; work around with reflection
> There is probably more to the Guava issue than just this change, but it will
> make Spark itself work with more versions and reduce our exposure to Guava
> along the way anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
