This is an automated email from the ASF dual-hosted git repository. zhangzc pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push: new d38ea9add [GLUTEN-5108][CH] Fix the classes in the hadoop-common conflict when running ut local (#5109) d38ea9add is described below commit d38ea9adda812e575167267254e673a81c9f2e8c Author: Zhichao Zhang <zhan...@apache.org> AuthorDate: Tue Mar 26 12:07:10 2024 +0800 [GLUTEN-5108][CH] Fix the classes in the hadoop-common conflict when running ut local (#5109) Fix the classes in the hadoop-common conflict when running ut local. Env: Idea + Spark 3.3 + Hadoop 3.3.1 UT: GlutenClickHouseMergeTreeWriteOnS3Suite / GlutenClickHouseMergeTreeWriteOnHDFSSuite Error: ``` java.lang.NoClassDefFoundError: xxxxx at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469) at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288) ``` RC: The jar `hive-hcatalog-core` includes the hadoop-common 2.7.2 and hadoop-hdfs 2.7.2, which conflict with the hadoop 3.3.1 when running ut with hadoop-3.3 profile. Close #5108. --- backends-clickhouse/pom.xml | 8 ++++++++ gluten-celeborn/clickhouse/pom.xml | 12 ++++++++++++ 2 files changed, 20 insertions(+) diff --git a/backends-clickhouse/pom.xml b/backends-clickhouse/pom.xml index f173a61c9..dd4240a3a 100644 --- a/backends-clickhouse/pom.xml +++ b/backends-clickhouse/pom.xml @@ -158,6 +158,14 @@ <artifactId>guava</artifactId> <groupId>com.google.guava</groupId> </exclusion> + <exclusion> + <artifactId>hadoop-common</artifactId> + <groupId>org.apache.hadoop</groupId> + </exclusion> + <exclusion> + <artifactId>hadoop-hdfs</artifactId> + <groupId>org.apache.hadoop</groupId> + </exclusion> </exclusions> </dependency> <dependency> diff --git a/gluten-celeborn/clickhouse/pom.xml b/gluten-celeborn/clickhouse/pom.xml index ed4784212..b44d0963c 100755 --- a/gluten-celeborn/clickhouse/pom.xml +++ b/gluten-celeborn/clickhouse/pom.xml @@ -96,6 +96,18 @@ <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> </exclusion> + <exclusion> + <artifactId>guava</artifactId> + <groupId>com.google.guava</groupId> + </exclusion> + <exclusion> + <artifactId>hadoop-common</artifactId> + <groupId>org.apache.hadoop</groupId> + </exclusion> + <exclusion> + <artifactId>hadoop-hdfs</artifactId> + <groupId>org.apache.hadoop</groupId> + </exclusion> </exclusions> </dependency> <dependency> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org