This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 6b29b28 [SPARK-38696][BUILD] Add `commons-collections` back for hadoop-3 profile 6b29b28 is described below commit 6b29b28deffd11edd65b69e0f5c79ed51d483b66 Author: Dongjoon Hyun <dongj...@apache.org> AuthorDate: Wed Mar 30 07:26:21 2022 -0700 [SPARK-38696][BUILD] Add `commons-collections` back for hadoop-3 profile ### What changes were proposed in this pull request? - SPARK-37968 replaced `commons-collections 3.x` with `commons-collections4`. - SPARK-37600 upgrades to Apache Hadoop 3.3.2. This PR adds it back for `hadoop-3` profile because Apache Hadoop 3.3.2 is still using it. Since SPARK-37968 didn't remove it from `hadoop-2` profile , this is a regression in `hadoop-3` profile only. ### Why are the changes needed? [HADOOP-17139](https://issues.apache.org/jira/browse/HADOOP-17139) added `commons-collections` usage at 3.3.2. Without this patch, `ClassNotFound` error happens while using S3A filesystem. https://github.com/apache/hadoop/blob/f91452b289aea1418f56d242c046b58d9f214a1d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java#L41 ``` import org.apache.commons.collections.comparators.ReverseComparator; ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This is a kind of dependency recovery. This should pass CIs. Closes #36009 from dongjoon-hyun/SPARK-38696. Authored-by: Dongjoon Hyun <dongj...@apache.org> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- core/pom.xml | 4 ++++ dev/deps/spark-deps-hadoop-3-hive-2.3 | 1 + pom.xml | 10 ++++++++-- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/core/pom.xml b/core/pom.xml index 24294a2..c1e494f 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -193,6 +193,10 @@ <artifactId>commons-io</artifactId> </dependency> <dependency> + <groupId>commons-collections</groupId> + <artifactId>commons-collections</artifactId> + </dependency> + <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-collections4</artifactId> </dependency> diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 27b0ec5..08fa631 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -38,6 +38,7 @@ chill-java/0.10.0//chill-java-0.10.0.jar chill_2.12/0.10.0//chill_2.12-0.10.0.jar commons-cli/1.5.0//commons-cli-1.5.0.jar commons-codec/1.15//commons-codec-1.15.jar +commons-collections/3.2.2//commons-collections-3.2.2.jar commons-collections4/4.4//commons-collections4-4.4.jar commons-compiler/3.0.16//commons-compiler-3.0.16.jar commons-compress/1.21//commons-compress-1.21.jar diff --git a/pom.xml b/pom.xml index 6c5b321..d45d6ca 100644 --- a/pom.xml +++ b/pom.xml @@ -160,7 +160,8 @@ <commons.httpcore.version>4.4.14</commons.httpcore.version> <commons.math3.version>3.6.1</commons.math3.version> <!-- managed up from 3.2.1 for SPARK-11652 --> - <commons.collections.version>4.4</commons.collections.version> + <commons.collections.version>3.2.2</commons.collections.version> + <commons.collections4.version>4.4</commons.collections4.version> <scala.version>2.12.15</scala.version> <scala.binary.version>2.12</scala.binary.version> <scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version> @@ -620,9 +621,14 @@ <version>${commons.math3.version}</version> </dependency> <dependency> + <groupId>commons-collections</groupId> + <artifactId>commons-collections</artifactId> + <version>${commons.collections.version}</version> + </dependency> + <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-collections4</artifactId> - <version>${commons.collections.version}</version> + <version>${commons.collections4.version}</version> </dependency> <dependency> <groupId>commons-beanutils</groupId> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org