This is an automated email from the ASF dual-hosted git repository.
SteNicholas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new a1ecf1926e [CELEBORN-2372] Remove Hadoop 2 client support
a1ecf1926e is described below
commit a1ecf1926e21590a0c1df816a9e6093b2ebbf69c
Author: Nicholas Jiang <[email protected]>
AuthorDate: Sun Jun 28 14:34:52 2026 +0800
[CELEBORN-2372] Remove Hadoop 2 client support
### What changes were proposed in this pull request?
This PR removes Hadoop 2 client support from Celeborn. Specifically:
- Remove the `hadoop-2` Maven profile from the root `pom.xml`, which was
activated when `hadoop.version` matched `2.*` and pulled in `hadoop-client`
with a long list of exclusions.
- Remove the corresponding `hadoop-2` profiles (activated via the
`hadoop-2-deps` property) from `common/pom.xml`, `master/pom.xml`, and
`client-mr/mr/pom.xml`.
- Update `README.md` to reflect that Celeborn supports Hadoop MapReduce 3
(instead of 2/3).
### Why are the changes needed?
Hadoop 2 is an old release line that is no longer actively maintained.
Dropping it reduces the build/dependency matrix and maintenance burden,
allowing the project to focus on Hadoop 3.
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [x] Yes
Celeborn no longer provides the `hadoop-2` build profile. Users building
against Hadoop 2 must either upgrade to Hadoop 3 or stay on a previous Celeborn
release.
### How was this patch tested?
Existing CI. No new functionality is introduced; the change only removes
build profiles and references.
Closes #3747 from SteNicholas/CELEBORN-2372.
Authored-by: Nicholas Jiang <[email protected]>
Signed-off-by: Nicholas Jiang <[email protected]>
---
README.md | 2 +-
client-mr/mr/pom.xml | 15 ---------------
common/pom.xml | 15 ---------------
master/pom.xml | 15 ---------------
pom.xml | 53 ----------------------------------------------------
5 files changed, 1 insertion(+), 99 deletions(-)
diff --git a/README.md b/README.md
index cb52ea91ae..4e435ce422 100644
--- a/README.md
+++ b/README.md
@@ -41,7 +41,7 @@ Celeborn Worker's slot count is decided by `total usable disk
size / average shu
Celeborn worker's slot count decreases when a partition is allocated and
increments when a partition is freed.
## Build
-1. Celeborn supports Spark 3.0/3.1/3.2/3.3/3.4/3.5/4.0, Flink
1.18/1.19/1.20/2.0/2.1/2.2 and Hadoop MapReduce 2/3.
+1. Celeborn supports Spark 3.0/3.1/3.2/3.3/3.4/3.5/4.0, Flink
1.18/1.19/1.20/2.0/2.1/2.2 and Hadoop MapReduce 3.
2. Celeborn tested under Scala 2.11/2.12/2.13 and Java 8/11/17 environment.
Build Celeborn via `make-distribution.sh`:
diff --git a/client-mr/mr/pom.xml b/client-mr/mr/pom.xml
index 7321c9dbba..06d72c2d66 100644
--- a/client-mr/mr/pom.xml
+++ b/client-mr/mr/pom.xml
@@ -82,20 +82,5 @@
</dependency>
</dependencies>
</profile>
- <profile>
- <id>hadoop-2</id>
- <activation>
- <property>
- <name>hadoop-2-deps</name>
- </property>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
diff --git a/common/pom.xml b/common/pom.xml
index 82c4e31f8f..4467279159 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -214,20 +214,5 @@
</dependency>
</dependencies>
</profile>
- <profile>
- <id>hadoop-2</id>
- <activation>
- <property>
- <name>hadoop-2-deps</name>
- </property>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
diff --git a/master/pom.xml b/master/pom.xml
index c3c64ff073..501475a2eb 100644
--- a/master/pom.xml
+++ b/master/pom.xml
@@ -208,21 +208,6 @@
</dependency>
</dependencies>
</profile>
- <profile>
- <id>hadoop-2</id>
- <activation>
- <property>
- <name>hadoop-2-deps</name>
- </property>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
<profile>
<id>aws</id>
<dependencies>
diff --git a/pom.xml b/pom.xml
index 8d106f9ac4..b2e1a9cacd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1441,59 +1441,6 @@
</dependency>
</dependencies>
</profile>
- <profile>
- <id>hadoop-2</id>
- <activation>
- <property>
- <name>hadoop.version</name>
- <value>/^2\..*$/</value>
- </property>
- </activation>
- <properties>
- <hadoop-2-deps>true</hadoop-2-deps>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <version>${hadoop.version}</version>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-annotations</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-app</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-client</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-registry</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-server-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.fusesource.leveldbjni</groupId>
- <artifactId>leveldbjni-all</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- </dependencies>
- </profile>
<profile>
<id>aws</id>
<modules>
