This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new cf4e6f761907 [SPARK-58053][DOC] Update `building-spark.md` to require
Hadoop 3.4.0 or later
cf4e6f761907 is described below
commit cf4e6f761907647b6e4cd1dd2285d6ffb1789722
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Jul 8 19:31:47 2026 -0700
[SPARK-58053][DOC] Update `building-spark.md` to require Hadoop 3.4.0 or
later
### What changes were proposed in this pull request?
This PR aims to update `building-spark.md` to document that building Apache
Spark requires Hadoop 3.4.0 or later.
Note that this is only a build requirement for Hadoop shaded client
dependencies which has been used since Apache Spark 3.2.0 (SPARK-33212). This
is irrelevant to the Hadoop cluster versions. **Apache Spark will support both
old and new Apache Hadoop clusters like we did so far as long as Apache Hadoop
clients support them.**
- #30701
### Why are the changes needed?
Apache Hadoop 3.3.x has no release for last 3 years. v3.3.6 was the last
one.
- https://github.com/apache/hadoop/releases/tag/rel%2Frelease-3.3.6
(2023-06-26)
The Apache Spark community has been using Hadoop 3.4.1+ by default since
Spark 4.0.0. We didn't test Hadoop 3.3 officially in our CIs for Spark 4.x.
Given that the lack of ASF infra resource, it becomes worse and worse.
- https://github.com/apache/spark/pull/48295 (Apache Spark 4.0.0)
- https://github.com/apache/spark/pull/51127 (Apache Spark 4.1.0)
- https://github.com/apache/spark/pull/54448 (Apache Spark 4.2.0)
- New Apache Spark Release Cadence implies that we use Hadoop 3.5.0 in next
Spark 4.x line (4.3.0, 4.4.0, 4.5.0).
Currently, the `Building Spark` guide does not mention the minimum
supported Hadoop version for the `hadoop.version` property. It's misleading to
the users because we claim we are not going to test. Since Apache Hadoop 3.3
and older release lines are apparently EOL or no longer maintained, Apache
Spark 5.0.0 had better support building against Hadoop 3.4, 3.5 or later only
to make it clear. This PR aim to clarify it in the build documentation.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only update.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57141 from dongjoon-hyun/SPARK-58053.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/building-spark.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 28f7c2b1c0d1..ad95b452ca31 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -80,6 +80,7 @@ For more information on usage, run
`./dev/make-distribution.sh --help`
## Specifying the Hadoop Version and Enabling YARN
You can enable the `yarn` profile and specify the exact version of Hadoop to
compile against through the `hadoop.version` property.
+Spark requires Hadoop 3.4.0 or later; building against older Hadoop versions
is not supported.
Example:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
