This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 904bfeacf58f [SPARK-57974][PYTHON][DOC] Add pandas upper bound
`<3.0.0` in `dev/requirements.txt` and `install.rst`
904bfeacf58f is described below
commit 904bfeacf58fbccef87117b98a407d505909e0cb
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Jul 6 14:18:26 2026 -0700
[SPARK-57974][PYTHON][DOC] Add pandas upper bound `<3.0.0` in
`dev/requirements.txt` and `install.rst`
### What changes were proposed in this pull request?
This PR aims to add an upper bound `<3.0.0` to the `pandas` requirement in
`dev/requirements.txt` and the installation guide
(`python/docs/source/getting_started/install.rst`) because Apache Spark 4.2.0
doesn't support it. Apache Spark 4.3.0 will support `Pandas 3` via SPARK-55139
officially.
### Why are the changes needed?
PySpark does not yet fully support pandas 3.x and warns at runtime when
pandas >= 3.0.0 is detected:
https://github.com/apache/spark/blob/4553429c621044bb30d20b901f61bed8b2089817/python/pyspark/sql/pandas/utils.py#L63-L65
### Does this PR introduce _any_ user-facing change?
No. This only affects the development requirements file and documentation.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57051 from dongjoon-hyun/SPARK-57974.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 3e88122235c95d671edd3019025e6929c7c3ca81)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/requirements.txt | 2 +-
python/docs/source/getting_started/install.rst | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/dev/requirements.txt b/dev/requirements.txt
index 525353a2b5f8..8d4def475c3a 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -4,7 +4,7 @@ py4j>=0.10.9.9
# PySpark dependencies (optional)
numpy>=1.22
pyarrow>=18.0.0
-pandas>=2.2.0
+pandas>=2.2.0,<3.0.0
scipy>=1.8.0
plotly<6.0.0
mlflow>=2.3.1
diff --git a/python/docs/source/getting_started/install.rst
b/python/docs/source/getting_started/install.rst
index 1b122e197c49..1ff1587035c9 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -237,7 +237,7 @@ Installable with ``pip install "pyspark[connect]"``.
========================== ================= ==========================
Package Supported version Note
========================== ================= ==========================
-`pandas` >=2.2.0 Required for Spark Connect
+`pandas` >=2.2.0,<3.0.0 Required for Spark Connect
`pyarrow` >=18.0.0 Required for Spark Connect
`grpcio` >=1.76.0 Required for Spark Connect
`grpcio-status` >=1.76.0 Required for Spark Connect
@@ -254,7 +254,7 @@ Installable with ``pip install "pyspark[sql]"``.
========= ================= ======================
Package Supported version Note
========= ================= ======================
-`pandas` >=2.2.0 Required for Spark SQL
+`pandas` >=2.2.0,<3.0.0 Required for Spark SQL
`pyarrow` >=18.0.0 Required for Spark SQL
========= ================= ======================
@@ -271,7 +271,7 @@ Installable with ``pip install "pyspark[pandas_on_spark]"``.
========= ================= ================================
Package Supported version Note
========= ================= ================================
-`pandas` >=2.2.0 Required for Pandas API on Spark
+`pandas` >=2.2.0,<3.0.0 Required for Pandas API on Spark
`pyarrow` >=18.0.0 Required for Pandas API on Spark
========= ================= ================================
@@ -321,7 +321,7 @@ Installable with ``pip install "pyspark[pipelines]"``.
Includes all dependencies
========================== =================
===================================================
Package Supported version Note
========================== =================
===================================================
-`pandas` >=2.2.0 Required for Spark Connect and
Spark SQL
+`pandas` >=2.2.0,<3.0.0 Required for Spark Connect and
Spark SQL
`pyarrow` >=18.0.0 Required for Spark Connect and
Spark SQL
`grpcio` >=1.76.0 Required for Spark Connect
`grpcio-status` >=1.76.0 Required for Spark Connect
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
