This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 8b8a0ffc6da0 [SPARK-57974][PYTHON][DOC] Add pandas upper bound
`<3.0.0` in `dev/requirements.txt` and `install.rst`
8b8a0ffc6da0 is described below
commit 8b8a0ffc6da05ebb02e486d08f7e0140490a718c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Jul 6 14:18:26 2026 -0700
[SPARK-57974][PYTHON][DOC] Add pandas upper bound `<3.0.0` in
`dev/requirements.txt` and `install.rst`
This PR aims to add an upper bound `<3.0.0` to the `pandas` requirement in
`dev/requirements.txt` and the installation guide
(`python/docs/source/getting_started/install.rst`) because Apache Spark 4.2.0
doesn't support it. Apache Spark 4.3.0 will support `Pandas 3` via SPARK-55139
officially.
PySpark does not yet fully support pandas 3.x and warns at runtime when
pandas >= 3.0.0 is detected:
https://github.com/apache/spark/blob/4553429c621044bb30d20b901f61bed8b2089817/python/pyspark/sql/pandas/utils.py#L63-L65
No. This only affects the development requirements file and documentation.
Manual review.
Generated-by: Claude Fable 5
Closes #57051 from dongjoon-hyun/SPARK-57974.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 3e88122235c95d671edd3019025e6929c7c3ca81)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 41dfd701c8aabdb0e9ac2dd832caf24a8aaa592c)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/requirements.txt | 2 +-
python/docs/source/getting_started/install.rst | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/dev/requirements.txt b/dev/requirements.txt
index 1ed5b4f72d65..ad804dbf5b75 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -5,7 +5,7 @@ py4j>=0.10.9.9
numpy>=1.21
pyarrow>=11.0.0
six==1.16.0
-pandas>=2.2.0
+pandas>=2.2.0,<3.0.0
scipy
plotly<6.0.0
mlflow>=2.3.1
diff --git a/python/docs/source/getting_started/install.rst
b/python/docs/source/getting_started/install.rst
index b13defdaadd6..aa75d83b7553 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -221,7 +221,7 @@ Installable with ``pip install "pyspark[connect]"``.
========================== ================= ==========================
Package Supported version Note
========================== ================= ==========================
-`pandas` >=2.0.0 Required for Spark Connect
+`pandas` >=2.0.0,<3.0.0 Required for Spark Connect
`pyarrow` >=11.0.0 Required for Spark Connect
`grpcio` >=1.67.0 Required for Spark Connect
`grpcio-status` >=1.67.0 Required for Spark Connect
@@ -237,7 +237,7 @@ Installable with ``pip install "pyspark[sql]"``.
========= ================= ======================
Package Supported version Note
========= ================= ======================
-`pandas` >=2.0.0 Required for Spark SQL
+`pandas` >=2.0.0,<3.0.0 Required for Spark SQL
`pyarrow` >=11.0.0 Required for Spark SQL
========= ================= ======================
@@ -254,7 +254,7 @@ Installable with ``pip install "pyspark[pandas_on_spark]"``.
========= ================= ================================
Package Supported version Note
========= ================= ================================
-`pandas` >=2.2.0 Required for Pandas API on Spark
+`pandas` >=2.2.0,<3.0.0 Required for Pandas API on Spark
`pyarrow` >=11.0.0 Required for Pandas API on Spark
========= ================= ================================
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
