This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new f6f0cd104430 [SPARK-58015][INFRA] Move dependency list from
requirements.txt to pyproject.toml
f6f0cd104430 is described below
commit f6f0cd1044307804982c7cd08d71f1f3e098d7df
Author: Tian Gao <[email protected]>
AuthorDate: Mon Jul 13 11:45:18 2026 -0700
[SPARK-58015][INFRA] Move dependency list from requirements.txt to
pyproject.toml
### What changes were proposed in this pull request?
1. Moved the dependency list from `requirements.txt` to `pyproject.toml`
(`requirements.txt` file is not deleted in this PR for compatibility, it will
eventually be removed).
2. Update the docs, including `AGENTS.md`, to use `pyproject.toml` instead
of `dev/requirements.txt`.
3. Added the explanation for both human and agents to use `pip>=25.1` which
supports the dependency groups.
### Why are the changes needed?
This is just the first step of an important project - to put all of our
python dependencies in a single place. To be more specific, dependencies for
1. development (currently `requirements.txt`)
2. CI (individually defined in docker files and workflow files)
3. packaging (defined in `setup.py`)
Every time we change a dependency, we need to change it all over our code
base and we probably will miss some. What's even worse is that we had to pin
dependencies for different release branches.
With the effort to decouple CI, we basically solve the release branch
dependency issue (except for 4.2 which has been in rc stage for a while - I
want to wait for its release to decouple it). We don't need to keep
dependencies for different release branches anymore.
Now we want to move dependency pins from docker files and workflows to
`pyproject.toml` - `requirements.txt` can't do it because the testing
environment is complicated. They share some dependencies but also require
different ones for each workflow. We need group structure to make it possible
and that's where [dependency
group](https://packaging.python.org/en/latest/specifications/dependency-groups/)
comes in. It's a relatively new feature
[PEP-735](https://peps.python.org/pep-0735/) w [...]
I believe with this approach, we can eliminate all the dependencies in the
docker file and workflow file. We should be able to have a single source of
truth of the Python dependency. And it will be more structured.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This PR should not affect CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57093 from gaogaotiantian/dependency-unify.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit a4d5970af510f2755133faadfdbc1c1c95d34644)
Signed-off-by: Tian Gao <[email protected]>
---
AGENTS.md | 3 +-
dev/spark-test-image/connect-gen-protos/Dockerfile | 2 +-
docs/README.md | 2 +-
pyproject.toml | 130 +++++++++++++++++++++
python/README.md | 2 +-
python/docs/source/development/contributing.rst | 12 +-
.../pyspark/sql/tests/streaming/KAFKA_TESTING.md | 2 +-
sql/connect/README.md | 2 +-
sql/connect/common/src/main/protobuf/README.md | 4 +-
9 files changed, 148 insertions(+), 11 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index ed48b1c7a8f1..4d1292ec385c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -140,7 +140,8 @@ If the default venv does not exist, create it:
python3 -m venv .venv
source .venv/bin/activate
- pip install -r dev/requirements.txt
+ pip install --upgrade pip
+ pip install --group dev
Run a single test suite:
diff --git a/dev/spark-test-image/connect-gen-protos/Dockerfile
b/dev/spark-test-image/connect-gen-protos/Dockerfile
index 13ff7682df29..5e90254f8167 100644
--- a/dev/spark-test-image/connect-gen-protos/Dockerfile
+++ b/dev/spark-test-image/connect-gen-protos/Dockerfile
@@ -51,7 +51,7 @@ ENV VIRTUAL_ENV=/opt/spark-venv
RUN python3.12 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
-# Keep these versions in sync with the pinned versions in dev/requirements.txt
+# Keep these versions in sync with the pinned versions in pyproject.toml
RUN python3.12 -m pip install \
'mypy==1.19.1' \
'mypy-protobuf==3.3.0' \
diff --git a/docs/README.md b/docs/README.md
index d896cae0b09a..bbef64ef218e 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -50,7 +50,7 @@ And the required Python dependencies can be installed using
pip:
```sh
$ cd "$SPARK_HOME"
-$ pip install --upgrade -r dev/requirements.txt
+$ pip install --upgrade --group dev
```
To generate the Python or R API docs, you'll also need to [install
Pandoc](https://pandoc.org/installing.html).
diff --git a/pyproject.toml b/pyproject.toml
index a28533c3539b..ccac0e9d371c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -68,3 +68,133 @@ line-length = 100
target-version = ['py39']
include = '\.pyi?$'
extend-exclude = 'cloudpickle|error_classes.py'
+
+[dependency-groups]
+_py4j = ["py4j>=0.10.9.9"]
+
+_numpy = ["numpy>=1.23.2"]
+_pandas = ["pandas>=2.2.0,<3.0.0"]
+_protobuf = ["protobuf==6.33.5"]
+_pyarrow = ["pyarrow>=18.0.0"]
+
+_pyspark_devtool = [
+ # Optional tools that are used by pyspark directly.
+ "memory-profiler>=0.61.0",
+ "flameprof==0.4",
+ "psutil",
+ "pystack>=1.6.0; sys_platform=='linux'",
+]
+
+connect = [
+ {include-group = "_py4j"},
+ {include-group = "_pandas"},
+ {include-group = "_protobuf"},
+ {include-group = "_pyarrow"},
+ "grpcio>=1.76.0",
+ "grpcio-status>=1.76.0",
+ "googleapis-common-protos>=1.71.0",
+ "zstandard>=0.25.0",
+ "graphviz==0.20.3",
+]
+
+sql = [
+ {include-group = "_py4j"},
+ {include-group = "_pandas"},
+ {include-group = "_pyarrow"},
+]
+
+pandas_on_spark = [
+ {include-group = "_py4j"},
+ {include-group = "_pandas"},
+ {include-group = "_pyarrow"},
+]
+
+pandas_on_spark_extra = [
+ "plotly<6.0.0",
+ "matplotlib",
+ "mlflow>=2.3.1",
+]
+
+ml = [
+ {include-group = "_py4j"},
+ {include-group = "_numpy"},
+]
+
+mllib = [
+ {include-group = "_py4j"},
+ {include-group = "_numpy"},
+]
+
+ml_extra = [
+ "scipy>=1.8.0",
+ "scikit-learn",
+ "torch",
+ "torchvision",
+ "torcheval",
+ "deepspeed; sys_platform != 'darwin'",
+]
+
+pipelines = [
+ {include-group = "connect"},
+ "pyyaml>=3.11",
+]
+
+_lint = [
+ "ruff==0.14.8",
+ "mypy==1.19.1",
+ "pytest-mypy-plugins==3.2.0",
+ "mypy-protobuf==3.3.0",
+ "googleapis-common-protos-stubs==2.2.0",
+ "grpc-stubs==1.24.11",
+ "pandas-stubs>=2.2.0",
+ "scipy-stubs",
+ "types-PyYAML"
+]
+
+_test = [
+ # Dependency for running tests
+ "unittest-xml-reporting",
+ # 3rd party libraries that unittests use.
+ "openpyxl",
+ "tabulate",
+ "testcontainers[kafka]>=3.7.0",
+ "kafka-python-ng>=2.0.2",
+]
+
+_docs = [
+ "sphinx==8.2.3",
+ "sphinx-plotly-directive",
+ "sphinx-copybutton",
+ "docutils",
+ "markupsafe",
+ "pydata_sphinx_theme>=0.13",
+ "ipython",
+ "nbsphinx",
+ "numpydoc",
+ "jinja2",
+]
+
+_devtools = [
+ # Devtools that are used by pyspark developers.
+ "debugpy",
+ "viztracer",
+ "jira>=3.5.2",
+ "PyGithub",
+ "asv",
+]
+
+dev = [
+ {include-group = "_pyspark_devtool"},
+ {include-group = "_lint"},
+ {include-group = "_test"},
+ {include-group = "_docs"},
+ {include-group = "_devtools"},
+ {include-group = "sql"},
+ {include-group = "connect"},
+ {include-group = "ml"},
+ {include-group = "ml_extra"},
+ {include-group = "mllib"},
+ {include-group = "pandas_on_spark"},
+ {include-group = "pandas_on_spark_extra"},
+ {include-group = "pipelines"},
+]
diff --git a/python/README.md b/python/README.md
index 1162990a18f7..ca4e493fdd39 100644
--- a/python/README.md
+++ b/python/README.md
@@ -30,4 +30,4 @@ The Python packaging for Spark is not intended to replace all
of the other use c
## Python Requirements
At its core PySpark depends on Py4J, but some additional sub-packages have
their own extra requirements for some features (including numpy, pandas, and
pyarrow).
-See also
[Dependencies](https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies)
for production, and
[dev/requirements.txt](https://github.com/apache/spark/blob/master/dev/requirements.txt)
for development.
+See also
[Dependencies](https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies)
for production, and
[pyproject.toml](https://github.com/apache/spark/blob/master/pyproject.toml)
for development.
diff --git a/python/docs/source/development/contributing.rst
b/python/docs/source/development/contributing.rst
index ddfd776c8f85..ee26de1af211 100644
--- a/python/docs/source/development/contributing.rst
+++ b/python/docs/source/development/contributing.rst
@@ -122,6 +122,12 @@ PySpark development requires to build Spark that needs a
proper JDK installed, e
Note that if you intend to contribute to Spark Connect in Python, ``buf`` is
required, see `Buf Installation <https://docs.buf.build/installation>`_ for
more details.
+``pip>=25.1`` is required to install the dependencies. If your ``pip`` version
is older than 25.1, you can upgrade it by:
+
+.. code-block:: bash
+
+ pip install --upgrade pip
+
Conda
~~~~~
@@ -132,7 +138,7 @@ If you are using Conda, the development environment can be
set as follows.
# Python 3.11+ is required
conda create --name pyspark-dev-env python=3.11
conda activate pyspark-dev-env
- pip install --upgrade -r dev/requirements.txt
+ pip install --upgrade --group dev
Once it is set up, make sure you switch to `pyspark-dev-env` before starting
the development:
@@ -152,7 +158,7 @@ You can use Python's built-in ``venv`` module to create an
isolated environment:
# Python 3.11+ is required
python3 -m venv .venv
source .venv/bin/activate
- pip install --upgrade -r dev/requirements.txt
+ pip install --upgrade --group dev
pip
~~~
@@ -161,7 +167,7 @@ With Python 3.11+, pip can be used as below to install and
set up the developmen
.. code-block:: bash
- pip install --upgrade -r dev/requirements.txt
+ pip install --upgrade --group dev
Now, you can start developing and `running the tests <testing.rst>`_.
diff --git a/python/pyspark/sql/tests/streaming/KAFKA_TESTING.md
b/python/pyspark/sql/tests/streaming/KAFKA_TESTING.md
index fe8631b6eea2..37a4c11d35dc 100644
--- a/python/pyspark/sql/tests/streaming/KAFKA_TESTING.md
+++ b/python/pyspark/sql/tests/streaming/KAFKA_TESTING.md
@@ -27,7 +27,7 @@ Or install all dev dependencies:
```bash
cd $SPARK_HOME
-pip install -r dev/requirements.txt
+pip install --group dev
```
## Quick Start
diff --git a/sql/connect/README.md b/sql/connect/README.md
index cc5ebcd9131d..27849fd308ea 100644
--- a/sql/connect/README.md
+++ b/sql/connect/README.md
@@ -28,7 +28,7 @@ Specifically, install `black` and dependencies from the
"Spark Connect python pr
```
-pip install -r dev/requirements.txt
+pip install --group dev
```
Install [buf](https://github.com/bufbuild/buf)
diff --git a/sql/connect/common/src/main/protobuf/README.md
b/sql/connect/common/src/main/protobuf/README.md
index 512359da258f..a3708a463e9d 100644
--- a/sql/connect/common/src/main/protobuf/README.md
+++ b/sql/connect/common/src/main/protobuf/README.md
@@ -41,14 +41,14 @@ Install the required tools:
- [`buf`](https://buf.build/docs/cli/installation/) — protobuf code generator
- Python 3.12+
-Install the required Python packages. Check `dev/requirements.txt` for the
latest
+Install the required Python packages. Check `pyproject.toml` for the latest
pinned versions of `mypy`, `mypy-protobuf`, and `black`, then run:
```bash
pip install 'mypy==<version>' 'mypy-protobuf==<version>' 'black==<version>'
```
-For example, based on the current `dev/requirements.txt`:
+For example, based on the current `pyproject.toml`:
```bash
pip install 'mypy==1.19.1' 'mypy-protobuf==3.3.0' 'black==26.3.1'
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
