This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5057e0d56104 [SPARK-57981][PYTHON][TEST] Move packaging test with
test-install
5057e0d56104 is described below
commit 5057e0d56104adeafa648e59e18c736d9c326ea8
Author: Tian Gao <[email protected]>
AuthorDate: Tue Jul 7 11:57:05 2026 -0700
[SPARK-57981][PYTHON][TEST] Move packaging test with test-install
### What changes were proposed in this pull request?
1. We have a python packaging test which used to run with
`pyspark-pipelines`. We move it to run with `pyspark-install`.
2. A few month ago, the packaging test executed with `pyspark-errors` so
there's some legacy code to trigger `pyspark-errors` for any changes in
`python` - that was removed.
3. We don't trigger `pyspark` tests anymore for `pyspark-install`.
### Why are the changes needed?
Python packaging test takes about 5 minutes. It's a low risk test which
does not really test any functionality. Running it for every commit is not
necessary. With this change, this test will run pre-merge for each PR, and in
scheduled CI if it's explicitly specified (`build_coverage` for now). This will
save us some minutes for workers.
Also, it's very natural to run packaging tests with installation tests -
running it with either pipeline or error is just very arbitrary.
More importantly, the current CI setup will run all pyspark tests if any of
the pyspark test needs to run. With the existing dependency, all pyspark tests
will run if anything changes in `python/`, including `python/docs` or
`python/packaging`, which is unnecessary.
After this change, we can skip post-merge pyspark tests if the change is
not in python/pyspark, which will save us some minutes too.
### Does this PR introduce _any_ user-facing change?
No, this is CI only.
### How was this patch tested?
CI should confirm that packaging test does not run with `pyspark-pipelines`
anymore.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57060 from gaogaotiantian/move-packaging-test.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
.github/workflows/build_and_test.yml | 4 ++--
dev/sparktestsupport/modules.py | 13 ++++++-------
dev/sparktestsupport/utils.py | 4 ++--
3 files changed, 10 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index b117b1692996..b0ae21f9801b 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -100,7 +100,7 @@ jobs:
id: set-outputs
run: |
if [ -z "${{ inputs.jobs }}" ]; then
- pyspark_modules=`cd dev && python -c "import
sparktestsupport.modules as m; print(','.join(m.name for m in m.all_modules if
m.name.startswith('pyspark') and not m.name.startswith('pyspark-pandas')))"`
+ pyspark_modules=`cd dev && python -c "import
sparktestsupport.modules as m; print(','.join(m.name for m in m.all_modules if
m.name.startswith('pyspark') and not m.name.startswith('pyspark-pandas') and
not m.name == 'pyspark-install'))"`
pyspark_pandas_modules=`cd dev && python -c "import
sparktestsupport.modules as m; print(','.join(m.name for m in m.all_modules if
m.name.startswith('pyspark-pandas')))"`
pyspark=`./dev/is-changed.py -m $pyspark_modules`
pandas=`./dev/is-changed.py -m $pyspark_pandas_modules`
@@ -792,7 +792,7 @@ jobs:
export SKIP_SCALA_BUILD=true
echo "Reusing precompiled artifact, skipping local SBT build."
fi
- if [[ "$MODULES_TO_TEST" == *"pyspark-pipelines"* ]]; then
+ if [[ "$MODULES_TO_TEST" == *"pyspark-install"* ]]; then
export SKIP_PACKAGING=false
echo "Python Packaging Tests Enabled!"
fi
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index f929a5d2c6b3..8f1395341884 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -817,8 +817,12 @@ pyspark_ml = Module(
pyspark_install = Module(
name="pyspark-install",
- dependencies=[core],
+ dependencies=[],
source_file_regexes=[
+ # Python package tests will be triggered with this module
+ # Any changes in python/ should trigger this module
+ # This module won't be executed for post-commit CIs so it's cheap
+ "python/",
"python/pyspark/install.py",
"python/pyspark/tests/test_install_spark.py",
],
@@ -1559,13 +1563,8 @@ pyspark_pandas_slow_connect = Module(
pyspark_errors = Module(
name="pyspark-errors",
- dependencies=[],
+ dependencies=[pyspark_core],
source_file_regexes=[
- # SPARK-44544: Force the execution of pyspark_errors when there are
any changes
- # in PySpark, since the Python Packaging Tests is only enabled within
this module.
- # This module is the smallest Python test module, it contains only 1
test file
- # and normally takes < 2 seconds, so the additional cost is small.
- "python/",
"python/pyspark/errors",
],
python_test_goals=[
diff --git a/dev/sparktestsupport/utils.py b/dev/sparktestsupport/utils.py
index fff418bdb348..029b1627bd0b 100755
--- a/dev/sparktestsupport/utils.py
+++ b/dev/sparktestsupport/utils.py
@@ -36,7 +36,7 @@ def determine_modules_for_files(filenames):
file to belong to the 'root' module. `.github` directory is counted only
in GitHub Actions.
>>> sorted(x.name for x in
determine_modules_for_files(["python/pyspark/a.py", "sql/core/foo"]))
- ['pyspark-core', 'pyspark-errors', 'sql']
+ ['pyspark-core', 'pyspark-install', 'sql']
>>> [x.name for x in
determine_modules_for_files(["file_not_matched_by_any_subproject"])]
['root']
"""
@@ -123,7 +123,7 @@ def determine_modules_to_test(changed_modules,
deduplicated=True):
... # doctest: +NORMALIZE_WHITESPACE
['avro', 'catalyst', 'connect', 'core', 'docker-integration-tests',
'examples', 'graphx',
'hive', 'hive-thriftserver', 'mllib', 'mllib-local', 'pipelines',
'protobuf',
- 'pyspark-connect', 'pyspark-core', 'pyspark-install', 'pyspark-ml',
'pyspark-ml-connect',
+ 'pyspark-connect', 'pyspark-core', 'pyspark-errors', 'pyspark-ml',
'pyspark-ml-connect',
'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-connect',
'pyspark-pandas-slow',
'pyspark-pandas-slow-connect', 'pyspark-pipelines', 'pyspark-resource',
'pyspark-sql',
'pyspark-streaming', 'pyspark-structured-streaming',
'pyspark-structured-streaming-connect',
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
