This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 748eaff4e21 [SPARK-44544][INFRA] Deduplicate
`run_python_packaging_tests`
748eaff4e21 is described below
commit 748eaff4e2177466dd746f6fbb82de8544bc7168
Author: Ruifeng Zheng <ruife...@apache.org>
AuthorDate: Wed Jul 26 15:52:38 2023 +0800
[SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`
### What changes were proposed in this pull request?
it seems that `run_python_packaging_tests` requires some disk space and
cause some pyspark modules fail, this PR is to make
`run_python_packaging_tests` only enabled within `pyspark-errors` (which is the
smallest pyspark test module)
### Why are the changes needed?
1, it seems it is the `run_python_packaging_tests` that cause the `No space
left` error;
2, the `run_python_packaging_tests` is tested in all `pyspark-*` test
modules, should be deduplicated;
### Does this PR introduce _any_ user-facing change?
no, infra-only
### How was this patch tested?
updated CI
Closes #42146 from zhengruifeng/infra_skip_py_packing_tests.
Authored-by: Ruifeng Zheng <ruife...@apache.org>
Signed-off-by: Ruifeng Zheng <ruife...@apache.org>
---
.github/workflows/build_and_test.yml | 16 ++++++++++++++--
dev/run-tests.py | 2 +-
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 7107af66129..02b3814a018 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -205,6 +205,7 @@ jobs:
HIVE_PROFILE: ${{ matrix.hive }}
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
+ SKIP_PACKAGING: true
steps:
- name: Checkout Spark repository
uses: actions/checkout@v3
@@ -344,6 +345,8 @@ jobs:
java:
- ${{ inputs.java }}
modules:
+ - >-
+ pyspark-errors
- >-
pyspark-sql, pyspark-mllib, pyspark-resource, pyspark-testing
- >-
@@ -353,7 +356,7 @@ jobs:
- >-
pyspark-pandas-slow
- >-
- pyspark-connect, pyspark-errors
+ pyspark-connect
- >-
pyspark-pandas-connect
- >-
@@ -366,6 +369,7 @@ jobs:
SPARK_LOCAL_IP: localhost
SKIP_UNIDOC: true
SKIP_MIMA: true
+ SKIP_PACKAGING: true
METASPACE_SIZE: 1g
steps:
- name: Checkout Spark repository
@@ -414,14 +418,20 @@ jobs:
python3.9 -m pip list
pypy3 -m pip list
- name: Install Conda for pip packaging test
+ if: ${{ matrix.modules == 'pyspark-errors' }}
run: |
curl -s
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh >
miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
run: |
- export PATH=$PATH:$HOME/miniconda/bin
+ if [[ "$MODULES_TO_TEST" == "pyspark-errors" ]]; then
+ export PATH=$PATH:$HOME/miniconda/bin
+ export SKIP_PACKAGING=false
+ echo "Python Packaging Tests Enabled!"
+ fi
./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST"
- name: Upload coverage to Codecov
if: fromJSON(inputs.envs).PYSPARK_CODECOV == 'true'
@@ -457,6 +467,7 @@ jobs:
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
SKIP_MIMA: true
+ SKIP_PACKAGING: true
steps:
- name: Checkout Spark repository
uses: actions/checkout@v3
@@ -911,6 +922,7 @@ jobs:
SPARK_LOCAL_IP: localhost
ORACLE_DOCKER_IMAGE_NAME: gvenzl/oracle-xe:21.3.0
SKIP_MIMA: true
+ SKIP_PACKAGING: true
steps:
- name: Checkout Spark repository
uses: actions/checkout@v3
diff --git a/dev/run-tests.py b/dev/run-tests.py
index c0c281b549e..9bf3095edb7 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -395,7 +395,7 @@ def run_python_tests(test_modules, parallelism,
with_coverage=False):
def run_python_packaging_tests():
- if not os.environ.get("SPARK_JENKINS"):
+ if not os.environ.get("SPARK_JENKINS") and
os.environ.get("SKIP_PACKAGING", "false") != "true":
set_title_and_block("Running PySpark packaging tests",
"BLOCK_PYSPARK_PIP_TESTS")
command = [os.path.join(SPARK_HOME, "dev", "run-pip-tests")]
run_cmd(command)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
