This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push: new f8d0688 [SPARK-35393][PYTHON][INFRA][TESTS] Recover pip packaging test in Github Actions f8d0688 is described below commit f8d06883853ff711d5d20e895a5eecfb87bd0131 Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Thu May 13 10:35:56 2021 -0700 [SPARK-35393][PYTHON][INFRA][TESTS] Recover pip packaging test in Github Actions Currently pip packaging test is being skipped: ``` ======================================================================== Running PySpark packaging tests ======================================================================== Constructing virtual env for testing Missing virtualenv & conda, skipping pip installability tests Cleaning up temporary directory - /tmp/tmp.iILYWISPXW ``` See https://github.com/apache/spark/runs/2568923639?check_suite_focus=true GitHub Actions's image has its default Conda installed at `/usr/share/miniconda` but seems like the image we're using for PySpark does not have it (which is legitimate). This PR proposes to install Conda to use in pip packaging tests in GitHub Actions. To recover the test coverage. No, dev-only. It was tested in my fork: https://github.com/HyukjinKwon/spark/runs/2575126882?check_suite_focus=true ``` ======================================================================== Running PySpark packaging tests ======================================================================== Constructing virtual env for testing Using conda virtual environments Testing pip installation with python 3.6 Using /tmp/tmp.qPjTenqfGn for virtualenv Collecting package metadata (current_repodata.json): ...working... done Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done environment location: /tmp/tmp.qPjTenqfGn/3.6 added / updated specs: - numpy - pandas - pip - python=3.6 - setuptools ... Successfully ran pip sanity check ``` Closes #32537 from HyukjinKwon/SPARK-35393. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Dongjoon Hyun <dh...@apple.com> (cherry picked from commit 7d371d27f2a974b682ffa16b71576e61e9338c34) Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- .github/workflows/build_and_test.yml | 11 ++++++----- dev/run-pip-tests | 4 ---- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 8749499..936a256 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -76,8 +76,6 @@ jobs: INCLUDED_TAGS: ${{ matrix.included-tags }} HADOOP_PROFILE: ${{ matrix.hadoop }} HIVE_PROFILE: ${{ matrix.hive }} - # GitHub Actions' default miniconda to use in pip packaging test. - CONDA_PREFIX: /usr/share/miniconda GITHUB_PREV_SHA: ${{ github.event.before }} SPARK_LOCAL_IP: localhost steps: @@ -160,8 +158,6 @@ jobs: MODULES_TO_TEST: ${{ matrix.modules }} HADOOP_PROFILE: hadoop2.7 HIVE_PROFILE: hive2.3 - # GitHub Actions' default miniconda to use in pip packaging test. - CONDA_PREFIX: /usr/share/miniconda GITHUB_PREV_SHA: ${{ github.event.before }} SPARK_LOCAL_IP: localhost steps: @@ -195,13 +191,18 @@ jobs: with: python-version: 2.7 architecture: x64 - - name: Install Python packages (Python 2.7 ) + - name: Install Python packages (Python 2.7) run: | python2.7 -m pip install numpy 'pyarrow<3.0.0' pandas scipy xmlrunner python2.7 -m pip list + - name: Install Conda for pip packaging test + run: | + curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh + bash miniconda.sh -b -p $HOME/miniconda # Run the tests. - name: Run tests run: | + export PATH=$PATH:$HOME/miniconda/bin ./dev/run-tests --parallelism 2 --modules "$MODULES_TO_TEST" - name: Upload test results to report if: always() diff --git a/dev/run-pip-tests b/dev/run-pip-tests index b322d3f..cb64e88 100755 --- a/dev/run-pip-tests +++ b/dev/run-pip-tests @@ -80,10 +80,6 @@ for python in "${PYTHON_EXECS[@]}"; do VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python rm -rf "$VIRTUALENV_PATH" if [ -n "$USE_CONDA" ]; then - if [ -f "$CONDA_PREFIX/etc/profile.d/conda.sh" ]; then - # See also https://github.com/conda/conda/issues/7980 - source "$CONDA_PREFIX/etc/profile.d/conda.sh" - fi conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools source activate "$VIRTUALENV_PATH" || conda activate "$VIRTUALENV_PATH" else --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org