Copilot commented on code in PR #12529:
URL: https://github.com/apache/gluten/pull/12529#discussion_r3592531647
##########
.github/workflows/velox_weekly.yml:
##########
@@ -67,6 +67,99 @@ jobs:
git fetch origin ${{ github.ref }}:pr_branch && git checkout
pr_branch
fi
./dev/package.sh --spark_version=3.5
+
+ spark-test-spark35-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
Review Comment:
`$MVN_CMD` is used later in this job but is never defined in this
workflow/job env, so the Maven commands will fail with an empty command. Define
`MVN_CMD` (consistent with other Velox workflows) before invoking it.
##########
.github/workflows/velox_weekly.yml:
##########
@@ -67,6 +67,99 @@ jobs:
git fetch origin ${{ github.ref }}:pr_branch && git checkout
pr_branch
fi
./dev/package.sh --spark_version=3.5
+
+ spark-test-spark35-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
Review Comment:
This job downloads artifact `velox-native-lib-centos-7-${{ github.sha }}`,
but `build-on-centos-7` does not upload any artifact with that name (and CentOS
7 container jobs generally can’t run `upload-artifact@v4` due to the
Node20/glibc requirement). As written, the download step will fail and block
the job.
##########
.github/workflows/velox_weekly.yml:
##########
@@ -67,6 +67,99 @@ jobs:
git fetch origin ${{ github.ref }}:pr_branch && git checkout
pr_branch
fi
./dev/package.sh --spark_version=3.5
+
+ spark-test-spark35-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare
+ run: |
+ dnf module -y install python39 && \
+ alternatives --set python3 /usr/bin/python3.9 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 cython && \
+ pip3 install pandas==2.2.3 pyarrow==20.0.0
+ - name: Build and Run unit test for Spark 3.5.5 (other tests)
+ run: |
+ cd $GITHUB_WORKSPACE/
+ export SPARK_SCALA_VERSION=2.12
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Ppaimon -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/
-Dspark.gluten.sql.columnar.forceShuffledHashJoin=false" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.SkipTest
+ - name: Upload test report
+ uses: actions/upload-artifact@v4
Review Comment:
Test reports are only uploaded on success by default. If the tests fail (the
main case where reports are useful), this step will be skipped. Consider
uploading the surefire reports with `if: always()` (this is how other Velox
workflows do it).
##########
.github/workflows/velox_weekly.yml:
##########
@@ -67,6 +67,99 @@ jobs:
git fetch origin ${{ github.ref }}:pr_branch && git checkout
pr_branch
fi
./dev/package.sh --spark_version=3.5
+
+ spark-test-spark35-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare
+ run: |
+ dnf module -y install python39 && \
+ alternatives --set python3 /usr/bin/python3.9 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 cython && \
+ pip3 install pandas==2.2.3 pyarrow==20.0.0
+ - name: Build and Run unit test for Spark 3.5.5 (other tests)
+ run: |
+ cd $GITHUB_WORKSPACE/
+ export SPARK_SCALA_VERSION=2.12
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Ppaimon -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/
-Dspark.gluten.sql.columnar.forceShuffledHashJoin=false" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.SkipTest
+ - name: Upload test report
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-report
+ path: '**/surefire-reports/TEST-*.xml'
+ - name: Upload unit tests log files
+ if: ${{ !success() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-test-log
+ path: |
+ **/target/*.log
+ **/gluten-ut/**/hs_err_*.log
+ **/gluten-ut/**/core.*
+
+ spark-test-spark35-slow-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ LANG: C.UTF-8 #TODO remove after image update
+ LC_ALL: C.UTF-8
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
Review Comment:
This job downloads artifact `velox-native-lib-centos-7-${{ github.sha }}`,
but `build-on-centos-7` does not upload any artifact with that name (and CentOS
7 container jobs generally can’t run `upload-artifact@v4` due to the
Node20/glibc requirement). As written, the download step will fail and block
the job.
##########
.github/workflows/velox_weekly.yml:
##########
@@ -67,6 +67,99 @@ jobs:
git fetch origin ${{ github.ref }}:pr_branch && git checkout
pr_branch
fi
./dev/package.sh --spark_version=3.5
+
+ spark-test-spark35-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare
+ run: |
+ dnf module -y install python39 && \
+ alternatives --set python3 /usr/bin/python3.9 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 cython && \
+ pip3 install pandas==2.2.3 pyarrow==20.0.0
+ - name: Build and Run unit test for Spark 3.5.5 (other tests)
+ run: |
+ cd $GITHUB_WORKSPACE/
+ export SPARK_SCALA_VERSION=2.12
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Ppaimon -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/
-Dspark.gluten.sql.columnar.forceShuffledHashJoin=false" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.SkipTest
+ - name: Upload test report
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-report
+ path: '**/surefire-reports/TEST-*.xml'
+ - name: Upload unit tests log files
+ if: ${{ !success() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-test-log
+ path: |
+ **/target/*.log
+ **/gluten-ut/**/hs_err_*.log
+ **/gluten-ut/**/core.*
+
+ spark-test-spark35-slow-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ LANG: C.UTF-8 #TODO remove after image update
+ LC_ALL: C.UTF-8
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare Spark Resources for Spark 3.5.5 #TODO remove after image
update
+ run: |
+ rm -rf /opt/shims/spark35
+ bash .github/workflows/util/install-spark-resources.sh 3.5
+ - name: Build and Run unit test for Spark 3.5.5 (slow tests)
+ run: |
+ cd $GITHUB_WORKSPACE/
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Ppaimon -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/
-Dspark.gluten.sql.columnar.forceShuffledHashJoin=false" \
+ -DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
+ $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Ppaimon -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/
-Dspark.gluten.sql.columnar.forceShuffledHashJoin=false" \
+ -DtagsToInclude=org.apache.spark.tags.SlowHiveTest
+ - name: Upload test report
+ uses: actions/upload-artifact@v4
Review Comment:
Test reports are only uploaded on success by default. If the tests fail (the
main case where reports are useful), this step will be skipped. Consider
uploading the surefire reports with `if: always()` (this is how other Velox
workflows do it).
##########
.github/workflows/velox_weekly.yml:
##########
@@ -67,6 +67,99 @@ jobs:
git fetch origin ${{ github.ref }}:pr_branch && git checkout
pr_branch
fi
./dev/package.sh --spark_version=3.5
+
+ spark-test-spark35-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: velox-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ - name: Prepare
+ run: |
+ dnf module -y install python39 && \
+ alternatives --set python3 /usr/bin/python3.9 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 cython && \
+ pip3 install pandas==2.2.3 pyarrow==20.0.0
+ - name: Build and Run unit test for Spark 3.5.5 (other tests)
+ run: |
+ cd $GITHUB_WORKSPACE/
+ export SPARK_SCALA_VERSION=2.12
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Ppaimon -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/
-Dspark.gluten.sql.columnar.forceShuffledHashJoin=false" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.SkipTest
+ - name: Upload test report
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-report
+ path: '**/surefire-reports/TEST-*.xml'
+ - name: Upload unit tests log files
+ if: ${{ !success() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: ${{ github.job }}-test-log
+ path: |
+ **/target/*.log
+ **/gluten-ut/**/hs_err_*.log
+ **/gluten-ut/**/core.*
+
+ spark-test-spark35-slow-smj:
+ needs: build-on-centos-7
+ runs-on: ubuntu-22.04
+ env:
+ SPARK_TESTING: true
+ LANG: C.UTF-8 #TODO remove after image update
+ LC_ALL: C.UTF-8
Review Comment:
`$MVN_CMD` is used later in this job but is never defined in this
workflow/job env, so the Maven commands will fail with an empty command. Define
`MVN_CMD` (consistent with other Velox workflows) before invoking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
