weiting-chen commented on code in PR #13520:
URL: https://github.com/apache/gluten/pull/13520#discussion_r4641250464
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -1669,3 +1686,150 @@ jobs:
**/target/*.log
**/gluten-ut/**/hs_err_*.log
**/gluten-ut/**/core.*
+
+ spark-test-spark43:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims43 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ # Split tests into 3 groups to run in parallel and cut the ~2h
wall-clock time.
+ # group1 – streaming tests
+ # group2 – execution / catalyst / errors / extension tests
+ # group3 – top-level sql, connector, sources, hive and remaining
tests
+ group: [1, 2, 3]
+ env:
+ SPARK_TESTING: true
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v7
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v8
+ with:
+ name: velox-native-lib-centos-8-${{github.sha}}
+ path: ./cpp/build/releases/
+ - name: Prepare
+ run: |
+ dnf install -y python3.11 python3.11-pip python3.11-devel && \
+ ls -la /usr/bin/python3.11 && \
+ alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11
1 && \
+ alternatives --set python3 /usr/bin/python3.11 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 cython && \
+ pip3 install pandas==2.2.3 pyarrow==20.0.0
+ - name: Prepare Spark 4.3 resources
+ run: |
+ # TODO: Remove once the CI images provide /opt/shims/spark43.
+ if [ ! -d /opt/shims/spark43/spark_home ]; then
+ bash .github/workflows/util/install-spark-resources.sh 4.3
+ fi
+ - name: Build and Run unit test for Spark 4.3.0 with scala-2.13 (other
tests, group ${{ matrix.group }})
+ run: |
+ cd $GITHUB_WORKSPACE/
+ export SPARK_SCALA_VERSION=2.13
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+
TAGS_EXCLUDE="org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest"
+ # Install all modules to the local repo first so gluten-ut resolves
its
+ # sibling Gluten modules (gluten-substrait etc.) before testing.
+ $MVN_CMD clean install -Pspark-4.3 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -DskipTests
+ if [ "${{ matrix.group }}" = "1" ]; then
+ # Group 1: streaming + gluten utils + top-level spark tests (~55
classes)
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.3 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut \
+ -DargLine="-Dspark.test.home=/opt/shims/spark43/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
+
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
Review Comment:
### Run the fractional-CPU regression suite on Spark 4.3
**Problem:** None of the three new normal-group selectors includes
`org.apache.spark.util.SparkResourceUtilSuite`. The parallel runner filters
discovered suites with `name.startswith(w)`, and this untagged `AnyFunSuite`
does not execute in either slow-tag lane. The build/install steps use
`-DskipTests`. As a result, the new `0.5` and `0.3` fractional-CPU assertions
can regress without failing these Spark 4.3 jobs.
**Evidence:**
```text
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
```
The other two normal groups likewise omit the suite's
`org.apache.spark.util` package. See `SparkResourceUtilSuite.scala:64-72` and
`parallel-ut/run.py:159-160`. This is a focused regression-coverage gap, not
evidence that the current arithmetic is wrong.
**Suggested Fix:** Add this exact suite to one normal group, without
broadening selection to intentionally unsupported shuffle suites:
```text
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten,org.apache.spark.util.SparkResourceUtilSuite"
```
Verify that the test report records execution of `getTaskSlots floors
executor cores divided by fractional task cpus`.
AI-generated review finding.
<!-- pr-review-fingerprint:
prf-v1:8a210c7127f8c86ec40d7488c0eb6e50f20b0e83b202ef86b60f9fda1b765762 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]