This is an automated email from the ASF dual-hosted git repository.
philo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new aab550f35 [GLUTEN-6893][VL] Change to using native libs generated by
vcpkg build in Gluten scala tests (#6894)
aab550f35 is described below
commit aab550f353345bcf6c131b2d2fc62460d3521883
Author: PHILO-HE <[email protected]>
AuthorDate: Wed Aug 21 08:53:27 2024 +0800
[GLUTEN-6893][VL] Change to using native libs generated by vcpkg build in
Gluten scala tests (#6894)
---
.github/workflows/velox_docker.yml | 189 ++++++++++++++-------------------
dev/ci-velox-buildshared-centos-8.sh | 3 +-
dev/ci-velox-buildstatic-centos-7.sh | 6 +-
dev/vcpkg/ports/simdjson/vcpkg.json | 3 +-
docs/developers/UsingGperftoolsInCH.md | 7 ++
docs/developers/UsingJemallocWithCH.md | 7 ++
docs/developers/VeloxUDF.md | 7 ++
docs/developers/velox-backend-CI.md | 24 +++++
8 files changed, 129 insertions(+), 117 deletions(-)
diff --git a/.github/workflows/velox_docker.yml
b/.github/workflows/velox_docker.yml
index c310d5298..644fe62b9 100644
--- a/.github/workflows/velox_docker.yml
+++ b/.github/workflows/velox_docker.yml
@@ -68,17 +68,16 @@ jobs:
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
run: |
df -a
+ cd $GITHUB_WORKSPACE/
bash dev/ci-velox-buildstatic-centos-7.sh
- - name: Upload Artifact Native
- uses: actions/upload-artifact@v2
+ - uses: actions/upload-artifact@v2
with:
- path: ./cpp/build/releases/
name: velox-native-lib-centos-7-${{github.sha}}
- - name: Upload Artifact Arrow Jar
- uses: actions/upload-artifact@v2
+ path: ./cpp/build/releases/
+ - uses: actions/upload-artifact@v2
with:
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: velox-arrow-jar-centos-7-${{github.sha}}
run-tpc-test-ubuntu:
needs: build-native-lib-centos-7
@@ -118,7 +117,7 @@ jobs:
- name: Download All Arrow Jar Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-arrow-jar-centos-7-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Setup tzdata
run: |
@@ -189,7 +188,7 @@ jobs:
- name: Download All Arrow Jar Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-arrow-jar-centos-7-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -274,7 +273,7 @@ jobs:
- name: Download All Arrow Jar Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-arrow-jar-centos-7-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /home/runner/.m2/repository/org/apache/arrow/
- name: Setup java and maven
run: |
@@ -386,7 +385,7 @@ jobs:
- name: Download All Arrow Jar Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-arrow-jar-centos-7-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /home/runner/.m2/repository/org/apache/arrow/
- name: Setup java and maven
run: |
@@ -479,7 +478,7 @@ jobs:
- name: Download All Arrow Jar Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-arrow-jar-centos-7-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -546,7 +545,7 @@ jobs:
- name: Download All Arrow Jar Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-arrow-jar-centos-7-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Setup tzdata
run: |
@@ -583,7 +582,7 @@ jobs:
GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox-with-celeborn --benchmark-type=ds
--error-on-memleak --off-heap-size=10g -s=1.0 --threads=8 --iterations=1
- build-native-lib-centos-8:
+ run-CPP-test-UDF-test:
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
steps:
@@ -606,50 +605,37 @@ jobs:
run: |
df -a
bash dev/ci-velox-buildshared-centos-8.sh
- - uses: actions/upload-artifact@v2
- with:
- name: velox-native-lib-centos-8-${{github.sha}}
- path: ./cpp/build/releases/
- - uses: actions/upload-artifact@v2
- with:
- name: udf-example-lib-centos-8-${{github.sha}}
- path: ./cpp/build/velox/udf/examples/
- - uses: actions/upload-artifact@v2
- with:
- name: benchmark-centos-8-${{github.sha}}
- path: ./cpp/build/velox/benchmarks/
- - uses: actions/upload-artifact@v2
- with:
- name: arrow-jars-centos-8-${{github.sha}}
- path: /root/.m2/repository/org/apache/arrow/
+ - name: Run CPP unit test
+ run: |
+ cd ./cpp/build && ctest -V
+ - name: Run CPP benchmark test
+ run: |
+ $MVN_CMD test -Pspark-3.5 -Pbackends-velox -pl backends-velox -am \
+ -DtagsToInclude="org.apache.gluten.tags.GenerateExample" -Dtest=none
-DfailIfNoTests=false -Dexec.skip
+ # This test depends on example.json generated by the above mvn test.
+ cd cpp/build/velox/benchmarks && sudo chmod +x ./generic_benchmark
+ ./generic_benchmark --run-example --with-shuffle --threads 1
--iterations 1
+ - name: Run UDF test
+ run: |
+ # Depends on --build_example=ON.
+ $MVN_CMD test -Pspark-3.5 -Pbackends-velox -Piceberg -Pdelta
-DtagsToExclude=None \
+ -DtagsToInclude=org.apache.gluten.tags.UDFTest
run-spark-test-spark32:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
CCACHE_DIR: "${{ github.workspace }}/.ccache"
steps:
- uses: actions/checkout@v2
- - name: Download All Artifacts
- uses: actions/download-artifact@v2
+ - uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- - name: Download UDF Example Lib
- uses: actions/download-artifact@v2
+ - uses: actions/download-artifact@v2
with:
- name: udf-example-lib-centos-8-${{github.sha}}
- path: ./cpp/build/velox/udf/examples/
- - name: Download Benchmark
- uses: actions/download-artifact@v2
- with:
- name: benchmark-centos-8-${{github.sha}}
- path: ./cpp/build/velox/benchmarks/
- - name: Download Arrow Jars
- uses: actions/download-artifact@v2
- with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -693,23 +679,18 @@ jobs:
run: |
cd $GITHUB_WORKSPACE/
export SPARK_SCALA_VERSION=2.12
- $MVN_CMD clean install -Pspark-3.2 -Pspark-ut -Pbackends-velox
-Pceleborn -Piceberg -Pdelta
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark32/spark_home/"
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
&& \
- $MVN_CMD test -Pspark-3.2 -Pbackends-velox -Piceberg -Pdelta
-DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest
+ $MVN_CMD clean install -Pspark-3.2 -Pspark-ut -Pbackends-velox
-Pceleborn -Piceberg \
+ -Pdelta
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark32/spark_home/" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
- name: Upload golden files
if: failure()
uses: actions/upload-artifact@v4
with:
name: golden-files-spark32
path: /tmp/tpch-approved-plan/**
- - name: Gluten CPP Benchmark Test
- run: |
- # This test depends on example.json generated by the above mvn test.
- cd $GITHUB_WORKSPACE/cpp/build/velox/benchmarks && \
- sudo chmod +x ./generic_benchmark && \
- ./generic_benchmark --run-example --with-shuffle --threads 1
--iterations 1
run-spark-test-spark32-slow:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -719,12 +700,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -759,7 +740,7 @@ jobs:
$MVN_CMD clean install -Pspark-3.2 -Pspark-ut -Pbackends-velox
-Pceleborn -Piceberg -Pdelta
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark32/spark_home/"
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
run-spark-test-spark33:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -769,17 +750,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- - name: Download UDF Example Lib
- uses: actions/download-artifact@v2
- with:
- name: udf-example-lib-centos-8-${{github.sha}}
- path: ./cpp/build/velox/udf/examples/
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -823,8 +799,9 @@ jobs:
run: |
cd $GITHUB_WORKSPACE/
export SPARK_SCALA_VERSION=2.12
- $MVN_CMD clean install -Pspark-3.3 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark33/spark_home/"
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
&& \
- $MVN_CMD test -Pspark-3.3 -Pbackends-velox -Piceberg -Pdelta
-DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest
+ $MVN_CMD clean install -Pspark-3.3 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut \
+
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark33/spark_home/" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
- name: Upload golden files
if: failure()
uses: actions/upload-artifact@v4
@@ -834,7 +811,7 @@ jobs:
run-spark-test-spark33-slow:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -844,12 +821,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -881,10 +858,12 @@ jobs:
- name: Build and Run unit test for Spark 3.3.1 (slow tests)
run: |
cd $GITHUB_WORKSPACE/
- $MVN_CMD clean install -Pspark-3.3 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark33/spark_home/"
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
+ $MVN_CMD clean install -Pspark-3.3 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut \
+
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark33/spark_home/" \
+ -DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
run-spark-test-spark34:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -894,17 +873,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- - name: Download UDF Example Lib
- uses: actions/download-artifact@v2
- with:
- name: udf-example-lib-centos-8-${{github.sha}}
- path: ./cpp/build/velox/udf/examples/
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -948,8 +922,9 @@ jobs:
run: |
cd $GITHUB_WORKSPACE/
export SPARK_SCALA_VERSION=2.12
- $MVN_CMD clean install -Pspark-3.4 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark34/spark_home/"
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
&& \
- $MVN_CMD test -Pspark-3.4 -Pbackends-velox -Piceberg -Pdelta
-DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest
+ $MVN_CMD clean install -Pspark-3.4 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut \
+
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark34/spark_home/" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
- name: Upload golden files
if: failure()
uses: actions/upload-artifact@v4
@@ -959,7 +934,7 @@ jobs:
run-spark-test-spark34-slow:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -969,12 +944,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -1006,10 +981,12 @@ jobs:
- name: Build and Run unit test for Spark 3.4.2 (slow tests)
run: |
cd $GITHUB_WORKSPACE/
- $MVN_CMD clean install -Pspark-3.4 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark34/spark_home/"
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
+ $MVN_CMD clean install -Pspark-3.4 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut \
+
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark34/spark_home/" \
+ -DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
run-spark-test-spark35:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -1019,17 +996,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- - name: Download UDF Example Lib
- uses: actions/download-artifact@v2
- with:
- name: udf-example-lib-centos-8-${{github.sha}}
- path: ./cpp/build/velox/udf/examples/
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -1073,8 +1045,9 @@ jobs:
run: |
cd $GITHUB_WORKSPACE/
export SPARK_SCALA_VERSION=2.12
- $MVN_CMD clean install -Pspark-3.5 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/"
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
&& \
- $MVN_CMD test -Pspark-3.5 -Pbackends-velox -Piceberg -Pdelta
-DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest
+ $MVN_CMD clean install -Pspark-3.5 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut \
+
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
- name: Upload golden files
if: failure()
uses: actions/upload-artifact@v4
@@ -1083,7 +1056,7 @@ jobs:
path: /tmp/tpch-approved-plan/**
run-spark-test-spark35-scala213:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -1093,17 +1066,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- - name: Download UDF Example Lib
- uses: actions/download-artifact@v2
- with:
- name: udf-example-lib-centos-8-${{github.sha}}
- path: ./cpp/build/velox/udf/examples/
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -1147,11 +1115,12 @@ jobs:
run: |
cd $GITHUB_WORKSPACE/
export SPARK_SCALA_VERSION=2.13
- $MVN_CMD clean install -Pspark-3.5 -Pscala-2.13 -Pbackends-velox
-Pceleborn -Piceberg -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/"
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
&& \
- $MVN_CMD test -Pspark-3.5 -Pscala-2.13 -Pbackends-velox -Piceberg
-Pdelta -DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest
+ $MVN_CMD clean install -Pspark-3.5 -Pscala-2.13 -Pbackends-velox
-Pceleborn -Piceberg \
+ -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/" \
+
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags
run-spark-test-spark35-slow:
- needs: build-native-lib-centos-8
+ needs: build-native-lib-centos-7
runs-on: ubuntu-20.04
container: ghcr.io/facebookincubator/velox-dev:centos8
env:
@@ -1161,12 +1130,12 @@ jobs:
- name: Download All Artifacts
uses: actions/download-artifact@v2
with:
- name: velox-native-lib-centos-8-${{github.sha}}
+ name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/releases
- name: Download Arrow Jars
uses: actions/download-artifact@v2
with:
- name: arrow-jars-centos-8-${{github.sha}}
+ name: arrow-jars-centos-7-${{github.sha}}
path: /root/.m2/repository/org/apache/arrow/
- name: Update mirror list
run: |
@@ -1204,4 +1173,6 @@ jobs:
- name: Build and Run unit test for Spark 3.5.1 (slow tests)
run: |
cd $GITHUB_WORKSPACE/
- $MVN_CMD clean install -Pspark-3.5 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/"
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
+ $MVN_CMD clean install -Pspark-3.5 -Pbackends-velox -Pceleborn
-Piceberg -Pdelta -Pspark-ut \
+
-DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/" \
+ -DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
diff --git a/dev/ci-velox-buildshared-centos-8.sh
b/dev/ci-velox-buildshared-centos-8.sh
index 362900bd0..f337185de 100755
--- a/dev/ci-velox-buildshared-centos-8.sh
+++ b/dev/ci-velox-buildshared-centos-8.sh
@@ -10,9 +10,8 @@ yum install sudo patch java-1.8.0-openjdk-devel wget -y
wget
https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
tar -xvf apache-maven-3.8.8-bin.tar.gz && mv apache-maven-3.8.8 /usr/lib/maven
export PATH="${PATH}:/usr/lib/maven/bin"
+echo "PATH=${PATH}:/usr/lib/maven/bin" >> $GITHUB_ENV
source /opt/rh/gcc-toolset-9/enable
./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_ep_cache=OFF
--build_tests=ON \
--build_examples=ON --build_benchmarks=ON --build_protobuf=ON
-
-cd ./cpp/build && ctest -V
diff --git a/dev/ci-velox-buildstatic-centos-7.sh
b/dev/ci-velox-buildstatic-centos-7.sh
index d83d443fc..3272de95d 100755
--- a/dev/ci-velox-buildstatic-centos-7.sh
+++ b/dev/ci-velox-buildstatic-centos-7.sh
@@ -3,8 +3,6 @@
set -e
source /opt/rh/devtoolset-9/enable
-cd $GITHUB_WORKSPACE/
-source ./dev/vcpkg/env.sh
export NUM_THREADS=4
-./dev/builddeps-veloxbe.sh --build_tests=OFF --build_benchmarks=OFF
--build_arrow=OFF --enable_s3=ON \
- --enable_gcs=ON --enable_hdfs=ON --enable_abfs=ON
+./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF
--build_tests=OFF --build_benchmarks=OFF \
+ --build_examples=OFF --enable_s3=ON --enable_gcs=ON
--enable_hdfs=ON --enable_abfs=ON
diff --git a/dev/vcpkg/ports/simdjson/vcpkg.json
b/dev/vcpkg/ports/simdjson/vcpkg.json
index 6e46382e4..2b74be554 100644
--- a/dev/vcpkg/ports/simdjson/vcpkg.json
+++ b/dev/vcpkg/ports/simdjson/vcpkg.json
@@ -17,8 +17,7 @@
"default-features": [
"deprecated",
"exceptions",
- "threads",
- "utf8-validation"
+ "threads"
],
"features": {
"deprecated": {
diff --git a/docs/developers/UsingGperftoolsInCH.md
b/docs/developers/UsingGperftoolsInCH.md
index f0d5c720b..5a4bbea3f 100644
--- a/docs/developers/UsingGperftoolsInCH.md
+++ b/docs/developers/UsingGperftoolsInCH.md
@@ -1,3 +1,10 @@
+---
+layout: page
+title: Debug for CH Backend with gpertools Tool
+nav_order: 11
+has_children: true
+parent: /developer-overview/
+---
We need using gpertools to find the memory or CPU issue. That's what this
document is about.
## Install gperftools
diff --git a/docs/developers/UsingJemallocWithCH.md
b/docs/developers/UsingJemallocWithCH.md
index 626f7522d..365a35dd3 100644
--- a/docs/developers/UsingJemallocWithCH.md
+++ b/docs/developers/UsingJemallocWithCH.md
@@ -1,3 +1,10 @@
+---
+layout: page
+title: Use Jemalloc for CH Backend
+nav_order: 12
+has_children: true
+parent: /developer-overview/
+---
We need using jemalloc to find the memory issue. That's what this document is
about.
## Change code of jemalloc
diff --git a/docs/developers/VeloxUDF.md b/docs/developers/VeloxUDF.md
index 25b896929..6872f2d0c 100644
--- a/docs/developers/VeloxUDF.md
+++ b/docs/developers/VeloxUDF.md
@@ -1,3 +1,10 @@
+---
+layout: page
+title: Velox UDF and UDAF
+nav_order: 13
+has_children: true
+parent: /developer-overview/
+---
# Velox User-Defined Functions (UDF) and User-Defined Aggregate Functions
(UDAF)
## Introduction
diff --git a/docs/developers/velox-backend-CI.md
b/docs/developers/velox-backend-CI.md
new file mode 100644
index 000000000..e4dceffdf
--- /dev/null
+++ b/docs/developers/velox-backend-CI.md
@@ -0,0 +1,24 @@
+---
+layout: page
+title: Velox Backend CI
+nav_order: 14
+parent: Developer Overview
+---
+# Velox Backend CI
+
+GHA workflows are defined under `.github/workflows/`.
+
+## Docker Build
+We have a weekly job to build a docker based on `Dockerfile.gha` for CI
verification, defined in `docker_image.yml`.
+
+## Vcpkg Caching
+Gluten main branch is pulled down during docker build. And vcpkg will cache
binary data of all dependencies defined under dev/vcpkg.
+These binary data is cached into `/var/cache/vcpkg` and CI job can re-use them
in new build. By setting `VCPKG_BINARY_SOURCES=clear` in env.,
+reusing vcpkg cache can be disabled.
+
+## Arrow Libs Pre-installation
+Arrow libs are pre-installed in docker, assuming they are not actively
changed, then not necessarily to be re-built every time.
+
+## Updating Docker Image
+Two GitHub secrets `DOCKERHUB_USER` & `DOCKERHUB_TOKEN` can be used to push
docker image to docker hub: https://hub.docker.com/r/apache/gluten/tags.
+Note GitHub secrets are not retrievable in PR from forked repo.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
