Copilot commented on code in PR #12745: URL: https://github.com/apache/gluten/pull/12745#discussion_r3752152492
########## docs/developers/velox-backend-CI.md: ########## @@ -38,4 +38,23 @@ Since the Docker image is rebuilt weekly, the ccache is mostly outdated, so it i ## Updating the Docker Image The GitHub secrets `DOCKERHUB_USER` and `DOCKERHUB_TOKEN` are used to push Docker images to [Docker Hub](https://hub.docker.com/r/apache/gluten/tags). -Note that GitHub secrets are not accessible in PRs from forked repos. \ No newline at end of file +Note that GitHub secrets are not accessible in PRs from forked repos. + +## Delta Spark UT +`delta_spark_ut.yml` runs delta-io/delta's own `spark` test suite against a Gluten Velox bundle, +so Gluten is validated against a real Delta release. Many of those tests are expected to fail +today (Gluten does not offload every Delta code path), so the job does not gate on "any failure": +it compares each run against a committed baseline of known failures in +`.github/workflows/util/delta-spark-ut/known-failures.txt` and fails only on a **new** failure, +or on a baseline test that starts **passing** (which means the baseline needs updating). + +It runs per PR only when Delta-relevant paths change (`gluten-delta/**`, +`backends-velox/src-delta*/**`, or the pipeline's own files), nightly at 05:00 UTC for full +coverage, and on demand via `workflow_dispatch` -- use the manual run to check a Velox/core +change against Delta before merging. + +To refresh the baseline after fixing something, run the workflow with `update_baseline=true`, +download the `delta-spark-ut-known-failures` artifact and commit it. See +[.github/workflows/util/delta-spark-ut/README.md](https://github.com/apache/gluten/blob/main/.github/workflows/util/delta-spark-ut/README.md) Review Comment: Baseline refresh instructions are a bit ambiguous: the uploaded artifact contains `known-failures.txt`, but contributors still need to replace `.github/workflows/util/delta-spark-ut/known-failures.txt` in this repo with that downloaded file. As written, someone might commit the downloaded artifact file as-is without updating the committed baseline in-place. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
