felipepessoto opened a new pull request, #13578:
URL: https://github.com/apache/gluten/pull/13578
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
Relates to #12743.
Remove `-Dspark.default.parallelism=1` from the Delta Spark UT JVM defaults
and document why the upstream context defaults must be preserved. Do not
replace it with a globally forced value of 2: each Delta test context should
retain its own Spark default.
The two-row nontrivial-projection temp-view UPDATE/DELETE fixtures are
sensitive to their input file layout and Delta's DML statistics-based file
pruning. The parallelism-1 override puts both rows in one file, while the
upstream two-core test session produces two one-row files. Controlled Spark 4.1
/ Delta 4.4.1 runs reproduce the historical mismatches without Gluten as well
as with Gluten under the one-file setup, and restore the four representative
SQL/Dataset-view assertions with the two-file setup. The exact upstream Spark
4.1 jobs also execute and pass those cases.
Keep the Gluten plugin, columnar shuffle manager, off-heap settings,
JVM/Netty flags, native Delta write setting, caller override precedence, and
failure-gate settings unchanged. No production implementation, upstream test
patch, expected answer, or baseline entry is changed.
**This is a draft for full CI validation.** The baseline intentionally
remains intact so the run can identify the exact now-passing and regression
sets. `fail_on_fixed=true` can therefore make the gate fail on newly passing
baseline entries; those entries should only be removed after the complete run
is reviewed. The other generated variants have not all been proven fixed by the
narrow reproduction.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
- Checked the Linux-exported staged shell script with `bash -n` and JDK 17
smoke probes: unset and empty `JAVA_TOOL_OPTIONS` no longer impose
`spark.default.parallelism`; explicit caller values of 1 and 2 still override
correctly, including repeated sourcing. The probes also confirm the required
plugin, shuffle, off-heap, shuffle-partition and native-write properties remain
unchanged.
- Ran the repository license-header checker (`header main --fix`) and Git
whitespace checks; no additional changes were needed. The diff is limited to
the JVM defaults and their README.
- Before this patch, controlled native/vanilla runs with parallelism 1
versus 2 measured the initial live-file statistics and DML candidate paths.
Both matched-configuration comparisons had 112 equal observations, including
affected-row counts, and all 48 native Delta/Parquet/in-memory read controls
matched. Disabling data skipping with two files restored the mismatches,
isolating the fixture/pruning interaction rather than a native projection
defect.
- The full eight-shard Delta Spark UT workflow is pending on this draft; its
outcome is not claimed by the local checks.
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: GitHub Copilot 1.0.102-0.unstable.r40949665530.gab1eef5
(GPT-6.1 Sol)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]