felipepessoto opened a new pull request, #13578:
URL: https://github.com/apache/gluten/pull/13578

   <!--
   Thank you for submitting a pull request! Here are some tips:
   
   1. For first-time contributors, please read our contributing guide:
      https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
   2. If necessary, create a GitHub issue for discussion beforehand to avoid 
duplicate work.
   3. If the PR is specific to a single backend, include [VL] or [CH] in the PR 
title to indicate the
      Velox or ClickHouse backend, respectively.
   4. If the PR is not ready for review, please mark it as a draft.
   -->
   
   ## What changes are proposed in this pull request?
   
   <!--
   Provide a clear and concise description of the changes introduced in this PR.
   Ensure the PR description aligns with the code changes, especially after 
updates.
   If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the 
corresponding issue
   when the PR is merged.
   -->
   
   Relates to #12743.
   
   Remove `-Dspark.default.parallelism=1` from the Delta Spark UT JVM defaults 
and document why the upstream context defaults must be preserved. Do not 
replace it with a globally forced value of 2: each Delta test context should 
retain its own Spark default.
   
   The two-row nontrivial-projection temp-view UPDATE/DELETE fixtures are 
sensitive to their input file layout and Delta's DML statistics-based file 
pruning. The parallelism-1 override puts both rows in one file, while the 
upstream two-core test session produces two one-row files. Controlled Spark 4.1 
/ Delta 4.4.1 runs reproduce the historical mismatches without Gluten as well 
as with Gluten under the one-file setup, and restore the four representative 
SQL/Dataset-view assertions with the two-file setup. The exact upstream Spark 
4.1 jobs also execute and pass those cases.
   
   Keep the Gluten plugin, columnar shuffle manager, off-heap settings, 
JVM/Netty flags, native Delta write setting, caller override precedence, and 
failure-gate settings unchanged. No production implementation, upstream test 
patch, expected answer, or baseline entry is changed.
   
   **This is a draft for full CI validation.** The baseline intentionally 
remains intact so the run can identify the exact now-passing and regression 
sets. `fail_on_fixed=true` can therefore make the gate fail on newly passing 
baseline entries; those entries should only be removed after the complete run 
is reviewed. The other generated variants have not all been proven fixed by the 
narrow reproduction.
   
   ## How was this patch tested?
   
   <!--
   Describe how the changes were tested, if applicable.
   Include new tests to validate the functionality, if necessary.
   For UI-related changes, attach screenshots to demonstrate the updates.
   -->
   
   - Checked the Linux-exported staged shell script with `bash -n` and JDK 17 
smoke probes: unset and empty `JAVA_TOOL_OPTIONS` no longer impose 
`spark.default.parallelism`; explicit caller values of 1 and 2 still override 
correctly, including repeated sourcing. The probes also confirm the required 
plugin, shuffle, off-heap, shuffle-partition and native-write properties remain 
unchanged.
   - Ran the repository license-header checker (`header main --fix`) and Git 
whitespace checks; no additional changes were needed. The diff is limited to 
the JVM defaults and their README.
   - Before this patch, controlled native/vanilla runs with parallelism 1 
versus 2 measured the initial live-file statistics and DML candidate paths. 
Both matched-configuration comparisons had 112 equal observations, including 
affected-row counts, and all 48 native Delta/Parquet/in-memory read controls 
matched. Disabling data skipping with two files restored the mismatches, 
isolating the fixture/pruning interaction rather than a native projection 
defect.
   - The full eight-shard Delta Spark UT workflow is pending on this draft; its 
outcome is not claimed by the local checks.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   <!--
   If generative AI tooling has been used in the process of authoring this 
patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling 
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   
   Generated-by: GitHub Copilot 1.0.102-0.unstable.r40949665530.gab1eef5 
(GPT-6.1 Sol)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to