felipepessoto opened a new pull request, #13508:
URL: https://github.com/apache/gluten/pull/13508
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
Remove 14 stale nested-map MERGE entries from
`.github/workflows/util/delta-spark-ut/known-failures.txt`, reducing the main
baseline from 485 to 471 entries. Existing comments and every other entry are
unchanged; no comments, production changes, test exclusions, or quarantine
entries are added.
This is a standalone cleanup against `main` at
`cce7716962d3152b46fc22882bed4a3887a792ef`, which already contains the merged
[transform_keys fix, #13490](https://github.com/apache/gluten/pull/13490)
(`1503e13b5fdd953fa46e20a91ee90d71c0494b22`). It is not stacked on
[#13502](https://github.com/apache/gluten/pull/13502) and does not copy that
PR's partition-materialization changes.
**Root cause:** MERGE schema evolution fills missing fields in struct map
keys with nulls. The old Presto `transform_keys` implementation rejected these
non-null struct keys with `map key cannot be indeterminate`, including
`{5,6,null}` and `{10,null,50}`. #13490 switches to Spark semantics, which
allow null fields inside a non-null struct key. The removed entries cover the
generated variants of:
- `schema evolution - new source column in map struct key`
- `schema evolution - source nested map struct key contains less columns
than target`
These are fixes in native execution, not passing results caused by removing
tests or newly falling back.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
**Saved CI evidence, not a new full Delta suite run:**
| Run | Actual tested merge | Main parent | Removed cases |
|---|---|---|---|
| [41106891076](https://github.com/apache/gluten/actions/runs/41106891076) |
`b36e93fe4cce134b8437f4ed0d140fe7915abee5` |
`9654b27eea55a859d25d9d0442cc08320c6d4d4a` (before #13490) | All 14 failed with
indeterminate-key errors |
| [41139661050](https://github.com/apache/gluten/actions/runs/41139661050) |
`f8a72a45805745375f42df08c1e3e2f72b68fd75` |
`1503e13b5fdd953fa46e20a91ee90d71c0494b22` (includes #13490) | All 14 ran and
passed; none skipped |
Parsed the original XML with the repository's unchanged
`compare-test-results.py` and cross-checked all eight shards' gate lists.
Executed and skipped test identities match exactly between the two runs. The
newer run has 22,571 executed tests: 22,090 passed, 481 failed, and 5,160
additional tests skipped. The 14 fixed cases are in shards 2, 3, 5, 6, and 7.
Replayed the unchanged gate with `--mode enforce --fail-on-fixed true`:
- **Affected suites against this PR's actual baseline:** all eight complete
suite reports, including both reporter copies, yield 256 passed, zero failed,
zero skipped, and a passing strict gate.
- **Whole run against this PR's actual main baseline:** now-passing entries
fall from 14 to zero. The replay still reports the same ten native-write
failures whose baseline entries were restored only on #13502 (the `invalid
replaceWhere`, backtick-enforcement, and concurrent OPTIMIZE cases). They are
outside this cleanup and are not added here.
- **Whole run against #13502's tested 495-entry baseline minus these 14:**
481 expected failures, zero unexpected failures, zero now-passing entries;
strict gate passes. This comparison isolates the unrelated ten-entry baseline
difference without changing the committed main baseline.
The preceding investigation also ran isolated reproductions with the exact
old and new CI bundles: two `transform_keys` expressions and the two Delta
v4.4.1 MERGE data scenarios, each repeated three times. The old bundle produced
12 expected indeterminate-key failures; the new bundle produced 12 successes
matching vanilla Spark. All six expression evaluations per bundle asserted a
`ProjectExecTransformer` containing `transform_keys` before execution. These
archived probes were not a fresh run of all 14 generated suite variants.
Checked that the diff removes exactly the requested 14 lines and preserves
all remaining text, including comments. `git diff --check` and the repository's
license-header checker on the sole changed file passed.
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: GitHub Copilot 1.0.102-0.unstable.r40949665530.gab1eef5
(gpt-6-astra)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
