PerumalsamyR opened a new pull request, #12521: URL: https://github.com/apache/gluten/pull/12521
## What changes are proposed in this pull request? `glutenNormalizeIds()` in the Gluten plan stability suites treats every `#<digits>` token in the explain text as an ExprId. String constants from the benchmark queries match the same pattern — `Brand#12`/`Brand#23`/`Brand#34` in TPCH q19, `Brand#45` in q16, `Brand#23` in q17, and the `i_brand` values like `scholaramalgamalg #14` in TPCDS q53/q63 — so they were renumbered as if they were ExprIds. Whether such a literal token collides with a real ExprId depends on the JVM's ExprId counter at the time the plan is printed: with the whole module running in one JVM the counter is large and the literals get their own normalization slots, but in an isolated suite run the counter is small (e.g. `p_brand#12` vs the literal `Brand#12` share the `#12` token), the equivalence classes merge, and the output no longer matches the committed goldens. This is the long-term fix requested in #12375. Changes: - `glutenNormalizeIds` now takes the string literals containing `#<digits>` (extracted from the query text with `getHashLiterals`) and shields their occurrences in the explain output from ExprId normalization by masking the `#` during the rewrite. Normalization output no longer depends on the JVM's ExprId counter. - The pure normalization helpers moved from the trait to a companion object so they can be unit-tested, and `GlutenNormalizeIdsSuite` (new) covers encounter-order normalization, plan-id/`_pre_` handling, literal extraction, CHAR-padded literals, and the collision/stability case. - The 22 affected golden files (q16/q17/q19 TPCH, q53/q63 TPCDS v1_4 and modified, with and without stats, for both spark40 and spark41) are updated: protected literals now keep their true values from the query, and the remaining ExprIds are renumbered without the slots the literals used to occupy. The golden update was derived mechanically from the committed goldens: the old normalization is a bijective, encounter-order-preserving renumbering, so applying the new normalizer to the golden with the literal values restored reproduces exactly what the new normalizer emits for the raw explain text. The transform asserted that literal slots were never merged with attribute ExprIds in the committed goldens (they were not, for all 22 files) and that the result is a fixed point of the new normalizer. The plan-stability suites in CI verify the result end to end. Fixes #12375 ## How was this patch tested? - New `GlutenNormalizeIdsSuite` passes locally (`dev/run-scala-test.sh -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut -pl gluten-ut/spark40 -s org.apache.spark.sql.GlutenNormalizeIdsSuite`), including a test that reproduces the isolated-run vs shared-JVM divergence without the fix and asserts stability with it. - Existing plan stability suites (`GlutenTPCHPlanStabilitySuite`, `GlutenTPCDS*PlanStabilitySuite`) validate the updated goldens in CI. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
