LuciferYang opened a new issue, #12837: URL: https://github.com/apache/gluten/issues/12837
### Description Sub-task of #12807, covering its first checkbox: `Migrate important tests to Spark-34+`. Three groups of test coverage exist only in `gluten-ut/spark33`. Removing that module deletes them, so they need to land on 3.4, 3.5, 4.0 and 4.1 first. 1. `GlutenImplicitsTest`, six `fallbackSummary` cases. No equivalent exists on 3.4+. 2. The four DPP `DisableProject` suites. All four surviving modules carry the `DisableScan` counterparts but not these. They landed in spark33 in `caf959e46` (2023) and the modules created afterwards never picked them up. 3. Eight Gluten-authored (`testGluten`) cases. Six of them trace to the commit that added the case to spark33 and skipped the newer modules: | Case | Missed by | |---|---| | `unhex` (MathExpressions) | `399a91b12` | | `flatten function` (DataFrameFunctions) | `de26ed2da` | | `date_from_unix_date` (DateFunctions) | `31adaf875` | | `SPARK-32932` (Velox AQE) | `d101cb80e` | | `SPARK-37652` (CH AQE) | `d2b50ac54` | | `avoid unnecessary filter binding for subfield during scan` (Hive) | `a662f3f75` | The remaining two are rewrite-style: spark33 excludes Spark's own case and supplies a Gluten version, and the surviving modules have neither half. Those are `SPARK-35675` in `GlutenEnsureRequirementsSuite` and `default` in `GlutenLiteralExpressionSuite`. Two groups are deliberately not migrated. The three ANSI cast suites (`GlutenCastSuiteWithAnsiModeOn`, `GlutenAnsiCastSuiteWithAnsiModeOn`, `GlutenAnsiCastSuiteWithAnsiModeOff`) are enabled only on 3.3, but they never reach Velox in CI. `spark.gluten.sql.columnar.backend.velox.ansiFallback.enabled` defaults to true and the `spark-test-spark33` job does not turn it off, so `FallbackOnANSIMode` tags every node in the plan for fallback. `AnsiCast`, a 3.3-only class, is not in `ExpressionMappings` either. Dropping these loses no coverage that CI actually exercises. Nine `.sql` files appear only in spark33's `VeloxSQLQueryTestSettings`. Each needs a UT run against Spark's own golden `.out` before it can be enabled, which does not fit a pure-addition change. Better handled as a separate issue. One gap in the inventory: this covers suite granularity, sql-file granularity, and Gluten-authored case granularity (8 out of 286 names). It does not cover vanilla `test(...)` cases inherited from Spark's own suites that are enabled on 3.3 but excluded in all four surviving modules' settings. spark33's settings carry hundreds of `.exclude` entries, and whether their complement is a subset of 3.4+ is still unanswered. That should be settled before the module is deleted. Nothing is removed by this work, so it can land independently of the decision to drop 3.3. This description was written with AI assistance (Claude Code). ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
