rdtr opened a new pull request, #13195: URL: https://github.com/apache/gluten/pull/13195
## What changes are proposed in this pull request? When I ran `MiscOperatorSuite` and `GlutenBloomFilterFallbackSuite` on Spark 4.2, both suites aborted in `beforeAll` before running any test: ``` *** RUN ABORTED *** java.lang.AssertionError: assertion failed: Function identifier must be fully qualified (3-part): velox_dummy_expression ``` Since Spark 4.2 (SPARK-55964), function names are managed as 3-part identifiers such as `system.session.my_func`, and `FunctionRegistry.registerFunction` asserts that the identifier is fully qualified. Both suites call `FunctionRegistry.registerFunction` directly with unqualified names: `MiscOperatorSuite` registers `velox_dummy_expression` through `VeloxDummyExpression.registerFunctions`, and `GlutenBloomFilterFallbackSuite` registers `bloom_filter_agg` and `might_contain`. Because the registration is in `beforeAll`, ScalaTest aborts the whole suite. Other suites are not affected: `spark.udf.register` goes through `createOrReplaceTempFunction`, which builds the 3-part name itself. This PR registers and drops these functions through `SessionCatalog.registerFunction` / `unregisterFunction` instead. `SessionCatalog` has the same API from Spark 3.4 to 4.2, and on Spark 4.2 it stores an unqualified name as a temporary function in the session namespace, so no shim is needed. `VeloxDummyExpression.registerFunctions` / `unregisterFunctions` now take a `SessionCatalog`; only `MiscOperatorSuite` calls them. The function names don't change, only the registry key on Spark 4.2. `SessionCatalog` builds the `ExpressionInfo` from the `FunctionIdentifier` and the class name, so the name is still `bloom_filter_agg`. For example, for `bloom_filter_agg`: | | Registry key | `ExpressionInfo` name | `DESCRIBE FUNCTION bloom_filter_agg` | |---|---|---|---| | Spark 4.1, before | `bloom_filter_agg` | `bloom_filter_agg` | `Function: bloom_filter_agg` | | Spark 4.1, after | `bloom_filter_agg` | `bloom_filter_agg` | `Function: bloom_filter_agg` | | Spark 4.2, before | (assertion error) | | | | Spark 4.2, after | `system.session.bloom_filter_agg` | `bloom_filter_agg` | `Function: bloom_filter_agg` | Queries still call the functions by the unqualified name, such as `bloom_filter_agg(...)`. The same direct registration exists outside the Velox backend, and this PR leaves it unchanged: - `backends-clickhouse`: `GlutenCustomAggExpressionSuite` and `GlutenClickhouseCustomerExpressionTransformerSuite`. These will hit the same assertion when the ClickHouse backend runs on Spark 4.2. - `backends-bolt`: the same `VeloxDummyExpression` and `MiscOperatorSuite` code. Bolt has no Spark 4.2 profile yet. Part of #13179 ## How was this patch tested? After the change, I ran `MiscOperatorSuite` and `GlutenBloomFilterFallbackSuite` again: - Spark 4.2: 107 tests pass. Before this change, both suites aborted with the error above. - Spark 4.1: 107 tests pass. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
