baibaichen opened a new pull request, #11294:
URL: https://github.com/apache/incubator-gluten/pull/11294
### What changes were proposed in this pull request?
This PR addresses Spark 4.0 compatibility issues and refactors the
ExpressionConverter to improve code quality.
**Main Changes:**
1. **Add compatibility layer for `StructsToJson` expression across Spark
versions**
- In Spark 4.0, `StructsToJson` was replaced with `Invoke` expression
using `StructsToJsonEvaluator`
- Added `StructsToJsonInvoke` extractor in shims for all Spark versions
(3.2-4.0) to provide a unified interface
- For Spark 4.0: Extracts evaluator options, child expression, and
timeZoneId from the Invoke pattern
- For Spark 3.2-3.5: Returns None to maintain backward compatibility
2. **Add support for `StaticInvoke` expressions in Spark 4.0**
- `lengthOfJsonArray` and `jsonObjectKeys` now use `StaticInvoke` in
Spark 4.0
- Added mappings to handle these functions properly in ExpressionConverter
3. **Refactor `ExpressionConverter` to reduce `return` usage**
- Restructured code to avoid explicit `return` statements, following
Scala best practices
- Improved readability by using pattern matching and Option-based control
flow
- Extracted common mappings (icebergStaticInvokeMap, staticInvokeMap) to
reduce code duplication
### Why are the changes needed?
- Spark 4.0 introduced breaking changes where `StructsToJson` was replaced
with `Invoke` expressions
- Without this compatibility layer, JSON-related functions fail in Spark 4.0
- The `return` keyword is discouraged in Scala as it breaks functional
programming style and can lead to unexpected behavior
### Does this PR introduce any user-facing change?
No. This is an internal compatibility fix.
### How was this patch tested?
- Removed Spark-version-specific test limitations in
`JsonFunctionsValidateSuite`
- Existing JSON function tests now pass across all Spark versions (3.2-4.0)
Related issue: #11088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
