kosiew opened a new issue, #20164: URL: https://github.com/apache/datafusion/issues/20164
## Summary Enhance the logical `CastExpr` to support struct-aware casting, field-level nullability checks, and schema-aware validation. Once complete, retire `CastColumnExpr` by updating the adapter to emit the enhanced `CastExpr` and consolidate all casting logic into one expression type. ## Scope This is a follow-up to #20163. It aims to unify casting behavior across the planner and execution layers. ### Tasks - [ ] Extend `CastExpr` in `datafusion/expr/` to support struct field casting - [ ] Add field-level nullability and metadata propagation to `CastExpr` - [ ] Implement schema-aware validation for target types - [ ] Update `CastExpr` serialization/deserialization (logical planning) - [ ] Update physical planner to convert enhanced `CastExpr` → optimized physical cast - [ ] Update `PhysicalExprAdapter` to emit enhanced `CastExpr` instead of `CastColumnExpr` - [ ] Deprecate `CastColumnExpr` (or convert to thin wrapper) - [ ] Add SQL logic tests for struct casts in standard SQL queries - [ ] Remove or archive `CastColumnExpr` once fully transitioned ### Files/Modules - `datafusion/expr/src/expr.rs` (CastExpr extension) - `datafusion/physical-planner/` (logical → physical conversion) - `datafusion/physical-expr-adapter/src/schema_rewriter.rs` (adapter integration) - `datafusion/sqllogictest/test_files/` (new SLT tests for struct casting) ### Key Features - **Unified casting:** Single expression type handles all casting scenarios - **Struct awareness:** Properly cast nested struct fields - **Nullability tracking:** Field-level nullability in logical and physical layers - **SQL support:** Struct casts work natively in SQL queries (not just internal adaptation) - **Backward compatibility:** Existing `CastExpr` uses continue to work ### Acceptance Criteria ✅ Enhanced `CastExpr` supports all `CastColumnExpr` features ✅ Physical planner correctly converts enhanced `CastExpr` to physical cast ✅ Adapter emits enhanced `CastExpr` for schema adaptation ✅ `CastColumnExpr` is deprecated or removed ✅ SQL logic tests validate struct casts work end-to-end ✅ All existing cast tests continue to pass ✅ Code compiles without warnings ## Notes - **Design consideration:** Ensure enhanced `CastExpr` doesn't become too broad; keep logical cast semantics clear while allowing physical optimization. - **Documentation:** Update casting documentation to reflect unified behavior. ## Future: Clean-Up and Removal Once this issue is complete and `CastExpr` has been adopted everywhere: - Archive or completely remove `CastColumnExpr` - Update documentation to guide users on struct casting in SQL - Consider whether other physical expressions could benefit from similar unification patterns -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
