kosiew commented on issue #16565:
URL: https://github.com/apache/datafusion/issues/16565#issuecomment-3035480855
@adriangb,
> incorporating the struct-aware casting logic from #16371 into `CastExpr`
and `TryCastExpr`
Yes. I think it is a necessary step to explore the feasibility of rewriting
Expr for schema adaptation.
A few thoughts on this:
1. **Reusability of struct-aware casting kernels**: The core struct-aware
casting kernel introduced in #16371 was designed to recursively handle casting
between nested struct types, dealing with missing or reordered fields.
Integrating this logic into `CastExpr` and `TryCastExpr` would unify casting
semantics across physical expressions, ensuring consistent behavior during
predicate rewrite and evaluation.
2. **Improved predicate pushdown**: Predicate pushdown often requires
projecting predicates down into struct subfields. If schemas differ between
scans (e.g., fields missing or reordered), leveraging struct-aware casts would
allow predicates to be adapted instead of dropped—thus generating more
selective filters pushed down to data sources.
3. **Implementation considerations**: We would need to extend `CastExpr` and
`TryCastExpr` to detect when the input/output types are structs and invoke the
struct-aware casting logic appropriately. This means:
- Adding recursive traversal logic in these expressions.
- Handling missing subfields by either filling with NULLs or using
default values as per the kernel's design.
- Handling field reordering and type promotion gracefully.
4. **Challenge**: Rewriting the struct-aware casting logic from #16371 into
manipulating expressions. Your expression rewrite prowess bends my mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
