albertlockett commented on PR #22647: URL: https://github.com/apache/datafusion/pull/22647#issuecomment-4611681005
> What I'm trying to highlight is this PR only adds this path for physical execution; meaning it is unreachable via normal SQL execution which goes through the optimizer path because the signature of the UDFs still coerce away from dictionary (hence why I wanted an end-to-end SLT example to prove this works). Also, taking another look, this physical execution path is implemented only for the array path and misses out the scalar path. Thanks @Jefffrey, that helps clarify things quite a bit. Sorry for confusion. I feel we were thinking about this change from slightly different perspectives. For my use-case, I only needed this corrected for physical expr evaluation, so I wasn't thinking about this from the SQL execution perspective. I can make the changes to have this path invoked for SQL execution as well, as you've suggested. Do you know off-hand if there are currently any UDFs with coercible signatures that accept dictionary arrays? I browsed briefly and didn't see any (maybe I'm missing something). I'm thinking that maybe we'll need to add some new capabilities to the logical type or type signature module to get this working. Currently, we're using `TypeSignatureClass::Native` which takes a `LogicalTypeRef`, which contains a `dyn LogicalType` https://github.com/apache/datafusion/blob/533ef35a65a879d55af7d88ad298980fb8099134/datafusion/expr-common/src/signature.rs#L383 And the only implementation we have of this is for `NativeType` https://github.com/apache/datafusion/blob/533ef35a65a879d55af7d88ad298980fb8099134/datafusion/common/src/types/native.rs#L266 My tentative plan would be to add create a implementation of `LogicalType` capable preserving dictionary arrays during type coercion. Does that sound like the right approach? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
