paleolimbot commented on issue #12644: URL: https://github.com/apache/datafusion/issues/12644#issuecomment-3492971986
> I feel like there is significant motion / momentum for better Field / metadata support in service of user defined types Glad to hear it! > Do you know of anything more specific? I don't! There is some good discussion on https://github.com/apache/datafusion/issues/18223 with respect to some first steps. > what we are trying to do (aka ensure Field is present everywhere instead of just DataType) I am not sure that is the consensus on this ticket or #18223 so far...I think we all want something better than that but aren't sure how to do it without changing arrow-rs or a lot of disruption of the DataFusion code base. Replacing the DataType with a Field in the logical plan and SQL parser is sort of the bare minimum for somebody else to invent their own type system on top of DataFusion with some effort. > If no such thing exists, would it be ok if I created one? That would be great! I am definitely willing to do work, it's just hard to do so without consensus and I am not sure what that is. My personal first step might be to replace any `FieldRef` that is meant to communicate a data type with `ExprType` or `ExprField` or `DFField` (that is a thin wrapper around a `FieldRef`) so that we can get metadata everywhere in the short term with the flexibility to update the internal representation (e.g., with a `dyn` type of some kind). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
