paleolimbot opened a new pull request, #15036: URL: https://github.com/apache/datafusion/pull/15036
As discussed in #12644 and on a recent DataFusion sync, a lightweight mechanism to patch in user-defined type support is to use the `Field` rather than the `DataType` in as many places as possible. This is a very in-progress experiment...I know there is also work on LogicalType that may also be able to solve this (as would adding an `Extension` member to the arrow-rs `DataType` enum). I figured experimenting in public might be productive but am happy to go back to experimenting in private if that is less confusing! ## Which issue does this PR close? - Closes #12644 (if the experiments in this PR work!) ## Rationale for this change Most database systems have the concept of a "user defined type" that can act as first-class types (or close to them). The exact things that can be customized about this vary by system but usually include the ability to define new functions whose signature matches that type, add overloads to existing functions (notably: cast), or sort/display values in a particular way. Arrow has the concept of an "extension type", which is implemented in the C data interface and IPC formats as two `Field` metadata fields: `ARROW:extension:name` and `ARROW:extension:metadata`. In arrow-rs there is the `ExtensionType` which provides a means by which to (at least) centralize the serialization and deserialization of `ARROW:extension:metadata`. ## What changes are included in this PR? Experiments on the way towards replacing `DataType` with `Field` where possible. ## Are these changes tested? If this is ever more than just an experiment, they will be! ## Are there any user-facing changes? If this is ever more than just an experiment, there will be! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
