tobixdev commented on issue #18223: URL: https://github.com/apache/datafusion/issues/18223#issuecomment-3777535998
After good discussions in https://github.com/apache/datafusion/pull/18552 and https://github.com/apache/arrow-rs/issues/8730 the most likely way to get started with extension type support is to begin with a "serialized first" approach as @paleolimbot called it in https://github.com/apache/datafusion/pull/18552#issuecomment-3774267690 . This allows us to re-use the already (mostly) accomplished plumbing around `Field` with the con that we must have access to registry in order to interpret the extension type. This allows implementing use cases such as record batch pretty-printing where a registry is available, but is difficult where we have no registry available such as formatting logical plans. I think another take away is that we should design the API in way that we can still switch to a different approach. For example, storing a `LogicalType` (or similar) instead of just the Metadata. I guess the "success" of extension type support and need for the associated use cases will decide how much resource we want to devote to another plumbing round. From my perspective, the discussion PR can be separated into three PRs: 1. A PR that adds the machinery for registering, looking-up, and resolving extension types. 2. A PR that adds the record batch pretty-printer 3. A PR that implements the record batch pretty-printer for the canonical extension types. I am happy to spend some time I can spare with these PRs if there is a chance that we can merge them after my vacation. @paleolimbot maybe we should discuss your `BoundExtension` approach before starting with that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
