kylebarron commented on issue #12644: URL: https://github.com/apache/datafusion/issues/12644#issuecomment-2588301284
> Is support for this blocked by lack of support in arrow-rs? arrow-rs intentionally doesn't "directly support" extension types through `DataType`. The alternative (now archived) [arrow2 implementation](https://github.com/jorgecarleitao/arrow2) had [an `Extension` variant](https://docs.rs/arrow2/latest/arrow2/datatypes/enum.DataType.html#variant.Extension), but this has the downside of requiring _all_ code that needs to match the `DataType` enum to be aware of logical extension types. See https://github.com/apache/arrow-rs/issues/4472 for more discussion. Instead, arrow-rs supports extension types indirectly via the metadata hashmap on the `Field` accompanying some `Array`. So e.g. in geoarrow-rs, we have custom types like `PointArray` that internally manage both the array data and the field metadata, but when exporting to `arrow` objects, you'd need to carry the `Array` and the `Field` together. I haven't closely followed the logical type work in progress in Datafusion but I assume it would be associating some Field with the physical type, and extension types could inject metadata there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
