yukkit commented on issue #7923: URL: https://github.com/apache/arrow-datafusion/issues/7923#issuecomment-1780585252
@alamb I read the discussion on data types in #7421. I understand that there is no concept of logical types in arrow-rs. Rather than adding an ExtensionType in arrow DataType, I am more inclined to introduce a LogicalType in DF, just as discussed in #7421. My thoughts are as follows: 1. Under the current arrow-rs architecture, adding logical types is a relatively difficult and labor-intensive task. Additionally, there is uncertainty about the actual benefits. I believe this is one of the reasons why @tustvold approaches this matter with caution. 2. Introducing a LogicalType in DF would have a relatively small impact. The LogicalType only comes into play during the planning phase and is not significantly different from the existing logic (based on arrow datatype). 3. It would be easier to add extension types. However, it is important to clearly define the behavior and scope of LogicalType and PhysicalType. Finally, in my opinion, both of the following plans are feasible: 1. Add ExtensionType in arrow DataType 2. Add LogicalType in DF I look forward to hearing more suggestions on which plan to ultimately implement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
