alamb commented on issue #50622: URL: https://github.com/apache/arrow/issues/50622#issuecomment-5072874853
> > For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property store_schema is enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read. > > arrow-go does the same using the same property information and is thus compatible with arrow-cpp (and the impls based on it like pyarrow) writing the arrow schema into the footer. Arrow-rs does the same thing It looks to me like @sdf-jkl observation is that there are certain arrow types that can not be round tripped through a Parquet Variant column without some sort of additional metadata Specifically something like `UInt8` --> `Variant` must be stored as Variant::Int16 (so the knowledge that the column only has UInt8 values is lost) As mentioned above, the normal Parquet logical types have the same problem, so the C++ writer adds extra metadata in the file so even though a value might be stored as Parquet type Int 16, Arrow readers know to try and read it as UInt8. This metadata convention seems to be commonly followed. Maybe we can think about additional metadata writers could add if they want to recreate the original schema. I am not sure how important this case is, though, because if you had data that was originally highly typed and structured you probably wouldn't be using Variant 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
