alamb commented on issue #7686: URL: https://github.com/apache/arrow-rs/issues/7686#issuecomment-3005706724
Here is my summary after the parquet sync call today: * Pedantically, according to the current spec there is no defined ordering for Int96 types and thus arrow-rs can not be writing "incorrect" values (as there is no definition of correct) * relevant portion of the spec: https://github.com/apache/parquet-format/blob/cf943c197f4fad826b14ba0c40eb0ffdab585285/src/main/thrift/parquet.thrift#L1079 * Practically speaking, arrow-rs is writing something different than Photon (Databricks proprietary spark engine) * What Photon is doing arguably makes more sense (to use the ordering of the only logical type to use Int96) * https://github.com/apache/arrow-rs/pull/7687 brings arrow-rs into line with that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
