alamb commented on code in PR #10106:
URL: https://github.com/apache/arrow-rs/pull/10106#discussion_r3714753007
##########
parquet/src/file/writer.rs:
##########
@@ -2619,4 +2619,97 @@ mod tests {
}
writer.close().unwrap();
}
+
+ #[test]
+ fn test_int96_interop() {
+ // this file has an INT96 column. rewrite it with min/max statistics
sorted per
+ // recent changes to the spec. (see
https://github.com/apache/parquet-format/pull/584)
+ let file = get_test_file("int96_timestamp_order.parquet");
+ let read_opts = ReadOptionsBuilder::new().with_page_index().build();
+ let reader = SerializedFileReader::new_with_options(file,
read_opts).unwrap();
+ let file_metadata = reader.metadata().file_metadata();
+ let schema = file_metadata.schema_descr().root_schema_ptr();
+ assert_eq!(
+ file_metadata.column_order(0),
+ ColumnOrder::INT96_TIMESTAMP_ORDER
+ );
+
+ // save read stats for later
+ let stats = reader
+ .metadata()
+ .row_group(0)
+ .column(0)
+ .statistics()
+ .expect("statistics missing");
+ let exp_min = stats.min_bytes_opt().expect("min stats missing");
+ let exp_max = stats.max_bytes_opt().expect("max stats missing");
+
+ let col_idx = reader
Review Comment:
a minor nit is that the mechanics to extract the INT96 stats from a column
is somewhat verbose and repeated twice -- it might help readability to make a
helper function or something that extracts the min/max values so there is less
code duplication and thus the actual intent of the test is clearer
##########
parquet/src/basic.rs:
##########
@@ -987,6 +987,10 @@ pub enum SortOrder {
UNDEFINED,
/// Use IEEE 754 total order.
TOTAL_ORDER,
+ /// Use INT96 timestamp order (see [parquet-format/#584]).
+ ///
+ /// [parquet-format/#584]:
https://github.com/apache/parquet-format/pull/584
Review Comment:
Maybe we should also link to the current parquet.thrift spec?
https://github.com/apache/parquet-format/blob/2076361bb64e2de9ca6a8d06eda025a6fa4e9df6/src/main/thrift/parquet.thrift#L1230-L1233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
