divjotarora commented on code in PR #584:
URL: https://github.com/apache/parquet-format/pull/584#discussion_r3474445110
##########
src/main/thrift/parquet.thrift:
##########
@@ -1108,20 +1113,22 @@ union ColumnOrder {
* BOOLEAN - false, true
* INT32 - signed comparison
* INT64 - signed comparison
- * INT96 (only used for legacy timestamps) - undefined(+)
+ * INT96 (only used for legacy timestamps) - depends on sort order (+)
* FLOAT - signed comparison of the represented value (*)
* DOUBLE - signed comparison of the represented value (*)
* BYTE_ARRAY - unsigned byte-wise comparison
* FIXED_LEN_BYTE_ARRAY - unsigned byte-wise comparison
*
* (+) While the INT96 type has been deprecated, at the time of writing it is
- * still used in many legacy systems. If a Parquet implementation chooses
- * to write statistics for INT96 columns, it is recommended to order them
- * according to the legacy rules:
- * - compare the last 4 bytes (days) as a little-endian 32-bit signed
integer
- * - if equal last 4 bytes, compare the first 8 bytes as a little-endian
- * 64-bit signed integer (nanos)
- * See https://github.com/apache/parquet-format/issues/502 for more
details
+ * still used in many legacy systems. It is optional for writers to emit
+ * statistics for INT96 columns. Writers that emit stats for such columns
+ * should use the INT96_TIMESTAMP_ORDER for this type and order the
values
+ * according to the legacy rules:
+ * - compare the last 4 bytes (days) as a little-endian 32-bit signed
integer
+ * - if equal last 4 bytes, compare the first 8 bytes as a little-endian
+ * 64-bit signed integer (nanos)
+ * If TYPE_ORDER is used for an INT96 column, readers should ignore
statistics
Review Comment:
I don't think it's necessary because the section below for `TYPE_ORDER` of
floating point values also says:
> If TYPE_ORDER is used for floating point types, then the following
> * compatibility rules should be applied when reading statistics:
But I pushed a small change to clarify, let me know if it's OK
##########
src/main/thrift/parquet.thrift:
##########
@@ -1108,20 +1113,22 @@ union ColumnOrder {
* BOOLEAN - false, true
* INT32 - signed comparison
* INT64 - signed comparison
- * INT96 (only used for legacy timestamps) - undefined(+)
+ * INT96 (only used for legacy timestamps) - depends on sort order (+)
* FLOAT - signed comparison of the represented value (*)
* DOUBLE - signed comparison of the represented value (*)
* BYTE_ARRAY - unsigned byte-wise comparison
* FIXED_LEN_BYTE_ARRAY - unsigned byte-wise comparison
*
* (+) While the INT96 type has been deprecated, at the time of writing it is
- * still used in many legacy systems. If a Parquet implementation chooses
- * to write statistics for INT96 columns, it is recommended to order them
- * according to the legacy rules:
- * - compare the last 4 bytes (days) as a little-endian 32-bit signed
integer
- * - if equal last 4 bytes, compare the first 8 bytes as a little-endian
- * 64-bit signed integer (nanos)
- * See https://github.com/apache/parquet-format/issues/502 for more
details
+ * still used in many legacy systems. It is optional for writers to emit
+ * statistics for INT96 columns. Writers that emit stats for such columns
+ * should use the INT96_TIMESTAMP_ORDER for this type and order the
values
+ * according to the legacy rules:
+ * - compare the last 4 bytes (days) as a little-endian 32-bit signed
integer
+ * - if equal last 4 bytes, compare the first 8 bytes as a little-endian
+ * 64-bit signed integer (nanos)
+ * If TYPE_ORDER is used for an INT96 column, readers should ignore
statistics
Review Comment:
I don't think it's necessary because the section below for `TYPE_ORDER` of
floating point values also says:
> If TYPE_ORDER is used for floating point types, then the following
> compatibility rules should be applied when reading statistics:
But I pushed a small change to clarify, let me know if it's OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
