alamb commented on code in PR #624:
URL: https://github.com/apache/parquet-format/pull/624#discussion_r4097145728
##########
LogicalTypes.md:
##########
@@ -1097,6 +1097,88 @@ optional group my_map (MAP_KEY_VALUE) {
}
```
+### Vectors
+
+`VECTOR` is used to annotate fixed-length ordered sequences of finite and
+non-null elements.
+
+`VectorType` annotation has one required parameter, `num_elements`, which is
+the number of elements in each non-null vector and must be greater than zero.
+
+`VECTOR` must always annotate the canonical 3-level structure:
+
+```
+<vector-repetition> group <name> (VECTOR(<num_elements>)) {
+ repeated group list {
+ required <element-type> element;
+ }
+}
+```
+
+* The outer level must be a group with `logicalType` set to `VECTOR` and
+ `converted_type` set to `LIST`. Its repetition must either be `optional` or
+ `required` and determines whether the vector may be null. It must contain a
+ single field named `list`.
+* The middle level must be a repeated group named `list` with a single field
+ named `element`.
+* The element MUST be a `required` primitive field, individual elements MUST
+ NOT be null. Group elements, including `LIST`, `MAP`, and `VECTOR`, are not
+ allowed.
+
+The element types supported are:
+
+* unannotated `BOOLEAN`, `FLOAT` or `DOUBLE`
+* `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or
+ `DECIMAL`
+* `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`.
+
+The 2-level structures accepted for `LIST` under its backward-compatibility
+rules are not valid for `VECTOR`.
+
+Every numeric element MUST be finite. For `FLOAT`, `DOUBLE`, and `FLOAT16`,
+this excludes NaN, positive infinity, and negative infinity. Whole vectors
+may still be null when the outer group is `optional`.
+
+For example, a nullable vector containing 768 required `FLOAT` elements is:
+
+```
+optional group embedding (VECTOR(768)) {
+ repeated group list {
+ required float element;
+ }
+}
+```
+
+Every non-null vector must contain exactly `num_elements` elements.
+Because `num_elements` is greater than zero, a non-null vector cannot be empty.
+Writers MUST enforce element count, non-null-element, and finite-element
+requirements. Readers MAY rely on these requirements without validating them. A
+reader that detects a different element count, a null element, or a non-finite
+element MUST treat the data as invalid.
+
+For example, with `num_elements = 3` and a `required float element`,
+`[1.0, 2.0, 3.0]` is valid. `[1.0, null, 3.0]`, `[null, null, null]`,
+`[1.0, NaN, 3.0]`, and `[1.0, +Infinity, 3.0]` are invalid. A null vector
+is valid only when the outer group is `optional`.
+
+Definition and repetition levels, value counts, encodings, compression,
+encryption, page boundaries, statistics, column indexes, size statistics, and
+Bloom filters are those of the primitive element column in an ordinary `LIST`.
+Encodings and metadata apply to elements rather than complete vectors.
+`VECTOR` adds no requirement that one vector be contained in a single data
+page.
+
+The sort order of `VECTOR` values is undefined and no statistics are defined
+for whole vectors. Min/max values, `distinct_count`, column-index bounds,
+and Bloom filters retain their ordinary LIST meaning. Bounds and distinct
+counts describe scalar elements. Writers may omit them.
+
Review Comment:
yeah that makes sense to me. Thank you
##########
LogicalTypes.md:
##########
@@ -1097,6 +1097,88 @@ optional group my_map (MAP_KEY_VALUE) {
}
```
+### Vectors
+
+`VECTOR` is used to annotate fixed-length ordered sequences of finite and
+non-null elements.
+
+`VectorType` annotation has one required parameter, `num_elements`, which is
+the number of elements in each non-null vector and must be greater than zero.
+
+`VECTOR` must always annotate the canonical 3-level structure:
+
+```
+<vector-repetition> group <name> (VECTOR(<num_elements>)) {
+ repeated group list {
+ required <element-type> element;
+ }
+}
+```
+
+* The outer level must be a group with `logicalType` set to `VECTOR` and
+ `converted_type` set to `LIST`. Its repetition must either be `optional` or
+ `required` and determines whether the vector may be null. It must contain a
+ single field named `list`.
+* The middle level must be a repeated group named `list` with a single field
+ named `element`.
+* The element MUST be a `required` primitive field, individual elements MUST
+ NOT be null. Group elements, including `LIST`, `MAP`, and `VECTOR`, are not
+ allowed.
+
+The element types supported are:
+
+* unannotated `BOOLEAN`, `FLOAT` or `DOUBLE`
+* `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or
+ `DECIMAL`
+* `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`.
+
+The 2-level structures accepted for `LIST` under its backward-compatibility
+rules are not valid for `VECTOR`.
+
+Every numeric element MUST be finite. For `FLOAT`, `DOUBLE`, and `FLOAT16`,
+this excludes NaN, positive infinity, and negative infinity. Whole vectors
+may still be null when the outer group is `optional`.
+
+For example, a nullable vector containing 768 required `FLOAT` elements is:
+
+```
+optional group embedding (VECTOR(768)) {
+ repeated group list {
+ required float element;
+ }
+}
+```
+
+Every non-null vector must contain exactly `num_elements` elements.
+Because `num_elements` is greater than zero, a non-null vector cannot be empty.
+Writers MUST enforce element count, non-null-element, and finite-element
+requirements. Readers MAY rely on these requirements without validating them. A
+reader that detects a different element count, a null element, or a non-finite
+element MUST treat the data as invalid.
+
+For example, with `num_elements = 3` and a `required float element`,
+`[1.0, 2.0, 3.0]` is valid. `[1.0, null, 3.0]`, `[null, null, null]`,
+`[1.0, NaN, 3.0]`, and `[1.0, +Infinity, 3.0]` are invalid. A null vector
+is valid only when the outer group is `optional`.
+
+Definition and repetition levels, value counts, encodings, compression,
Review Comment:
Do we need to say this explicitly? I feel like this just reiterates that the
physical storage is a list 🤔
##########
src/main/thrift/parquet.thrift:
##########
@@ -314,6 +314,12 @@ struct Statistics {
* or DOUBLE, or logical type is FLOAT16.
* If this field is not present, readers MUST assume NaNs may be present
* (i.e. MUST assume nan_count > 0 and MAY NOT assume nan_count == 0).
+ * If the column is the element leaf of a VECTOR, whose elements MUST by
+ * convention always be finite (see LogicalTypes.md) nan_count MUST be
+ * zero when present.
+ * If the column is the element leaf of a VECTOR, whose elements MUST by
+ * convention always be finite (see LogicalTypes.md) writers SHOULD omit
Review Comment:
this still seems overly specific to me. If the logical type forbids nans,
why do we also need to specify anything about the statistics?
I would expect writers that write Vector logical type simply don't write nan
statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
