rok opened a new pull request, #592: URL: https://github.com/apache/parquet-format/pull/592
### Rationale for this change As discussed in #430, fixed-size list / vector data in Parquet is currently encoded as standard `LIST`, even when every value has the same number of elements. This forces readers and writers through variable-length list machinery and repetition-level handling for data whose shape is known from the schema. This is increasingly relevant for embeddings, tensors, images, and other fixed-shape arrays. The topic was discussed on the [mailing list](https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3) and in a [design doc](https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0). Several approaches were prototyped ([A](https://github.com/rok/arrow/pull/53), [B](https://github.com/rok/arrow/pull/51), [C](https://github.com/rok/arrow/pull/52)) and [benchmarked](https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef). ### What changes are included in this PR? This PR adds a new `VECTOR` logical and repetition types for fixed-size lists. Thrift additions: - `FieldRepetitionType.VECTOR = 3` - `struct VectorType {}` - `LogicalType.VECTOR = 19` - `SchemaElement.vector_length = 11` The layout for vector data would look like so: ```text <required|optional> group emb (VECTOR) { vector group list [N] { <required|optional> <element-type> element; } } ``` Key semantics: - LogicalType.VECTOR annotates the outer group. - The middle group has `repetition_type = VECTOR` and carries `SchemaElement.vector_length`. - The VECTOR level contributes no definition or repetition level. - Nullable vectors are represented by an optional outer group. - Nullable elements are represented by an optional element field. - Each vector occurrence contributes exactly vector_length element slots / level entries. - Statistics remain element-level statistics. - Encodings remain element-level encodings. - Writers should not split one vector value across data pages. ### Do these changes have PoC implementations? * https://github.com/apache/arrow-go/pull/854 Closes #430 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
