Hi all, As proposed during the general community call we held a focused call today to discuss the proposal for adding a logical vector type to parquet. This was called because it seems:
* Vectors (as in vector DB) storage needs are different to what we need for fixed size list applications we've been disucssing so far [0]. This is driven also by Iceberg's need for Vector type [1] * We don't yet have a consensus of what physical representation of Vector type should be. Notes from the call can be found here [2]. Short summary: We have converged on two options that could work, but have not yet decided which one to propose: - Option A - FLBA as physical storage (FLBA(num_elements * element_width)). Vector logical type would provide an enum specifying the element type, and the element would be non-nullable. The element type enum could be independent of the parquet type system, making evolution easier. Downside is loss of element-level encodings available today, though they can be added later. Upside is simplicity and ease of evolution. - Option C1 - LIST with numeric parquet type elements as physical storage. New float types (e.g. BF16) can be added in parquet proper as needed. Downside is we need to add new types to parquet, DL/RL overhead is likely not a concern. Upside is we have encoding support now. Regarding NaN/Inf limitations, the consensus was that these should be handled by statistics. This leads to a need for a new inf_count statustics. It would be great to discuss the merits of these two proposals here on the mailing list for maximum visibility and participation. We will likely continue to discuss this on tomorrow's community call too. [0] https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3 [1] https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0#heading=h.j5pn3fef87b2 [2] https://docs.google.com/document/d/1rPTjEJTwN97UfHvNlvdriX9XBQsW8PGwIHMXHvy-XlU/edit?tab=t.0 Best, Rok
