Hi folks, In the Parquet and Iceberg communities, we are having ongoing discussions about introducing new data types for vector data, i.e., fixed-size lists of numeric values that typically represent vector embeddings for machine learning contexts.
In Parquet [1], we will likely introduce a new logical type FINITE_VECTOR with a property "num_elements" that annotates a 3-level LIST of numeric values. In Iceberg [2], we want to introduce a similar vector type. For Parquet data files, Iceberg tables will use this newly introduced Parquet logical type. We are now discussing how best to approach Avro files [3], and wanted to get feedback from the Avro community. One way would be to use the "array" type, and annotate it with a logicalType "vector" and a "num_elements" (or similar field). Would you be interested in standardizing such a vector annotation in Avro? Or are there alternatives that make more sense? Best, Philipp [1] https://github.com/apache/parquet-format/pull/624 [2] https://github.com/apache/iceberg/pull/18416 [3] https://iceberg.apache.org/spec/#avro
