Let's stop worrying about performance aspects of this. If people find optimizations, that's great, but it's not the intent behind the addition of this type.
Yes, both Iceberg and Delta table formats are interested in adding this type with the specific semantics we've discussed. In Iceberg we generally define the Iceberg type to file format type mapping <https://iceberg.apache.org/spec/#parquet> and strongly prefer to have a representative logical type. Avro and ORC are supported to a lesser extent due to how they're used and the level of support available across implementations. I wouldn't worry about the other formats as we prefer to lean into parquet for consistency and ecosystem compatibility. As to how it's used, that's not really for Parquet or Iceberg to define. It's entirely possible there will be built-in functions or UDFs that allow or return only Vector defined types. Others could support both, but functions are largely left to engine implementations to define, as are runtime management and casting behaviors. At this point, I think we've done a good job of narrowing this type definition to 'FINITE_VECTOR' to both clarify the semantics and deconflict it with other potential future solutions that have different behaiors or use cases. This representation is also aligned with other vector implementations and with proposals from upstream projects that want to leverage this type. Will, I understand you may have different use cases, but we shouldn't be blocking progress here. There's substantial agreement that this is a useful addition and if you're looking for something different, please make a proposal that addresses your needs. This proposal doesn't need to address every use case; it's well defined and narrowly scoped. I think it's more important to make meaningful progress than try to solve for every related use case. Other proposals are more than welcome and expected as we get more examples of well defined types (embeddings, tensors, etc). I think the next steps are for Rok to update the proposal and then I'd urge the community to move forward with a vote on the more narrowly defined type. -Dan On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev < [email protected]> wrote: > Thx for replying. > > I keep trying to explain that there are no performance advantages. But > anyway :) > > I understand you are envisaging a vector column type (with the fixed length > nan inf constraints you’ve been detailing) being added to iceberg? Then > the following choices are interesting: > > 1. do you expect that the backing files that are parquet MUST use the new > logical vector type? > > 2. do you expect the software accessing an iceberg table - sparksql, for > example - to offer different functions on vector columns compared to naked > lists? Will a nearest neighbour search on a naked list of doubles be > supported, for example? > > 3. And will non-parquet backing files like avro and orc be supported for > the vector columns? > > thx > > Will > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now and > would > work with VECTOR. Jan asked the same question, and I don't think either of > us has seen one yet. > > I think this is the wrong question to ask, but I'm glad that you're asking > directly. Clearly, a variable-length list is a superset of a fixed-length > list. And a list of floats is a superset of a list of finite floats. A list > of optional elements is a superset of a list of required elements. I don't > think there's a use case that _can't_ be done. > > Is there a use case that can't be done with binary in place of string? And > why don't we use JSON or Variant for everything? Just because we can use a > wider type doesn't mean that we don't benefit from communicating guarantees > like "this must be UTF-8". > > Types introduce guarantees that we can use to get better performance and to > either prevent errors or raise them closer to where data is produced. > > For vectors, the guarantee that elements are finite gives us better > performance because we don't need to check each element. This also moves > the error closer to where data is produced because writers are required to > verify the guarantees about each element and will detect bad values at > write time rather than at read time. > > Ryan >
