> 1. do you expect that the backing files that are parquet MUST use the new logical vector type?
I expect that the Iceberg spec will require writers to produce the annotation. > 2. do you expect the software accessing an iceberg table . . . to offer different functions on vector columns compared to naked lists? Yes, I expect different functions in engines. I think pgvector uses more specific functions for vectors, like `vector_dims` and `vector_norms`. As far as I'm aware, these aren't available for arrays without an explicit cast. > Will a nearest neighbour search on a naked list of doubles be supported, for example? This is an engine decision, but a list like this would not be exposed as a vector. You would most likely cast it to a vector in order to call vector functions. > 3. And will non-parquet backing files like avro and orc be supported for the vector columns? All Iceberg types can be stored in supported file formats. It seems like the direction for this line of questioning is basically the same as before: Can we just treat list<float> as a synonym of vector? And again, I think this is the wrong question. We have users right now that struggle with errors that are found at read time, like the dimensions of their vectors not matching. Finding this out at read time is a problem and we want to shift validation to write time so we can catch errors earlier and have guarantees that the data is usable. As I mentioned, one of the ways we have customers using data is sharing it with systems like pgvector that also have these requirements. Just because you can think of a vector as a very special list of floats doesn't mean that write time guarantees are unreasonable or unnecessary. Ryan On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev < [email protected]> wrote: > Thx for replying. > > I keep trying to explain that there are no performance advantages. But > anyway :) > > I understand you are envisaging a vector column type (with the fixed length > nan inf constraints you’ve been detailing) being added to iceberg? Then > the following choices are interesting: > > 1. do you expect that the backing files that are parquet MUST use the new > logical vector type? > > 2. do you expect the software accessing an iceberg table - sparksql, for > example - to offer different functions on vector columns compared to naked > lists? Will a nearest neighbour search on a naked list of doubles be > supported, for example? > > 3. And will non-parquet backing files like avro and orc be supported for > the vector columns? > > thx > > Will > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now and > would > work with VECTOR. Jan asked the same question, and I don't think either of > us has seen one yet. > > I think this is the wrong question to ask, but I'm glad that you're asking > directly. Clearly, a variable-length list is a superset of a fixed-length > list. And a list of floats is a superset of a list of finite floats. A list > of optional elements is a superset of a list of required elements. I don't > think there's a use case that _can't_ be done. > > Is there a use case that can't be done with binary in place of string? And > why don't we use JSON or Variant for everything? Just because we can use a > wider type doesn't mean that we don't benefit from communicating guarantees > like "this must be UTF-8". > > Types introduce guarantees that we can use to get better performance and to > either prevent errors or raise them closer to where data is produced. > > For vectors, the guarantee that elements are finite gives us better > performance because we don't need to check each element. This also moves > the error closer to where data is produced because writers are required to > verify the guarantees about each element and will detect bad values at > write time rather than at read time. > > Ryan >
