> Could the proposal include a compatibility section addressing unannotated list data before we vote?
Will, would a paragraph like this [1] be what you have in mind? On the topic of the vote - I would like to call it later today based on parquet-format [2]. [1] https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4 [2] https://github.com/apache/parquet-format/pull/624 Best, Rok On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev <[email protected]> wrote: > Thanks Ryan, Dan. > > My questions about what the type enables, and how you think Iceberg tooling > would use it, stem from the backward-compatibility concerns I've raised > earlier in the thread. I was trying to weigh the cost against the benefit. > I worry that this annotation will fragment and orphan a lot of data: > perfectly good vectors that become second-class just because they weren't > written with the annotation. > > There is an enormous amount of embedding data in Parquet today. Many many > lottabytes of it :) Some of it is packed in bespoke formats, but a very > large share is plain, unannotated lists of floats or doubles. In addition > to all this existing data, many pipelines will keep writing new data this > way too, because they are unaware of the new annotation or can't be > changed. > > Ryan, you said compatibility isn't a concern because this is a new type. I > think that assumes the data is new data written by an Iceberg writer. But > many Iceberg tables are built from files written by other engines. The > standard implementations support this directly (add_files, migrate, name > mapping), and the popular public embedding datasets are a typical source. > If Iceberg requires vector columns to carry the annotation, none of those > files can back a vector column without being rewritten, even though the > bytes would be identical. > > It makes sense for Iceberg writers to check for inf/nan/null when they > write a vector column. But since many files come from other writers, robust > readers should check correctness anyway, as they do with UTF-8 for strings. > > So my position remains that this would be better done entirely in Iceberg > rather than pushed down to Parquet. Iceberg ought to accept unannotated > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via add_files > and a metadata-only list → vector promotion. > > Could the proposal include a compatibility section addressing unannotated > list data before we vote? > > Will > > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <[email protected]> > wrote: > > > > 1. do you expect that the backing files that are parquet MUST use the > new > > logical vector type? > > > > I expect that the Iceberg spec will require writers to produce the > > annotation. > > > > > 2. do you expect the software accessing an iceberg table . . . to > offer > > different functions on vector columns compared to naked lists? > > > > Yes, I expect different functions in engines. I think pgvector uses more > > specific functions for vectors, like `vector_dims` and `vector_norms`. As > > far as I'm aware, these aren't available for arrays without an explicit > > cast. > > > > > Will a nearest neighbour search on a naked list of doubles be > supported, > > for example? > > > > This is an engine decision, but a list like this would not be exposed as > a > > vector. You would most likely cast it to a vector in order to call vector > > functions. > > > > > 3. And will non-parquet backing files like avro and orc be supported > for > > the > > vector columns? > > > > All Iceberg types can be stored in supported file formats. > > > > It seems like the direction for this line of questioning is basically the > > same as before: Can we just treat list<float> as a synonym of vector? > > > > And again, I think this is the wrong question. We have users right now > that > > struggle with errors that are found at read time, like the dimensions of > > their vectors not matching. Finding this out at read time is a problem > and > > we want to shift validation to write time so we can catch errors earlier > > and have guarantees that the data is usable. As I mentioned, one of the > > ways we have customers using data is sharing it with systems like > pgvector > > that also have these requirements. Just because you can think of a vector > > as a very special list of floats doesn't mean that write time guarantees > > are unreasonable or unnecessary. > > > > Ryan > > > > > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev < > > [email protected]> wrote: > > > > > Thx for replying. > > > > > > I keep trying to explain that there are no performance advantages. But > > > anyway :) > > > > > > I understand you are envisaging a vector column type (with the fixed > > length > > > nan inf constraints you’ve been detailing) being added to iceberg? > Then > > > the following choices are interesting: > > > > > > 1. do you expect that the backing files that are parquet MUST use the > new > > > logical vector type? > > > > > > 2. do you expect the software accessing an iceberg table - sparksql, > for > > > example - to offer different functions on vector columns compared to > > naked > > > lists? Will a nearest neighbour search on a naked list of doubles be > > > supported, for example? > > > > > > 3. And will non-parquet backing files like avro and orc be supported > for > > > the vector columns? > > > > > > thx > > > > > > Will > > > > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: > > > > > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now and > > > would > > > work with VECTOR. Jan asked the same question, and I don't think either > > of > > > us has seen one yet. > > > > > > I think this is the wrong question to ask, but I'm glad that you're > > asking > > > directly. Clearly, a variable-length list is a superset of a > fixed-length > > > list. And a list of floats is a superset of a list of finite floats. A > > list > > > of optional elements is a superset of a list of required elements. I > > don't > > > think there's a use case that _can't_ be done. > > > > > > Is there a use case that can't be done with binary in place of string? > > And > > > why don't we use JSON or Variant for everything? Just because we can > use > > a > > > wider type doesn't mean that we don't benefit from communicating > > guarantees > > > like "this must be UTF-8". > > > > > > Types introduce guarantees that we can use to get better performance > and > > to > > > either prevent errors or raise them closer to where data is produced. > > > > > > For vectors, the guarantee that elements are finite gives us better > > > performance because we don't need to check each element. This also > moves > > > the error closer to where data is produced because writers are required > > to > > > verify the guarantees about each element and will detect bad values at > > > write time rather than at read time. > > > > > > Ryan > > > > > >
