I think all these expansions of types understood by Iceberg belong in a separate discussion. I think vector basically can exist completely independently of introducing more subtypes for Vectors. I think there is probably a lot more discussion needed regarding which specific variants for sub-width types we want to support and which engines are actually capable of supporting them. I would get Vector in first, then expand the spec later with additional types.
My ideal path forward would be 1. Vote in current vector type (regardless of what is happening in Parquet since I don't think that really changes what we want to allow here, just the complexity of the implementation) 2. Start a new discussion about other vector subtypes. On Wed, Oct 21, 2026 at 6:43 AM Péter Váry <[email protected]> wrote: > Hi Kurtis, > Are the bf16/bf8 widely used? > I know that int8, and float16 are used by our users, and we already have > Parquet logical, and physical types defined for them. > > There is none for bf16/bf8, so we should first make them available there > if we want to use them. > > Thanks, > Peter > > <[email protected]> ezt írta (időpont: 2026. okt. 21., Sze, 1:03): > >> If we are discussing int8 and float16 for AI support shouldn’t we also >> consider brain float (bf16/bf8) as it’s often used in AI (training and >> inference) as well. >> >> On Oct 20, 2026, at 04:25, Péter Váry <[email protected]> >> wrote: >> >> >> int8 and float16 are needed for AI Vectors. There were suggestions to >> support them only in Vector types, but I think it would be a bad idea. >> Let's discuss it in the upcoming Vector type sync, and we could start its >> own DISCUSS thread if/once we agree on how to proceed. >> >> Daniel Weeks <[email protected]> ezt írta (időpont: 2026. okt. 19., H, >> 17:27): >> >>> Peter, >>> >>> I think we need to have a broader discussion around adding narrower >>> types. This probably deserves its own DISCUSS thread (not buried in a sync >>> follow up). >>> >>> -Dan >>> >>> On Thu, Oct 15, 2026 at 2:01 PM Péter Váry <[email protected]> >>> wrote: >>> >>>> Hi Philipp, >>>> >>>> We’re working on vector indexes on top of the upcoming index framework, >>>> primarily for AI workloads. Supporting int8 and float16 vectors is a must >>>> for these use cases. >>>> >>>> The main motivation is vectors, but I’m not convinced these should be >>>> vector-specific types. Since int8 and float16 are well-defined scalar types >>>> that Iceberg would need to support as vector elements anyway, I think it’s >>>> cleaner to make them regular primitive types rather than special-case them >>>> as only valid inside vectors. >>>> >>>> This is also the feedback I heard during the community sync and my >>>> discussions at ApacheCon. >>>> >>>> Thanks, >>>> Peter >>>> >>>> >>>> Philipp Fischbeck via dev <[email protected]> ezt írta (időpont: >>>> 2026. okt. 15., Cs, 17:54): >>>> >>>>> Hi Peter, >>>>> >>>>> I see that in the current draft, you suggest adding int8 and float16 >>>>> as primitive Iceberg types overall, not just vector element types. I don't >>>>> think this is the right approach because I don't see how engines would use >>>>> these types outside of vectors. Do you have use cases in mind here? >>>>> >>>>> One alternative (as discussed in the design doc and during calls) >>>>> would be to allow such types only as vector element types. We would still >>>>> have to define the Parquet/Avro type mapping, but the type definition >>>>> would >>>>> be limited to the encoding itself, not "spilling" to other parts like >>>>> bounds or schema evolution. >>>>> >>>>> Best, >>>>> Philipp >>>>> >>>>> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry < >>>>> [email protected]> wrote: >>>>> >>>>>> Talked with a bunch of people on ApacheCon, and I’d still very much >>>>>> like to see int8 and float16 vector support in Iceberg. >>>>>> >>>>>> I’ve created a draft PR to add these types: >>>>>> https://github.com/apache/iceberg/pull/18485. >>>>>> Parquet already provides a logical type for int8 and a physical type >>>>>> for float16, so adding support in Iceberg seems like a reasonable effort. >>>>>> >>>>>> I’d be happy to spearhead this work so we can get these types into >>>>>> V4, if the community agrees. >>>>>> >>>>>> Thanks, >>>>>> Peter >>>>>> >>>>>> >>>>>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, >>>>>> 2:33): >>>>>> >>>>>>> Thank you everyone for the discussion today! The meeting notes can >>>>>>> be found here: >>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>>>>> and below is a TLDR version: >>>>>>> - Focus on a 1D numeric vector type with fixed length and non-null, >>>>>>> finite coordinates; +0 and -0 remain valid. >>>>>>> - Keep the initial spec simple and build on Parquet’s emerging >>>>>>> vector representation. >>>>>>> - Initially support int, long, float, and double; defer >>>>>>> narrower/special types such as int8 as additional Iceberg primitive >>>>>>> types >>>>>>> (while allowing such parquet representation for vector). >>>>>>> - Keep higher level properties such as distance functions out of >>>>>>> scope, and revisit them with the index design. Sparse vectors are also >>>>>>> out >>>>>>> of scope for now: alternative options exist and waiting for concrete use >>>>>>> cases. >>>>>>> - For formats without native vector support, use a list >>>>>>> representation with writer enforcement. Will explore an Avro vector >>>>>>> logical >>>>>>> type with Avro community; ORC support remains open. >>>>>>> >>>>>>> I'll update the spec PR in the next few days (also appreciate the >>>>>>> quick feedback from everyone!) >>>>>>> >>>>>>> Thanks, >>>>>>> Yan >>>>>>> >>>>>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >>>>>>> >>>>>>>> Thank you Steven for helping secure the time slot for the vector >>>>>>>> type sync! And thanks everyone for your feedback on the design doc. >>>>>>>> >>>>>>>> In preparation for tomorrow's discussion, I've created a draft PR >>>>>>>> https://github.com/apache/iceberg/pull/18416 for the spec changes >>>>>>>> based on the progress in the parquet community on the Vector type >>>>>>>> proposal [ >>>>>>>> 1 <https://github.com/apache/parquet-format/pull/624>] [2 >>>>>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>>>>> , and will continue to iterate on both the spec PR and the design doc >>>>>>>> based >>>>>>>> on the discussion feedback. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Yan >>>>>>>> >>>>>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> As some of you are aware, there is an ongoing discussion about a >>>>>>>>> new *Vector* type in both the Parquet and Iceberg communities. >>>>>>>>> >>>>>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>>>>> >>>>>>>>> To accelerate the Vector type discussion, we are >>>>>>>>> reusing/extending Thursday's biweekly series to cover both File and >>>>>>>>> Vector >>>>>>>>> types (*starting on Oct 8*). Since the file type discussion is >>>>>>>>> reaching consensus, more time will likely be spent on the Vector >>>>>>>>> type discussion. The corresponding dev calendar event title has been >>>>>>>>> updated. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Steven >>>>>>>>> >>>>>>>>
