Hi Kurtis, Are the bf16/bf8 widely used? I know that int8, and float16 are used by our users, and we already have Parquet logical, and physical types defined for them.
There is none for bf16/bf8, so we should first make them available there if we want to use them. Thanks, Peter <[email protected]> ezt írta (időpont: 2026. okt. 21., Sze, 1:03): > If we are discussing int8 and float16 for AI support shouldn’t we also > consider brain float (bf16/bf8) as it’s often used in AI (training and > inference) as well. > > On Oct 20, 2026, at 04:25, Péter Váry <[email protected]> wrote: > > > int8 and float16 are needed for AI Vectors. There were suggestions to > support them only in Vector types, but I think it would be a bad idea. > Let's discuss it in the upcoming Vector type sync, and we could start its > own DISCUSS thread if/once we agree on how to proceed. > > Daniel Weeks <[email protected]> ezt írta (időpont: 2026. okt. 19., H, > 17:27): > >> Peter, >> >> I think we need to have a broader discussion around adding narrower >> types. This probably deserves its own DISCUSS thread (not buried in a sync >> follow up). >> >> -Dan >> >> On Thu, Oct 15, 2026 at 2:01 PM Péter Váry <[email protected]> >> wrote: >> >>> Hi Philipp, >>> >>> We’re working on vector indexes on top of the upcoming index framework, >>> primarily for AI workloads. Supporting int8 and float16 vectors is a must >>> for these use cases. >>> >>> The main motivation is vectors, but I’m not convinced these should be >>> vector-specific types. Since int8 and float16 are well-defined scalar types >>> that Iceberg would need to support as vector elements anyway, I think it’s >>> cleaner to make them regular primitive types rather than special-case them >>> as only valid inside vectors. >>> >>> This is also the feedback I heard during the community sync and my >>> discussions at ApacheCon. >>> >>> Thanks, >>> Peter >>> >>> >>> Philipp Fischbeck via dev <[email protected]> ezt írta (időpont: >>> 2026. okt. 15., Cs, 17:54): >>> >>>> Hi Peter, >>>> >>>> I see that in the current draft, you suggest adding int8 and float16 as >>>> primitive Iceberg types overall, not just vector element types. I don't >>>> think this is the right approach because I don't see how engines would use >>>> these types outside of vectors. Do you have use cases in mind here? >>>> >>>> One alternative (as discussed in the design doc and during calls) would >>>> be to allow such types only as vector element types. We would still have to >>>> define the Parquet/Avro type mapping, but the type definition would be >>>> limited to the encoding itself, not "spilling" to other parts like bounds >>>> or schema evolution. >>>> >>>> Best, >>>> Philipp >>>> >>>> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry < >>>> [email protected]> wrote: >>>> >>>>> Talked with a bunch of people on ApacheCon, and I’d still very much >>>>> like to see int8 and float16 vector support in Iceberg. >>>>> >>>>> I’ve created a draft PR to add these types: >>>>> https://github.com/apache/iceberg/pull/18485. >>>>> Parquet already provides a logical type for int8 and a physical type >>>>> for float16, so adding support in Iceberg seems like a reasonable effort. >>>>> >>>>> I’d be happy to spearhead this work so we can get these types into V4, >>>>> if the community agrees. >>>>> >>>>> Thanks, >>>>> Peter >>>>> >>>>> >>>>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, >>>>> 2:33): >>>>> >>>>>> Thank you everyone for the discussion today! The meeting notes can be >>>>>> found here: >>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>>>> and below is a TLDR version: >>>>>> - Focus on a 1D numeric vector type with fixed length and non-null, >>>>>> finite coordinates; +0 and -0 remain valid. >>>>>> - Keep the initial spec simple and build on Parquet’s emerging vector >>>>>> representation. >>>>>> - Initially support int, long, float, and double; defer >>>>>> narrower/special types such as int8 as additional Iceberg primitive types >>>>>> (while allowing such parquet representation for vector). >>>>>> - Keep higher level properties such as distance functions out of >>>>>> scope, and revisit them with the index design. Sparse vectors are also >>>>>> out >>>>>> of scope for now: alternative options exist and waiting for concrete use >>>>>> cases. >>>>>> - For formats without native vector support, use a list >>>>>> representation with writer enforcement. Will explore an Avro vector >>>>>> logical >>>>>> type with Avro community; ORC support remains open. >>>>>> >>>>>> I'll update the spec PR in the next few days (also appreciate the >>>>>> quick feedback from everyone!) >>>>>> >>>>>> Thanks, >>>>>> Yan >>>>>> >>>>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >>>>>> >>>>>>> Thank you Steven for helping secure the time slot for the vector >>>>>>> type sync! And thanks everyone for your feedback on the design doc. >>>>>>> >>>>>>> In preparation for tomorrow's discussion, I've created a draft PR >>>>>>> https://github.com/apache/iceberg/pull/18416 for the spec changes >>>>>>> based on the progress in the parquet community on the Vector type >>>>>>> proposal [ >>>>>>> 1 <https://github.com/apache/parquet-format/pull/624>] [2 >>>>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>>>> , and will continue to iterate on both the spec PR and the design doc >>>>>>> based >>>>>>> on the discussion feedback. >>>>>>> >>>>>>> Thanks, >>>>>>> Yan >>>>>>> >>>>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> As some of you are aware, there is an ongoing discussion about a >>>>>>>> new *Vector* type in both the Parquet and Iceberg communities. >>>>>>>> >>>>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>>>> >>>>>>>> To accelerate the Vector type discussion, we are reusing/extending >>>>>>>> Thursday's biweekly series to cover both File and Vector types >>>>>>>> (*starting >>>>>>>> on Oct 8*). Since the file type discussion is reaching consensus, more >>>>>>>> time will likely be spent on the Vector type discussion. The >>>>>>>> corresponding >>>>>>>> dev calendar event title has been updated. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Steven >>>>>>>> >>>>>>>
