Primitive type feels most natural for me since we aren't actually attempting to allow cell manipulation within a vector (or are we?) but since I assume most consumers take the entire object as a whole, the primitive type feels like the correct representation to me.
On Wed, Oct 28, 2026 at 8:50 AM Philipp Fischbeck via dev < [email protected]> wrote: > Hi folks, > > As discussed in the last sync, I created a basic type support PR for > vector in Iceberg Java [1]. In this PR, treating Iceberg vectors as a > primitive type felt natural. When I prototyped some Parquet plumbing, the > mismatch (Iceberg primitive vs. Parquet nested) led to a few special cases, > but these would require handling regardless. So I still favor treating the > Iceberg vector as a primitive type, but I'm indifferent about subcategories > (simple vs. complex). > > Best, Philipp > > [1] https://github.com/apache/iceberg/pull/18676 > > On Mon, Oct 26, 2026 at 8:50 AM Péter Váry <[email protected]> > wrote: > >> One thing to highlight: to speed up writes, I turned off Parquet stats >> for vector columns. Iceberg doesn't use them, and they cost up to ~30% of >> write time. So we need table- or schema-level info to decide whether a >> column is int8 or float16, and column stats do not work. >> >> On Sat, Oct 24, 2026, 23:32 Péter Váry <[email protected]> >> wrote: >> >>> Hi all, >>> >>> As discussed at the last Vector sync, I wrote some code and ran >>> benchmarks to measure how well Parquet handles embeddings stored as FLOAT >>> and as FLOAT16. >>> >>> *ALP*: Parquet Java doesn't support ALP yet, so I prototyped it. It >>> doesn't compress these FLOAT16 embeddings well: >>> >>> - Plain ALP needs 36.5 bits/value even with zstd, which is more than >>> raw float32. >>> - ALP_rd + zstd needs 17.9 bits/value, 18% more than dictionary >>> encoding (15.1). >>> - Dictionary encoding needs 15.1 bits/value even without zstd: the >>> embeddings have only ~24k distinct values, so every value becomes a >>> 15-bit >>> id. >>> >>> *I have ruled out using ALP.* >>> >>> *Read/write/heap*: Java has no native float16 type yet ( >>> https://openjdk.org/jeps/508), so I expect Java users to store float16 >>> values in shorts for now. The benchmark therefore uses float[] and short[] >>> as the in-memory embeddings. >>> >>> Optimizations I made for both FLOAT and FLOAT16: >>> >>> - FLOAT16 values are decoded straight from the page, without >>> allocating a Binary per value >>> - Bulk reads of float/float16 lists, with faster PLAIN, >>> BYTE_STREAM_SPLIT and dictionary decoding >>> - Fixed-size lists: repetition/definition levels are derived instead >>> of decoded >>> - About 20% faster zstd page decompression >>> - Column/size statistics turned off for columns nested in >>> lists/maps, which Iceberg doesn't use anyway >>> >>> Results on my MacBook Pro (M4 Pro, JDK 21), with 100,000 embeddings of >>> 2,560 dimensions in local files: >>> >>> - *Smallest*: FLOAT16 + BSS + zstd is 10% smaller than FLOAT + DICT >>> + zstd (415 vs 459 MiB), and also reads 22% faster (284 vs 362 ms). >>> - *Fastest reads*: FLOAT16 + PLAIN + uncompressed reads 26% faster >>> than FLOAT + PLAIN + uncompressed (134 vs 180 ms), at half the file size. >>> On blob storage the extra transfer would increase the gap even more. >>> - *Overall*: the best FLOAT setup is DICT + uncompressed, and the >>> best FLOAT16 setup is PLAIN + uncompressed. They have similar sizes (459 >>> vs >>> 489 MiB), but FLOAT16 reads 57% faster (134 vs 312 ms), writes 55% >>> faster, >>> and uses half the heap. >>> >>> *I clearly see a lot to gain from Parquet's FLOAT16 type.* >>> >>> *ALP numbers*: >>> storage encoding bits/value bits/value + zstd details >>> *FLOAT (32-bit)* *DICTIONARY* *15.16* *15.13* *23,908 distinct values, >>> 15-bit ids, 93 KiB dictionary* >>> FLOAT (32-bit) ALP 39.77 36.47 e=10, f=0 in every vector, avg 31.0-bit >>> integers, 18.06% exceptions >>> FLOAT (32-bit) ALP_rd 27.52 17.89 8 left + 24 right bits, 2.09% >>> exceptions >>> FLOAT16 (16-bit) DICTIONARY 15.09 15.09 23,908 distinct values, 15-bit >>> ids, 47 KiB dictionary >>> FLOAT16 (16-bit) ALP_rd (not part of ALP) 14.46 14.46 5 left + 11 right >>> bits, 2.09% exceptions >>> >>> *Embedding benchmark numbers*: >>> source / storage encoding codec file MiB write ms read ms heap MiB >>> float[] / FLOAT PLAIN uncompressed 977.1 1,707 ±48 180 ±5 979 >>> *float[] / FLOAT* *DICT* *uncompressed* *459.2* *4,119 ±12* *312 ±2* >>> *980* >>> float[] / FLOAT PLAIN zstd 579.2 5,730 ±30 950 ±3 980 >>> float[] / FLOAT DICT zstd 459.1 4,304 ±16 362 ±2 979 >>> *short[] / FLOAT16* *PLAIN* *uncompressed* *488.8* *1,871 ±8* *134 ±27* >>> *491* >>> short[] / FLOAT16 BSS uncompressed 488.8 2,813 ±17 156 ±16 491 >>> short[] / FLOAT16 PLAIN zstd 449.3 2,214 ±8 444 ±15 490 >>> short[] / FLOAT16 BSS zstd 415.0 3,003 ±33 284 ±17 491 >>> >>> Code: >>> https://github.com/pvary/iceberg/commits/parquet_float16_benchmark/ >>> >>> To run the benchmarks: >>> >>> *./gradlew :iceberg-data:jmh * >>> *-PjmhIncludeRegex=ParquetFloat16EmbeddingBenchmark* >>> >>> *./gradlew :iceberg-data:jmh -PjmhIncludeRegex=AlpPrototype* >>> >>> >>> The first command runs all 24 combinations, which takes about 30 >>> minutes. The ALP prototype takes under a minute. >>> >>> Thanks, Peter >>> >>> Russell Spitzer <[email protected]> ezt írta (időpont: 2026. >>> okt. 21., Sze, 17:38): >>> >>>> I think all these expansions of types understood by Iceberg belong in a >>>> separate discussion. I think vector basically can exist >>>> completely independently of introducing more subtypes for Vectors. I think >>>> there is probably a lot more discussion needed regarding which specific >>>> variants for sub-width types we want to support and which engines are >>>> actually capable of supporting them. I would get Vector in first, then >>>> expand the spec later with additional types. >>>> >>>> My ideal path forward would be >>>> >>>> 1. Vote in current vector type (regardless of what is happening in >>>> Parquet since I don't think that really changes what we want to allow here, >>>> just the complexity of the implementation) >>>> 2. Start a new discussion about other vector subtypes. >>>> >>>> On Wed, Oct 21, 2026 at 6:43 AM Péter Váry <[email protected]> >>>> wrote: >>>> >>>>> Hi Kurtis, >>>>> Are the bf16/bf8 widely used? >>>>> I know that int8, and float16 are used by our users, and we already >>>>> have Parquet logical, and physical types defined for them. >>>>> >>>>> There is none for bf16/bf8, so we should first make them available >>>>> there if we want to use them. >>>>> >>>>> Thanks, >>>>> Peter >>>>> >>>>> <[email protected]> ezt írta (időpont: 2026. okt. 21., Sze, >>>>> 1:03): >>>>> >>>>>> If we are discussing int8 and float16 for AI support shouldn’t we >>>>>> also consider brain float (bf16/bf8) as it’s often used in AI (training >>>>>> and >>>>>> inference) as well. >>>>>> >>>>>> On Oct 20, 2026, at 04:25, Péter Váry <[email protected]> >>>>>> wrote: >>>>>> >>>>>> >>>>>> int8 and float16 are needed for AI Vectors. There were suggestions >>>>>> to support them only in Vector types, but I think it would be a bad idea. >>>>>> Let's discuss it in the upcoming Vector type sync, and we could start >>>>>> its own DISCUSS thread if/once we agree on how to proceed. >>>>>> >>>>>> Daniel Weeks <[email protected]> ezt írta (időpont: 2026. okt. 19., >>>>>> H, 17:27): >>>>>> >>>>>>> Peter, >>>>>>> >>>>>>> I think we need to have a broader discussion around adding narrower >>>>>>> types. This probably deserves its own DISCUSS thread (not buried in a >>>>>>> sync >>>>>>> follow up). >>>>>>> >>>>>>> -Dan >>>>>>> >>>>>>> On Thu, Oct 15, 2026 at 2:01 PM Péter Váry < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Philipp, >>>>>>>> >>>>>>>> We’re working on vector indexes on top of the upcoming index >>>>>>>> framework, primarily for AI workloads. Supporting int8 and float16 >>>>>>>> vectors >>>>>>>> is a must for these use cases. >>>>>>>> >>>>>>>> The main motivation is vectors, but I’m not convinced these should >>>>>>>> be vector-specific types. Since int8 and float16 are well-defined >>>>>>>> scalar >>>>>>>> types that Iceberg would need to support as vector elements anyway, I >>>>>>>> think >>>>>>>> it’s cleaner to make them regular primitive types rather than >>>>>>>> special-case >>>>>>>> them as only valid inside vectors. >>>>>>>> >>>>>>>> This is also the feedback I heard during the community sync and my >>>>>>>> discussions at ApacheCon. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Peter >>>>>>>> >>>>>>>> >>>>>>>> Philipp Fischbeck via dev <[email protected]> ezt írta >>>>>>>> (időpont: 2026. okt. 15., Cs, 17:54): >>>>>>>> >>>>>>>>> Hi Peter, >>>>>>>>> >>>>>>>>> I see that in the current draft, you suggest adding int8 and >>>>>>>>> float16 as primitive Iceberg types overall, not just vector element >>>>>>>>> types. >>>>>>>>> I don't think this is the right approach because I don't see how >>>>>>>>> engines >>>>>>>>> would use these types outside of vectors. Do you have use cases in >>>>>>>>> mind >>>>>>>>> here? >>>>>>>>> >>>>>>>>> One alternative (as discussed in the design doc and during calls) >>>>>>>>> would be to allow such types only as vector element types. We would >>>>>>>>> still >>>>>>>>> have to define the Parquet/Avro type mapping, but the type definition >>>>>>>>> would >>>>>>>>> be limited to the encoding itself, not "spilling" to other parts like >>>>>>>>> bounds or schema evolution. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Philipp >>>>>>>>> >>>>>>>>> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Talked with a bunch of people on ApacheCon, and I’d still very >>>>>>>>>> much like to see int8 and float16 vector support in Iceberg. >>>>>>>>>> >>>>>>>>>> I’ve created a draft PR to add these types: >>>>>>>>>> https://github.com/apache/iceberg/pull/18485. >>>>>>>>>> Parquet already provides a logical type for int8 and a physical >>>>>>>>>> type for float16, so adding support in Iceberg seems like a >>>>>>>>>> reasonable >>>>>>>>>> effort. >>>>>>>>>> >>>>>>>>>> I’d be happy to spearhead this work so we can get these types >>>>>>>>>> into V4, if the community agrees. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Peter >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., >>>>>>>>>> P, 2:33): >>>>>>>>>> >>>>>>>>>>> Thank you everyone for the discussion today! The meeting notes >>>>>>>>>>> can be found here: >>>>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>>>>>>>>> and below is a TLDR version: >>>>>>>>>>> - Focus on a 1D numeric vector type with fixed length and >>>>>>>>>>> non-null, finite coordinates; +0 and -0 remain valid. >>>>>>>>>>> - Keep the initial spec simple and build on Parquet’s emerging >>>>>>>>>>> vector representation. >>>>>>>>>>> - Initially support int, long, float, and double; defer >>>>>>>>>>> narrower/special types such as int8 as additional Iceberg primitive >>>>>>>>>>> types >>>>>>>>>>> (while allowing such parquet representation for vector). >>>>>>>>>>> - Keep higher level properties such as distance functions out of >>>>>>>>>>> scope, and revisit them with the index design. Sparse vectors are >>>>>>>>>>> also out >>>>>>>>>>> of scope for now: alternative options exist and waiting for >>>>>>>>>>> concrete use >>>>>>>>>>> cases. >>>>>>>>>>> - For formats without native vector support, use a list >>>>>>>>>>> representation with writer enforcement. Will explore an Avro vector >>>>>>>>>>> logical >>>>>>>>>>> type with Avro community; ORC support remains open. >>>>>>>>>>> >>>>>>>>>>> I'll update the spec PR in the next few days (also appreciate >>>>>>>>>>> the quick feedback from everyone!) >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Yan >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank you Steven for helping secure the time slot for the >>>>>>>>>>>> vector type sync! And thanks everyone for your feedback on the >>>>>>>>>>>> design doc. >>>>>>>>>>>> >>>>>>>>>>>> In preparation for tomorrow's discussion, I've created a draft >>>>>>>>>>>> PR https://github.com/apache/iceberg/pull/18416 for the spec >>>>>>>>>>>> changes based on the progress in the parquet community on the >>>>>>>>>>>> Vector type >>>>>>>>>>>> proposal [1 <https://github.com/apache/parquet-format/pull/624>] >>>>>>>>>>>> [2 >>>>>>>>>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>>>>>>>>> , and will continue to iterate on both the spec PR and the design >>>>>>>>>>>> doc based >>>>>>>>>>>> on the discussion feedback. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Yan >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> As some of you are aware, there is an ongoing discussion about >>>>>>>>>>>>> a new *Vector* type in both the Parquet and Iceberg >>>>>>>>>>>>> communities. >>>>>>>>>>>>> >>>>>>>>>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>>>>>>>>> >>>>>>>>>>>>> To accelerate the Vector type discussion, we are >>>>>>>>>>>>> reusing/extending Thursday's biweekly series to cover both File >>>>>>>>>>>>> and Vector >>>>>>>>>>>>> types (*starting on Oct 8*). Since the file type discussion >>>>>>>>>>>>> is reaching consensus, more time will likely be spent on the >>>>>>>>>>>>> Vector type discussion. The corresponding dev calendar event >>>>>>>>>>>>> title has been >>>>>>>>>>>>> updated. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Steven >>>>>>>>>>>>> >>>>>>>>>>>>
