Hi folks, As discussed in the last sync, I created a basic type support PR for vector in Iceberg Java [1]. In this PR, treating Iceberg vectors as a primitive type felt natural. When I prototyped some Parquet plumbing, the mismatch (Iceberg primitive vs. Parquet nested) led to a few special cases, but these would require handling regardless. So I still favor treating the Iceberg vector as a primitive type, but I'm indifferent about subcategories (simple vs. complex).
Best, Philipp [1] https://github.com/apache/iceberg/pull/18676 On Mon, Oct 26, 2026 at 8:50 AM Péter Váry <[email protected]> wrote: > One thing to highlight: to speed up writes, I turned off Parquet stats for > vector columns. Iceberg doesn't use them, and they cost up to ~30% of write > time. So we need table- or schema-level info to decide whether a column is > int8 or float16, and column stats do not work. > > On Sat, Oct 24, 2026, 23:32 Péter Váry <[email protected]> > wrote: > >> Hi all, >> >> As discussed at the last Vector sync, I wrote some code and ran >> benchmarks to measure how well Parquet handles embeddings stored as FLOAT >> and as FLOAT16. >> >> *ALP*: Parquet Java doesn't support ALP yet, so I prototyped it. It >> doesn't compress these FLOAT16 embeddings well: >> >> - Plain ALP needs 36.5 bits/value even with zstd, which is more than >> raw float32. >> - ALP_rd + zstd needs 17.9 bits/value, 18% more than dictionary >> encoding (15.1). >> - Dictionary encoding needs 15.1 bits/value even without zstd: the >> embeddings have only ~24k distinct values, so every value becomes a 15-bit >> id. >> >> *I have ruled out using ALP.* >> >> *Read/write/heap*: Java has no native float16 type yet ( >> https://openjdk.org/jeps/508), so I expect Java users to store float16 >> values in shorts for now. The benchmark therefore uses float[] and short[] >> as the in-memory embeddings. >> >> Optimizations I made for both FLOAT and FLOAT16: >> >> - FLOAT16 values are decoded straight from the page, without >> allocating a Binary per value >> - Bulk reads of float/float16 lists, with faster PLAIN, >> BYTE_STREAM_SPLIT and dictionary decoding >> - Fixed-size lists: repetition/definition levels are derived instead >> of decoded >> - About 20% faster zstd page decompression >> - Column/size statistics turned off for columns nested in lists/maps, >> which Iceberg doesn't use anyway >> >> Results on my MacBook Pro (M4 Pro, JDK 21), with 100,000 embeddings of >> 2,560 dimensions in local files: >> >> - *Smallest*: FLOAT16 + BSS + zstd is 10% smaller than FLOAT + DICT + >> zstd (415 vs 459 MiB), and also reads 22% faster (284 vs 362 ms). >> - *Fastest reads*: FLOAT16 + PLAIN + uncompressed reads 26% faster >> than FLOAT + PLAIN + uncompressed (134 vs 180 ms), at half the file size. >> On blob storage the extra transfer would increase the gap even more. >> - *Overall*: the best FLOAT setup is DICT + uncompressed, and the >> best FLOAT16 setup is PLAIN + uncompressed. They have similar sizes (459 >> vs >> 489 MiB), but FLOAT16 reads 57% faster (134 vs 312 ms), writes 55% faster, >> and uses half the heap. >> >> *I clearly see a lot to gain from Parquet's FLOAT16 type.* >> >> *ALP numbers*: >> storage encoding bits/value bits/value + zstd details >> *FLOAT (32-bit)* *DICTIONARY* *15.16* *15.13* *23,908 distinct values, >> 15-bit ids, 93 KiB dictionary* >> FLOAT (32-bit) ALP 39.77 36.47 e=10, f=0 in every vector, avg 31.0-bit >> integers, 18.06% exceptions >> FLOAT (32-bit) ALP_rd 27.52 17.89 8 left + 24 right bits, 2.09% >> exceptions >> FLOAT16 (16-bit) DICTIONARY 15.09 15.09 23,908 distinct values, 15-bit >> ids, 47 KiB dictionary >> FLOAT16 (16-bit) ALP_rd (not part of ALP) 14.46 14.46 5 left + 11 right >> bits, 2.09% exceptions >> >> *Embedding benchmark numbers*: >> source / storage encoding codec file MiB write ms read ms heap MiB >> float[] / FLOAT PLAIN uncompressed 977.1 1,707 ±48 180 ±5 979 >> *float[] / FLOAT* *DICT* *uncompressed* *459.2* *4,119 ±12* *312 ±2* >> *980* >> float[] / FLOAT PLAIN zstd 579.2 5,730 ±30 950 ±3 980 >> float[] / FLOAT DICT zstd 459.1 4,304 ±16 362 ±2 979 >> *short[] / FLOAT16* *PLAIN* *uncompressed* *488.8* *1,871 ±8* *134 ±27* >> *491* >> short[] / FLOAT16 BSS uncompressed 488.8 2,813 ±17 156 ±16 491 >> short[] / FLOAT16 PLAIN zstd 449.3 2,214 ±8 444 ±15 490 >> short[] / FLOAT16 BSS zstd 415.0 3,003 ±33 284 ±17 491 >> >> Code: https://github.com/pvary/iceberg/commits/parquet_float16_benchmark/ >> >> To run the benchmarks: >> >> *./gradlew :iceberg-data:jmh * >> *-PjmhIncludeRegex=ParquetFloat16EmbeddingBenchmark* >> >> *./gradlew :iceberg-data:jmh -PjmhIncludeRegex=AlpPrototype* >> >> >> The first command runs all 24 combinations, which takes about 30 minutes. >> The ALP prototype takes under a minute. >> >> Thanks, Peter >> >> Russell Spitzer <[email protected]> ezt írta (időpont: 2026. >> okt. 21., Sze, 17:38): >> >>> I think all these expansions of types understood by Iceberg belong in a >>> separate discussion. I think vector basically can exist >>> completely independently of introducing more subtypes for Vectors. I think >>> there is probably a lot more discussion needed regarding which specific >>> variants for sub-width types we want to support and which engines are >>> actually capable of supporting them. I would get Vector in first, then >>> expand the spec later with additional types. >>> >>> My ideal path forward would be >>> >>> 1. Vote in current vector type (regardless of what is happening in >>> Parquet since I don't think that really changes what we want to allow here, >>> just the complexity of the implementation) >>> 2. Start a new discussion about other vector subtypes. >>> >>> On Wed, Oct 21, 2026 at 6:43 AM Péter Váry <[email protected]> >>> wrote: >>> >>>> Hi Kurtis, >>>> Are the bf16/bf8 widely used? >>>> I know that int8, and float16 are used by our users, and we already >>>> have Parquet logical, and physical types defined for them. >>>> >>>> There is none for bf16/bf8, so we should first make them available >>>> there if we want to use them. >>>> >>>> Thanks, >>>> Peter >>>> >>>> <[email protected]> ezt írta (időpont: 2026. okt. 21., Sze, >>>> 1:03): >>>> >>>>> If we are discussing int8 and float16 for AI support shouldn’t we also >>>>> consider brain float (bf16/bf8) as it’s often used in AI (training and >>>>> inference) as well. >>>>> >>>>> On Oct 20, 2026, at 04:25, Péter Váry <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> int8 and float16 are needed for AI Vectors. There were suggestions to >>>>> support them only in Vector types, but I think it would be a bad idea. >>>>> Let's discuss it in the upcoming Vector type sync, and we could start >>>>> its own DISCUSS thread if/once we agree on how to proceed. >>>>> >>>>> Daniel Weeks <[email protected]> ezt írta (időpont: 2026. okt. 19., >>>>> H, 17:27): >>>>> >>>>>> Peter, >>>>>> >>>>>> I think we need to have a broader discussion around adding narrower >>>>>> types. This probably deserves its own DISCUSS thread (not buried in a >>>>>> sync >>>>>> follow up). >>>>>> >>>>>> -Dan >>>>>> >>>>>> On Thu, Oct 15, 2026 at 2:01 PM Péter Váry < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Philipp, >>>>>>> >>>>>>> We’re working on vector indexes on top of the upcoming index >>>>>>> framework, primarily for AI workloads. Supporting int8 and float16 >>>>>>> vectors >>>>>>> is a must for these use cases. >>>>>>> >>>>>>> The main motivation is vectors, but I’m not convinced these should >>>>>>> be vector-specific types. Since int8 and float16 are well-defined scalar >>>>>>> types that Iceberg would need to support as vector elements anyway, I >>>>>>> think >>>>>>> it’s cleaner to make them regular primitive types rather than >>>>>>> special-case >>>>>>> them as only valid inside vectors. >>>>>>> >>>>>>> This is also the feedback I heard during the community sync and my >>>>>>> discussions at ApacheCon. >>>>>>> >>>>>>> Thanks, >>>>>>> Peter >>>>>>> >>>>>>> >>>>>>> Philipp Fischbeck via dev <[email protected]> ezt írta >>>>>>> (időpont: 2026. okt. 15., Cs, 17:54): >>>>>>> >>>>>>>> Hi Peter, >>>>>>>> >>>>>>>> I see that in the current draft, you suggest adding int8 and >>>>>>>> float16 as primitive Iceberg types overall, not just vector element >>>>>>>> types. >>>>>>>> I don't think this is the right approach because I don't see how >>>>>>>> engines >>>>>>>> would use these types outside of vectors. Do you have use cases in mind >>>>>>>> here? >>>>>>>> >>>>>>>> One alternative (as discussed in the design doc and during calls) >>>>>>>> would be to allow such types only as vector element types. We would >>>>>>>> still >>>>>>>> have to define the Parquet/Avro type mapping, but the type definition >>>>>>>> would >>>>>>>> be limited to the encoding itself, not "spilling" to other parts like >>>>>>>> bounds or schema evolution. >>>>>>>> >>>>>>>> Best, >>>>>>>> Philipp >>>>>>>> >>>>>>>> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Talked with a bunch of people on ApacheCon, and I’d still very >>>>>>>>> much like to see int8 and float16 vector support in Iceberg. >>>>>>>>> >>>>>>>>> I’ve created a draft PR to add these types: >>>>>>>>> https://github.com/apache/iceberg/pull/18485. >>>>>>>>> Parquet already provides a logical type for int8 and a physical >>>>>>>>> type for float16, so adding support in Iceberg seems like a reasonable >>>>>>>>> effort. >>>>>>>>> >>>>>>>>> I’d be happy to spearhead this work so we can get these types into >>>>>>>>> V4, if the community agrees. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Peter >>>>>>>>> >>>>>>>>> >>>>>>>>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, >>>>>>>>> 2:33): >>>>>>>>> >>>>>>>>>> Thank you everyone for the discussion today! The meeting notes >>>>>>>>>> can be found here: >>>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>>>>>>>> and below is a TLDR version: >>>>>>>>>> - Focus on a 1D numeric vector type with fixed length and >>>>>>>>>> non-null, finite coordinates; +0 and -0 remain valid. >>>>>>>>>> - Keep the initial spec simple and build on Parquet’s emerging >>>>>>>>>> vector representation. >>>>>>>>>> - Initially support int, long, float, and double; defer >>>>>>>>>> narrower/special types such as int8 as additional Iceberg primitive >>>>>>>>>> types >>>>>>>>>> (while allowing such parquet representation for vector). >>>>>>>>>> - Keep higher level properties such as distance functions out of >>>>>>>>>> scope, and revisit them with the index design. Sparse vectors are >>>>>>>>>> also out >>>>>>>>>> of scope for now: alternative options exist and waiting for concrete >>>>>>>>>> use >>>>>>>>>> cases. >>>>>>>>>> - For formats without native vector support, use a list >>>>>>>>>> representation with writer enforcement. Will explore an Avro vector >>>>>>>>>> logical >>>>>>>>>> type with Avro community; ORC support remains open. >>>>>>>>>> >>>>>>>>>> I'll update the spec PR in the next few days (also appreciate the >>>>>>>>>> quick feedback from everyone!) >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Yan >>>>>>>>>> >>>>>>>>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thank you Steven for helping secure the time slot for the vector >>>>>>>>>>> type sync! And thanks everyone for your feedback on the design doc. >>>>>>>>>>> >>>>>>>>>>> In preparation for tomorrow's discussion, I've created a draft >>>>>>>>>>> PR https://github.com/apache/iceberg/pull/18416 for the spec >>>>>>>>>>> changes based on the progress in the parquet community on the >>>>>>>>>>> Vector type >>>>>>>>>>> proposal [1 <https://github.com/apache/parquet-format/pull/624>] >>>>>>>>>>> [2 >>>>>>>>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>>>>>>>> , and will continue to iterate on both the spec PR and the design >>>>>>>>>>> doc based >>>>>>>>>>> on the discussion feedback. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Yan >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> As some of you are aware, there is an ongoing discussion about >>>>>>>>>>>> a new *Vector* type in both the Parquet and Iceberg >>>>>>>>>>>> communities. >>>>>>>>>>>> >>>>>>>>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>>>>>>>> >>>>>>>>>>>> To accelerate the Vector type discussion, we are >>>>>>>>>>>> reusing/extending Thursday's biweekly series to cover both File >>>>>>>>>>>> and Vector >>>>>>>>>>>> types (*starting on Oct 8*). Since the file type discussion is >>>>>>>>>>>> reaching consensus, more time will likely be spent on the >>>>>>>>>>>> Vector type discussion. The corresponding dev calendar event title >>>>>>>>>>>> has been >>>>>>>>>>>> updated. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Steven >>>>>>>>>>>> >>>>>>>>>>>
