One thing to highlight: to speed up writes, I turned off Parquet stats for vector columns. Iceberg doesn't use them, and they cost up to ~30% of write time. So we need table- or schema-level info to decide whether a column is int8 or float16, and column stats do not work.
On Sat, Oct 24, 2026, 23:32 Péter Váry <[email protected]> wrote: > Hi all, > > As discussed at the last Vector sync, I wrote some code and ran benchmarks > to measure how well Parquet handles embeddings stored as FLOAT and as > FLOAT16. > > *ALP*: Parquet Java doesn't support ALP yet, so I prototyped it. It > doesn't compress these FLOAT16 embeddings well: > > - Plain ALP needs 36.5 bits/value even with zstd, which is more than > raw float32. > - ALP_rd + zstd needs 17.9 bits/value, 18% more than dictionary > encoding (15.1). > - Dictionary encoding needs 15.1 bits/value even without zstd: the > embeddings have only ~24k distinct values, so every value becomes a 15-bit > id. > > *I have ruled out using ALP.* > > *Read/write/heap*: Java has no native float16 type yet ( > https://openjdk.org/jeps/508), so I expect Java users to store float16 > values in shorts for now. The benchmark therefore uses float[] and short[] > as the in-memory embeddings. > > Optimizations I made for both FLOAT and FLOAT16: > > - FLOAT16 values are decoded straight from the page, without > allocating a Binary per value > - Bulk reads of float/float16 lists, with faster PLAIN, > BYTE_STREAM_SPLIT and dictionary decoding > - Fixed-size lists: repetition/definition levels are derived instead > of decoded > - About 20% faster zstd page decompression > - Column/size statistics turned off for columns nested in lists/maps, > which Iceberg doesn't use anyway > > Results on my MacBook Pro (M4 Pro, JDK 21), with 100,000 embeddings of > 2,560 dimensions in local files: > > - *Smallest*: FLOAT16 + BSS + zstd is 10% smaller than FLOAT + DICT + > zstd (415 vs 459 MiB), and also reads 22% faster (284 vs 362 ms). > - *Fastest reads*: FLOAT16 + PLAIN + uncompressed reads 26% faster > than FLOAT + PLAIN + uncompressed (134 vs 180 ms), at half the file size. > On blob storage the extra transfer would increase the gap even more. > - *Overall*: the best FLOAT setup is DICT + uncompressed, and the best > FLOAT16 setup is PLAIN + uncompressed. They have similar sizes (459 vs 489 > MiB), but FLOAT16 reads 57% faster (134 vs 312 ms), writes 55% faster, and > uses half the heap. > > *I clearly see a lot to gain from Parquet's FLOAT16 type.* > > *ALP numbers*: > storage encoding bits/value bits/value + zstd details > *FLOAT (32-bit)* *DICTIONARY* *15.16* *15.13* *23,908 distinct values, > 15-bit ids, 93 KiB dictionary* > FLOAT (32-bit) ALP 39.77 36.47 e=10, f=0 in every vector, avg 31.0-bit > integers, 18.06% exceptions > FLOAT (32-bit) ALP_rd 27.52 17.89 8 left + 24 right bits, 2.09% exceptions > FLOAT16 (16-bit) DICTIONARY 15.09 15.09 23,908 distinct values, 15-bit > ids, 47 KiB dictionary > FLOAT16 (16-bit) ALP_rd (not part of ALP) 14.46 14.46 5 left + 11 right > bits, 2.09% exceptions > > *Embedding benchmark numbers*: > source / storage encoding codec file MiB write ms read ms heap MiB > float[] / FLOAT PLAIN uncompressed 977.1 1,707 ±48 180 ±5 979 > *float[] / FLOAT* *DICT* *uncompressed* *459.2* *4,119 ±12* *312 ±2* *980* > float[] / FLOAT PLAIN zstd 579.2 5,730 ±30 950 ±3 980 > float[] / FLOAT DICT zstd 459.1 4,304 ±16 362 ±2 979 > *short[] / FLOAT16* *PLAIN* *uncompressed* *488.8* *1,871 ±8* *134 ±27* > *491* > short[] / FLOAT16 BSS uncompressed 488.8 2,813 ±17 156 ±16 491 > short[] / FLOAT16 PLAIN zstd 449.3 2,214 ±8 444 ±15 490 > short[] / FLOAT16 BSS zstd 415.0 3,003 ±33 284 ±17 491 > > Code: https://github.com/pvary/iceberg/commits/parquet_float16_benchmark/ > > To run the benchmarks: > > *./gradlew :iceberg-data:jmh * > *-PjmhIncludeRegex=ParquetFloat16EmbeddingBenchmark* > > *./gradlew :iceberg-data:jmh -PjmhIncludeRegex=AlpPrototype* > > > The first command runs all 24 combinations, which takes about 30 minutes. > The ALP prototype takes under a minute. > > Thanks, Peter > > Russell Spitzer <[email protected]> ezt írta (időpont: 2026. okt. > 21., Sze, 17:38): > >> I think all these expansions of types understood by Iceberg belong in a >> separate discussion. I think vector basically can exist >> completely independently of introducing more subtypes for Vectors. I think >> there is probably a lot more discussion needed regarding which specific >> variants for sub-width types we want to support and which engines are >> actually capable of supporting them. I would get Vector in first, then >> expand the spec later with additional types. >> >> My ideal path forward would be >> >> 1. Vote in current vector type (regardless of what is happening in >> Parquet since I don't think that really changes what we want to allow here, >> just the complexity of the implementation) >> 2. Start a new discussion about other vector subtypes. >> >> On Wed, Oct 21, 2026 at 6:43 AM Péter Váry <[email protected]> >> wrote: >> >>> Hi Kurtis, >>> Are the bf16/bf8 widely used? >>> I know that int8, and float16 are used by our users, and we already >>> have Parquet logical, and physical types defined for them. >>> >>> There is none for bf16/bf8, so we should first make them available there >>> if we want to use them. >>> >>> Thanks, >>> Peter >>> >>> <[email protected]> ezt írta (időpont: 2026. okt. 21., Sze, 1:03): >>> >>>> If we are discussing int8 and float16 for AI support shouldn’t we also >>>> consider brain float (bf16/bf8) as it’s often used in AI (training and >>>> inference) as well. >>>> >>>> On Oct 20, 2026, at 04:25, Péter Váry <[email protected]> >>>> wrote: >>>> >>>> >>>> int8 and float16 are needed for AI Vectors. There were suggestions to >>>> support them only in Vector types, but I think it would be a bad idea. >>>> Let's discuss it in the upcoming Vector type sync, and we could start >>>> its own DISCUSS thread if/once we agree on how to proceed. >>>> >>>> Daniel Weeks <[email protected]> ezt írta (időpont: 2026. okt. 19., H, >>>> 17:27): >>>> >>>>> Peter, >>>>> >>>>> I think we need to have a broader discussion around adding narrower >>>>> types. This probably deserves its own DISCUSS thread (not buried in a >>>>> sync >>>>> follow up). >>>>> >>>>> -Dan >>>>> >>>>> On Thu, Oct 15, 2026 at 2:01 PM Péter Váry < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Philipp, >>>>>> >>>>>> We’re working on vector indexes on top of the upcoming index >>>>>> framework, primarily for AI workloads. Supporting int8 and float16 >>>>>> vectors >>>>>> is a must for these use cases. >>>>>> >>>>>> The main motivation is vectors, but I’m not convinced these should be >>>>>> vector-specific types. Since int8 and float16 are well-defined scalar >>>>>> types >>>>>> that Iceberg would need to support as vector elements anyway, I think >>>>>> it’s >>>>>> cleaner to make them regular primitive types rather than special-case >>>>>> them >>>>>> as only valid inside vectors. >>>>>> >>>>>> This is also the feedback I heard during the community sync and my >>>>>> discussions at ApacheCon. >>>>>> >>>>>> Thanks, >>>>>> Peter >>>>>> >>>>>> >>>>>> Philipp Fischbeck via dev <[email protected]> ezt írta >>>>>> (időpont: 2026. okt. 15., Cs, 17:54): >>>>>> >>>>>>> Hi Peter, >>>>>>> >>>>>>> I see that in the current draft, you suggest adding int8 and float16 >>>>>>> as primitive Iceberg types overall, not just vector element types. I >>>>>>> don't >>>>>>> think this is the right approach because I don't see how engines would >>>>>>> use >>>>>>> these types outside of vectors. Do you have use cases in mind here? >>>>>>> >>>>>>> One alternative (as discussed in the design doc and during calls) >>>>>>> would be to allow such types only as vector element types. We would >>>>>>> still >>>>>>> have to define the Parquet/Avro type mapping, but the type definition >>>>>>> would >>>>>>> be limited to the encoding itself, not "spilling" to other parts like >>>>>>> bounds or schema evolution. >>>>>>> >>>>>>> Best, >>>>>>> Philipp >>>>>>> >>>>>>> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Talked with a bunch of people on ApacheCon, and I’d still very much >>>>>>>> like to see int8 and float16 vector support in Iceberg. >>>>>>>> >>>>>>>> I’ve created a draft PR to add these types: >>>>>>>> https://github.com/apache/iceberg/pull/18485. >>>>>>>> Parquet already provides a logical type for int8 and a physical >>>>>>>> type for float16, so adding support in Iceberg seems like a reasonable >>>>>>>> effort. >>>>>>>> >>>>>>>> I’d be happy to spearhead this work so we can get these types into >>>>>>>> V4, if the community agrees. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Peter >>>>>>>> >>>>>>>> >>>>>>>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, >>>>>>>> 2:33): >>>>>>>> >>>>>>>>> Thank you everyone for the discussion today! The meeting notes can >>>>>>>>> be found here: >>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>>>>>>> and below is a TLDR version: >>>>>>>>> - Focus on a 1D numeric vector type with fixed length and >>>>>>>>> non-null, finite coordinates; +0 and -0 remain valid. >>>>>>>>> - Keep the initial spec simple and build on Parquet’s emerging >>>>>>>>> vector representation. >>>>>>>>> - Initially support int, long, float, and double; defer >>>>>>>>> narrower/special types such as int8 as additional Iceberg primitive >>>>>>>>> types >>>>>>>>> (while allowing such parquet representation for vector). >>>>>>>>> - Keep higher level properties such as distance functions out of >>>>>>>>> scope, and revisit them with the index design. Sparse vectors are >>>>>>>>> also out >>>>>>>>> of scope for now: alternative options exist and waiting for concrete >>>>>>>>> use >>>>>>>>> cases. >>>>>>>>> - For formats without native vector support, use a list >>>>>>>>> representation with writer enforcement. Will explore an Avro vector >>>>>>>>> logical >>>>>>>>> type with Avro community; ORC support remains open. >>>>>>>>> >>>>>>>>> I'll update the spec PR in the next few days (also appreciate the >>>>>>>>> quick feedback from everyone!) >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Yan >>>>>>>>> >>>>>>>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Thank you Steven for helping secure the time slot for the vector >>>>>>>>>> type sync! And thanks everyone for your feedback on the design doc. >>>>>>>>>> >>>>>>>>>> In preparation for tomorrow's discussion, I've created a draft PR >>>>>>>>>> https://github.com/apache/iceberg/pull/18416 for the spec >>>>>>>>>> changes based on the progress in the parquet community on the Vector >>>>>>>>>> type >>>>>>>>>> proposal [1 <https://github.com/apache/parquet-format/pull/624>] >>>>>>>>>> [2 >>>>>>>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>>>>>>> , and will continue to iterate on both the spec PR and the design >>>>>>>>>> doc based >>>>>>>>>> on the discussion feedback. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Yan >>>>>>>>>> >>>>>>>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> As some of you are aware, there is an ongoing discussion about a >>>>>>>>>>> new *Vector* type in both the Parquet and Iceberg communities. >>>>>>>>>>> >>>>>>>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>>>>>>> >>>>>>>>>>> To accelerate the Vector type discussion, we are >>>>>>>>>>> reusing/extending Thursday's biweekly series to cover both File and >>>>>>>>>>> Vector >>>>>>>>>>> types (*starting on Oct 8*). Since the file type discussion is >>>>>>>>>>> reaching consensus, more time will likely be spent on the >>>>>>>>>>> Vector type discussion. The corresponding dev calendar event title >>>>>>>>>>> has been >>>>>>>>>>> updated. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Steven >>>>>>>>>>> >>>>>>>>>>
