If we are discussing int8 and float16 for AI support shouldn’t we also consider brain float (bf16/bf8) as it’s often used in AI (training and inference) as well. int8 and float16 are needed for AI Vectors. There were suggestions to support them only in Vector types, but I think it would be a bad idea.Let's discuss it in the upcoming Vector type sync, and we could start its own DISCUSS thread if/once we agree on how to proceed. Peter,
I think we need to have a broader discussion around adding narrower types. This probably deserves its own DISCUSS thread (not buried in a sync follow up).
-Dan
Hi Philipp,
We’re working on vector indexes on top of the upcoming index framework, primarily for AI workloads. Supporting int8 and float16 vectors is a must for these use cases.
The main motivation is vectors, but I’m not convinced these should be vector-specific types. Since int8 and float16 are well-defined scalar types that Iceberg would need to support as vector elements anyway, I think it’s cleaner to make them regular primitive types rather than special-case them as only valid inside vectors.
This is also the feedback I heard during the community sync and my discussions at ApacheCon.
Thanks,
Peter
Philipp Fischbeck via dev < [email protected]> ezt írta (időpont: 2026. okt. 15., Cs, 17:54): Hi Peter,
I see that in the current draft, you suggest adding int8 and float16 as primitive Iceberg types overall, not just vector element types. I don't think this is the right approach because I don't see how engines would use these types outside of vectors. Do you have use cases in mind here?
One alternative (as discussed in the design doc and during calls) would be to allow such types only as vector element types. We would still have to define the Parquet/Avro type mapping, but the type definition would be limited to the encoding itself, not "spilling" to other parts like bounds or schema evolution.
Best,
Philipp
Talked with a bunch of people on ApacheCon, and I’d still very much like to see int8 and float16 vector support in Iceberg. I’ve created a draft PR to add these types: https://github.com/apache/iceberg/pull/18485. Parquet already provides a logical type for int8 and a physical type for float16, so adding support in Iceberg seems like a reasonable effort. I’d be happy to spearhead this work so we can get these types into V4, if the community agrees. Thanks, Peter
Thank you everyone for the discussion today! The meeting notes can be found here: https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he and below is a TLDR version: - Focus on a 1D numeric vector type with fixed length and non-null, finite coordinates; +0 and -0 remain valid. - Keep the initial spec simple and build on Parquet’s emerging vector representation. - Initially support int, long, float, and double; defer narrower/special types such as int8 as additional Iceberg primitive types (while allowing such parquet representation for vector). - Keep higher level properties such as distance functions out of scope, and revisit them with the index design. Sparse vectors are also out of scope for now: alternative options exist and waiting for concrete use cases. - For formats without native vector support, use a list representation with writer enforcement. Will explore an Avro vector logical type with Avro community; ORC support remains open.
I'll update the spec PR in the next few days (also appreciate the quick feedback from everyone!)
Thanks, Yan
Thank you Steven for helping secure the time slot for the vector type sync! And thanks everyone for your feedback on the design doc.
In preparation for tomorrow's discussion, I've created a draft PR https://github.com/apache/iceberg/pull/18416 for the spec changes based on the progress in the parquet community on the Vector type proposal [ 1] [ 2] , and will continue to iterate on both the spec PR and the design doc based on the discussion feedback.
Thanks, Yan
Hi,
As some of you are aware, there is an ongoing discussion about a new Vector type in both the Parquet and Iceberg communities.
To accelerate the Vector type discussion, we are reusing/extending Thursday's biweekly series to cover both File and Vector types (starting on Oct 8). Since the file type discussion is reaching consensus, more time will likely be spent on the Vector type discussion. The corresponding dev calendar event title has been updated.
Thanks, Steven
|
