Hi folks, Sharing an update on the Variant shredding proposal - [Link <https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing> ].
After a round of feedback on the initial design, I ran benchmarks across a set of inference strategies to settle the algorithm question. Results are in the Shredding Strategy Benchmarks tab and the code can be found [strategy benchmark] <https://github.com/nssalian/iceberg/blob/iceberg-variant-benchmark/benchmark/strategies.md> The proposal now has the resulting inference algorithm and moves the tunable thresholds from constants to table properties. Please take a look and let me know what you think. Thank you. On Fri, Jun 5, 2026 at 12:26 PM Neelesh Salian <[email protected]> wrote: > Hi folks, > > Thanks for those joining the Variant sync yesterday to discuss this > proposal. > And thanks Kurtis for the note. Appreciate the feedback and discussion as > well. > > Based on the feedback and the discussion during the call [Notes and > Recording here > <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>]; > the proposal should head towards aligning on the best inference algorithm > to yield the best results for shredding. > I'll be adjusting the proposal as I, along with folks in the community, > work on getting the data + testing for the algorithm. > Will surface the proposal again with this new direction when it's in a > better shape. > > Thank you and feel free to reach out if you have any questions. > > > On Thu, Jun 4, 2026 at 5:43 PM <[email protected]> wrote: > >> Hi Neelesh, >> >> Thank you for organizing the meeting earlier today and starting this >> thread. I think it’s awesome to see the community align not just on types >> but implementations for Variant. >> >> I wanted to highlight one additional item I heard discussed on the sync. >> >> - We need to align on an inference algorithm that defines how a >> variant object is shredded. >> >> >> - The goal is we can align using data from high quality benchmarking >> data sets if anyone or company is willing to contribute. >> >> Best Regards, >> Kurtis C. Wright >> >> On Jun 4, 2026, at 09:49, Neelesh Salian <[email protected]> >> wrote: >> >> >> Hi all, >> >> Variant shredding writers now exist across iceberg-java, arrow-go, and >> arrow-rs, and shredded write paths are landing soon in iceberg-go, >> iceberg-rust, and (future) iceberg-python. We don't yet have a shared >> policy on whether to shred, how the layout is declared, and how it's >> expressed so every implementation produces compatible output. >> >> I'd like to propose one: Variant Shredding Policy Across Iceberg >> Implementations >> <https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing> >> . >> Please take a look. >> >> Next steps: >> 1. The community reviews the proposal and agrees on the shredding >> representation. >> 2. Once consensus is reached, I'll start a [VOTE] thread. >> 3. After the vote passes, I'll land the adjustments in the Java >> implementation. >> 4. iceberg-go, iceberg-rust, and others follow to align. >> >> Thanks, >> Neelesh >> >>
