Hi Alex Thanks for the email, the doc is fairly long. it will be great if we have some TLDR for us to track & discuss. thanks.
On Mon, Nov 2, 2026 at 1:28 PM Alex Cruise <[email protected]> wrote: > I've also prepared a backgrounder at > https://github.com/acruise/iceberg-indexing-ecosystem/blob/main/docs/proposals/puffin-blob-types-background.md > which will hopefully provide a gentler, more informal learning curve. :) > > -0xe1a > > On Mon, Nov 2, 2026 at 2:26 PM Alex Cruise <[email protected]> wrote: > >> Hey folks! >> >> I've been working on a proposal aimed at enabling an ecosystem of Puffin >> blob types, with opinionated (but hopefully lightweight) metadata about >> configuration, scope, and how future blobs interact with deletion and >> compaction. >> >> For context, separate from any Iceberg specifics, I've been daydreaming >> about an idea of "progressive indexing" for a few years: after a file is >> written, perhaps weeks or months later, we may discover that read workloads >> could be much faster and/or more efficient if we'd made different indexing >> choices before. Unfortunately, it's hard to predict whether any such change >> might impact ingest workloads, and rewriting data, while commonplace on >> small files during early compaction eras, is a non-starter for files that >> are large and old, so some read-heavy workloads are stuck with scanning >> huge volumes of data that often has high-selectivity features lurking in >> dusty corners. >> >> While building out an early, as yet unpublished version of the >> progressive indexing idea, I realized that while I was initially focused on >> the _timing_ of index builds (still important!) I hadn't spent enough time >> on the _diversity_ of available indexes, and how to rein in the predictable >> chaos if my earlier goals were achieved and such choices became broadly >> available. >> >> So, in recent days, this proposal has evolved to focus on what I now >> consider a prerequisite for the timing and diversity visions, which is: how >> do we construct and operate an ecosystem that would support the desired >> diversity, within guardrails that make the chaos less likely and more >> contained, in which query accuracy can be safeguarded as long as >> implementers play by the rules? >> >> I have looked at the parallel v4 proposal for secondary indexes, and I >> believe my proposal is complementary, even though there are significant >> overlaps in scope. In particular mine focuses mainly (but not exclusively) >> on filters, or "negative" indices, in which an out-of-date index merely >> costs efficiency (i.e. a file/chunk that does not contain rows that satisfy >> the predicate is scanned), whereas the `scalar` index described in the v4 >> proposal is a "positive" index, in which much more care is required to >> prevent mutations from corrupting indexes, which would cause query >> correctness problems. Section 12 of my proposal is a detailed comparison. >> >> This email is already long enough, so rather than go into further detail, >> I'll ask you to have a look at >> https://github.com/acruise/iceberg-indexing-ecosystem/blob/main/docs/proposals/puffin-blob-types-and-compaction.md >> which is admittedly far too long, just like this email! >> >> The same doc is open for comments at >> https://docs.google.com/document/d/18qlGNPfPy5iT_aRgxLVCqh5W1LuINR7x8Eu2zC6HMoU/edit?usp=sharing >> and I'm on the Iceberg Slack now as @arrgh if you prefer that kind of >> thing. :) >> >> Thanks! >> >> -0xe1a >> >
