Hi Alex, Thanks, these are the right questions. Let me answer them, but I think all three are really facets of one decision worth pulling out, so I'll do that at the end.
Original values vs sort keys. I don't think the two limitations are symmetric. You're right that a bound stored under version X may not hold under Y for either representation, and a naive reader prunes only on an exact version match either way. But they degrade differently: a sort key from X is incomparable under Y (nothing a Y reader can do with it) while an original value is the actual string, so a Y reader can re-interpret it. In the case you describe at the end of your mail, where only a small code-point range moved and a file's values fall outside it, that reader can prove the X bound still holds and prune across versions. Sort keys foreclose that; original values keep it open. So original values are a superset: worst case they match sort keys, best case they prune across versions. Your two sort-key advantages are real, I just don't think they belong in the format. Truncation: agreed it's hard for collated strings (your abcเก contraction case is exactly the trap), so I sidestepped it, collation bounds must be tight, a writer that can't store the exact min/max omits the bound. Collation-aware truncation with CollationElementIterator is a possible later optimization. Compare cost: pruning is per-file at planning time, not per-row, so collator vs byte compare is in the noise; and an engine that wants the byte path can derive and cache the sort key from the stored value. Original values don't block that, they just don't bake a version-specific encoding into the format. Column vs file-level version. As you say, every engine can read regardless, so this is a pruning-performance choice, not correctness. In the schema-registered-metrics design a file carries bounds under a declared (collation, version), and a reader prunes any file with a metric for a version it can produce, not only files it wrote. So convergence on one ICU version gives full cross-engine pruning, same as column-level, and a writer or compaction can populate several versions at once. It gives the column-level benefit by convention without making a version bump a format-breaking change. [One data point from the engine side, since I'm coming at this from the Databricks runtime: our runtime is effectively versionless (customers don't pin an ICU version, and upgrades happen under them) so "the same table read by clients on different ICU versions" isn't a corner case for us, it's the default. That's what pushes me toward per-file versioning: pinning one version per table or column means either forcing the whole fleet to upgrade in lockstep or breaking pruning on every bump, and neither survives a versionless fleet. And in practice most version bumps we've gone through don't reorder the data in a given column at all, which is exactly why keeping original values, and eventually your code-point-range idea, lets a reader keep pruning across a bump instead of falling back to a scan.] Providers. Agreed. I'll tighten the spec to a registered set like geo, icu to start, utf8 reserved, non-ICU collations added by spec change rather than ad hoc. Interop is the whole point and an open namespace undercuts it. ----------- Stepping back: I think the three above collapse into one question that needs broader alignment than the two of us: > how much cross-engine pruning interoperability should the format guarantee, versus leave to convention? Original-vs-sortkey, column-vs-file version, and open-vs-restricted providers are all that same tradeoff from different angles. It's a values call more than a correctness one, and it binds every engine that hasn't weighed in yet: Trino, Flink, Spark, PyIceberg, rust. >From the DBR side I can say the multi-version case is real rather than theoretical, but that's one engine's vantage point. I'd like to get the interop question in front of the other implementers before we fix field IDs, the dev list is probably enough for now, and a community sync is there if it needs more than async. No rush on that; I'd rather let the thread settle the mechanics first. Best, Andrei On Fri, Jul 3, 2026 at 12:22 AM Alexander Löser <[email protected]> wrote: > Hi Andrei, > > Thanks for putting together the spec PR and the detailed write-up! The > approach mostly looks solid to me. I have a few questions/initial thoughts > regarding the changes you proposed (compared to the original proposal): > > > 1 - Bounds store original values, not sort keys, tagged with a per-file > collation version. ICU/CLDR sort keys aren't stable across versions, so > storing keys ties every reader to one exact version; original values plus a > per-file version (readers prune only on an exact match) degrade gracefully > instead of breaking. The schema keeps the collation name unversioned so > anyone can read > > If I understand correctly, we’re talking about two separate things here: > > 1. Tagging a column vs a single file with a certain ICU version > 2. Using collation keys vs original strings (the ones that will > produce the min/max collation keys) > > > For 1, it comes down to a tradeoff: > > - If we tag the column with the ICU version, we’d force engines to > support one agreed-on ICU version if they want to prune files. Engines > would be able to prune every file (if they support the specific ICU > version), or none at all, so there is more incentive to support a specific > version > - If we tag individual files with ICU versions, we gain the big > advantage that engines do not need to agree on a single ICU version. > However, if I understand correctly, this comes at the cost of “fractured” > pruning - engines will only be able to prune files that were written by > themselves (or rather, with the same ICU version). As a consequence, > performance might not really be interoperable between different engines. > > Regardless of the approach we choose, all engines should be able to read > the data - they might just not be able to prune files. > > For 2, I’m not sure if I understand the advantages of original strings > yet. As you already pointed out, the collation keys depend on the ICU > version. However, if I understand correctly, the same limitation would > apply to the original strings: the sort order may (and does) change between > different ICU versions, too. As a consequence, we can’t assume the original > lower/upper bound strings we stored for version X will also be lower/upper > bounds for version Y - at least in the general case. So if I understand > correctly, we would not gain additional pruning opportunities compared to > using collation keys. Or am I missing something here? > > At the same time, sort keys do have advantages: > > - Iceberg allows the truncation of upper- and lower bounds. This is > trivial for binary collation keys. For original strings, the task becomes > significantly harder: truncating at a character boundary, for example, > would lead to wrong results, as there are some context-sensitive sequences: > e.g., with the CLDR root locale, abcเก < abcเ. I think it might be doable > with ICU’s CollationElementIterator, but it will be tricky to get right. > - Lower/upper bounds are computed once, but will be compared many > times. With original strings, we would need to either convert to the > collation key on the fly, or use ICU’s collator for a direct comparison. > Both options will be slower than a raw byte-sequence comparison > > > There is one scenario where original values would shine, though. I > analyzed the order-changes between various ICU versions: in many cases, > only a small range of code points changes/is moved. If we had additional > metadata about which code point ranges a file contains (e.g., whether it is > ASCII only), engines might be able to prove that the original string bounds > for version X are still valid for version Y. > If I'm not mistaken, this could allow to prune across different ICU > versions in certain situations, which I’d consider a point in favor of > original values (and file-level ICU versions). > > > > > 2 - A provider-qualified identifier (icu.en_US-ci), leaving room for > non-ICU collations like Spark's UTF8_LCASE, rather than assuming ICU as the > sole provider. > > Adding a provider-mechanism sounds like a good approach to keep the spec > open for future collations [image: :slightly_smiling_face:] I wonder > whether we should restrict the set of allowed providers, though, similar to > how it was done with geo > <https://lists.apache.org/thread/r5x0do8f241bpf565rx8s5s3wc9ogp0f>. My > main motivation for this proposal is interoperability. I worry that > interoperability might suffer or vanish completely if every engine can come > up with their own definitions. > > > > Happy to hear your thoughts on this! > > Best, Alex > > > On 6/27/26 01:37, Szehon Ho wrote: > > Very nice direction, left some comments on the spec proposal. > > Thanks to you folks for working on it ! > Szehon > > On Fri, Jun 26, 2026 at 3:29 AM Andrei Tserakhau via dev < > [email protected]> wrote: > >> Hi all, >> >> I've spend some cycle on the collation discussion and make something more >> concrete to react to: a spec-change PR plus reference implementations (go >> and java). >> >> - Spec change (apache/iceberg#16972): a "collation" annotation on string >> fields, and a data_file.collation_bounds field so collated columns stay >> prunable. >> - Reference implementation in iceberg-go (apache/iceberg-go#1318): the >> full path end to end - schema annotation, collation-aware comparison >> (CLDR/UCA), collation bounds in the manifest, and version-gated data-file >> pruning, with an Avro round-trip and pruning tests. >> - A lightweight Java POC (link below): the schema annotation plus a >> Collator-backed comparator, to match where the discussion is. I >> deliberately left the manifest/bounds side out of Java for now. >> >> The design follows the original proposal but takes a few different turns, >> mostly to adopt what we learned in Delta. The ones I'd most like input on: >> >> 1 - Bounds store original values, not sort keys, tagged with a per-file >> collation version. ICU/CLDR sort keys aren't stable across versions, so >> storing keys ties every reader to one exact version; original values plus a >> per-file version (readers prune only on an exact match) degrade gracefully >> instead of breaking. The schema keeps the collation name unversioned so >> anyone can read. >> >> 2 - A provider-qualified identifier (icu.en_US-ci), leaving room for >> non-ICU collations like Spark's UTF8_LCASE, rather than assuming ICU as the >> sole provider. >> >> 3 - One structural question I don't have a strong opinion on yet: I put >> collation_bounds on data_file as a standalone v3 field, but field id 146 is >> already the v4 content_stats struct, and collation bounds might belong >> inside that typed-stats framework instead. Worth settling before we fix >> field ids. >> >> The full set of differences and the reader/writer rules are in the PR >> description and the write-up. Comments very welcome — both on the calls >> above and on whether the standalone-field vs content_stats direction is the >> right one. >> >> Best, Andrei >> >> - original proposal: >> https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.0 >> - spec change: https://github.com/apache/iceberg/pull/16972 >> - POC in go: https://github.com/apache/iceberg-go/pull/1318 >> - java POC: >> https://github.com/laskoviymishka/iceberg/tree/prototype/collation-support >> >> On Mon, Mar 30, 2026 at 10:54 PM Alexander Löser <[email protected]> >> wrote: >> >>> Hi Andrei, >>> >>> I'm glad you're interested. Looking forward to collaborate with you! >>> Thanks for all the feedback here and in the doc. I only had a quick >>> glance, but I think you raised some good points. I'll address/respond to >>> your comments as soon as I get the chance, hopefully tomorrow. >>> I think you also left some comments in this mail that are not yet in the >>> doc - I'll move those to a dedicated section at the end of the doc, so we >>> can use the doc as a single source of truth/discussion. >>> >>> > Happy to share our Delta design doc and implementation learnings in >>> more detail. >>> >>> Sure, sounds good :) >>> >>> Best, >>> Alex >>> On 3/29/26 01:25, Andrei Tserakhau via dev wrote: >>> >>> Hi Alexander, >>> >>> This looks really interesting. We've been working on collation support >>> in Delta and have shipped it in production for some time, so this is an >>> area we care about a lot. If this proposal moves forward we'd be happy to >>> collaborate on the design and implementation. >>> >>> The pseudo-field approach for collation metrics is clean and composes >>> well with existing Iceberg infrastructure. The specifier coverage is >>> comprehensive. >>> >>> A few areas worth discussing as this evolves: >>> >>> 1 - Sort key stability and versioning >>> >>> ICU sort keys are not stable across versions, so a pinned ICU version >>> bump in a future Iceberg release would invalidate all existing collation >>> metrics. In multi-engine environments, requiring all engines to converge on >>> one ICU version is unrealistic. >>> >>> We store original string values instead of sort keys and allow per-file >>> version annotations -- worth discussing whether something similar could >>> work here. >>> >>> 2 - Provider abstraction >>> >>> The proposal assumes ICU as the sole provider, but Spark ships non-ICU >>> collations like UTF8_LCASE that are widely used. A provider or namespace >>> layer would prevent name collisions and support engine-specific collations >>> without future spec changes. >>> >>> 3 - Operational surface >>> >>> A few things that turned out correctness-critical in our implementation: >>> partition transforms on collated columns (collation-equal but byte-distinct >>> values in different directories), sort order semantics, equality deletes >>> under collation, and Parquet filter pushdown (must be disabled since >>> Parquet has no collation concept). >>> >>> These don't all need to be solved in v1 but would help to scope them. >>> >>> 4 - Smaller items (nit's) >>> >>> UTF-8 bounds for the original field id should be "must write" not >>> "should" -- otherwise backward compat breaks for non-aware engines. Engine >>> fallback behavior (case-sensitive vs older ICU vs fail) could use a >>> recommended preference order to avoid divergent results across engines. The >>> collation specifier syntax would benefit from a formal grammar. >>> >>> --- >>> >>> Happy to share our Delta design doc and implementation learnings in more >>> detail. Looking forward to the discussion. >>> >>> Best, >>> Andrei >>> >>> On Sat, Mar 28, 2026 at 11:49 PM Alexander Löser <[email protected]> >>> wrote: >>> >>>> Hi everyone, >>>> >>>> this is my first interaction with the Iceberg community, so here a few >>>> words about myself: >>>> - I'm Alex, a Berlin-based software engineer >>>> - I've been working at Snowflake for 4 years now >>>> - I spend most of my time on data types, particularly binary, strings >>>> and collations. >>>> >>>> I'd like to start a discussion about adding collations to the Iceberg >>>> spec. >>>> >>>> Conceptually, collations are an annotation on the string data type. By >>>> default, most engines perform string operations case-sensitively. >>>> Collations allow specifying alternative comparison rules. This is >>>> useful for achieving, e.g., case- or accent-insensitive string operations, >>>> or language-specific string sorting. >>>> Collations are supported by many engines: Databricks >>>> <https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-collation>, >>>> Spark >>>> <https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.collate.html>, >>>> Snowflake <https://docs.snowflake.com/en/sql-reference/collation>, >>>> Oracle >>>> <https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/COLLATION.html> >>>> - to >>>> name just a few - this list is not complete. >>>> >>>> In Snowflake, we see heavy use of the collation feature. Several users >>>> have approached us, mentioning they want to migrate to Iceberg tables, but >>>> are currently blocked by Iceberg's lack of collation support. >>>> >>>> Given the widespread support for collations across different engines, I >>>> believe introducing collations to Iceberg will increase interoperability >>>> and boost its adoption. >>>> I'd be curious about your thoughts. >>>> >>>> *Goal of the proposal* >>>> - Support collation specifications for columns >>>> - Define how collation bounds should be stored - UTF-8 based bounds are >>>> not useful for collated columns >>>> >>>> *Required Changes* >>>> - Extend the schema to let (string) fields be annotated with a collation >>>> >>>> More details can be found in this doc >>>> <https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.0#heading=h.y1ant4w2163k> >>>> . >>>> >>>> I'm also hoping to present the idea in the next community sync. >>>> >>>> Best, Alex >>>> >>>> >>>>
