Hi Peter Thanks for the interest and bumping the thread! Yes Prasanth and I are cleaning up the doc/proposal. There's been some other interest in this as well from Andrei.
I think it is more a consumer of Non-materialized column idea, and consumed by the Secondary index proposal, is that your understanding as well? I am also open to it being initially embedded in the Secondary Index spec as well, and not being blocked by adoption in Table spec, would that make sense? The latter may require more time, and putting it in the Index spec may be a good validation for the concept. Hopefully we can re-use the model in both cases once we agree on it. I'm happy to work with the Index group on this as well, and Prashant may as well, to get learnings from this effort. Also welcome more in community to take a look at this, I realized from CoC that it was lost in the emails. Thanks Szehon On Thu, Oct 15, 2026 at 4:06 AM Péter Váry <[email protected]> wrote: > I would like to review this conversation. > The feature is very useful is on its own, but the derived column concept > could be used for several ongoing projects: > - Non-materialized columns: > https://lists.apache.org/thread.html/4h53r1nvvgxqymfvx7vcyhqgh1fk8180 > - Iceberg secondary indexes: > https://lists.apache.org/thread/gtpn4ydovmoxb1bffz5wcnsso20n379c > > Was there any more progress in this regard? > Is someone driving this feature, or shall I jump in? > > Thanks, > Peter > > Szehon Ho <[email protected]> ezt írta (időpont: 2026. aug. 11., K, > 0:41): > >> HI Prasant, sure thanks! I added some brief context to the doc from our >> previous research: >> >> 1. Put the official SQL spec definition >> 2. Define 'Generated Column' >> 3. Define 'Virtual Column' >> 4. Clarify what we gain/lose with deterministic expressions. >> >> I didn't modify the proposal yet, just wanted to make sure we are on the >> same page on the definition of derived column as we go into the proposal. >> Hope it looks ok with you. Maybe we can also chat on slack channel if it's >> more convenient, up to you folks. >> >> Hi Stefan, it's a good point. Some database like SQLServer seem to also >> support a similar concept (computed column), but there's a lot that the >> column cannot do if it's non-deterministic. As in the points I put in the >> doc, these are thing like VIRTUAL column, partition column, index key, >> check constraint, even Update. So while its not so useful in the >> traditional database sense, it may make sense to let the engine have an >> option. >> >> Aside from determinism, another question is portability, ie have engines >> (at least different instance of same engine) understand each other's column >> expression. Sounds like we want to align with UDF/View spec, and upcoming >> Check Constraint and Default Value spec, as mentioned in Andrei's email? >> >> Thanks >> Szehon >> >> On Mon, Aug 10, 2026 at 5:23 AM Prashant Sharma <[email protected]> >> wrote: >> >>> I am fairly new to Iceberg. Thanks for the encouragement. Here is a >>> first draft (still WIP) >>> https://docs.google.com/document/d/1u93kyIrTc8bt9VSTB4O0GM9S7zuXekqixJR6Id2FZC4/edit?usp=sharing. >>> Please take a look ! >>> >>> https://github.com/apache/iceberg/issues/15923 >>> >>> @Andrei Tserakhau <[email protected]> >>> Your summary has helped a lot in catching up with parallel work and also >>> the history - Thanks a lot ! >>> >>> @[email protected] <[email protected]> and >>> @[email protected] <[email protected]> Happy to collaborate - was >>> trying to add you as editor - let me know your google account enabled id. >>> >>> In the proposal document above, I have also tried to address two problem >>> which are common to multiple v4 features >>> >>> 1. Expressions reuse >>> 2. Backward compatibility: V4-features need the ability to know data was >>> modified by an engine unaware of the feature and how does the engine tell >>> that? For example: derived columns feature on optimizer rewrite could give >>> incorrect results. >>> >>> On Fri, Aug 7, 2026 at 9:09 PM Stefan Grafberger via dev < >>> [email protected]> wrote: >>> >>>> Hi all, >>>> >>>> Thanks a lot for bringing up this topic! >>>> >>>> I think one important topic to consider here is multi-modal workloads. >>>> Both Lance and other databases like BigQuery use derived columns also >>>> for embedding generation, where materializing the output of >>>> expressions like embedding functions provides huge cost and compute >>>> savings. >>>> >>>> - Lance Embedding Generation: https://docs.lancedb.com/embedding >>>> - BigQuery Autonomous Embedding Generation: >>>> >>>> https://docs.cloud.google.com/bigquery/docs/autonomous-embedding-generation >>>> >>>> To support these multi-modal AI patterns in Iceberg, it would be >>>> beneficial not to broadly prohibit all non-deterministic expressions >>>> in the spec. >>>> >>>> Instead, it might be worth distinguishing between: >>>> 1. Purely deterministic expressions (e.g., LOWER(col)), >>>> 2. Purely non-deterministic expressions (e.g., RAND(), UUID()), and >>>> 3. Non-deterministic but reusable expressions (e.g., AI.EMBED() in >>>> BigQuery), where the value generation is AI based, so there's some >>>> non-determinism involved, but once generated, the materialized result >>>> acts as a reusable derived property. >>>> >>>> By making this distinction at the spec level, we could allow reusable >>>> non-deterministic expressions without sacrificing the predictability >>>> needed for query optimization. >>>> >>>> Curious to hear what others think about supporting such AI use cases. >>>> >>>> Best, >>>> Stefan >>>> >>>> On Thu, Aug 6, 2026 at 8:09 PM Szehon Ho <[email protected]> >>>> wrote: >>>> > >>>> > Hi, nice, I have also been looking at implementing this from the >>>> Spark DSV2 side, would love to collaborate on this as well. >>>> > >>>> > Yes it seems like we can go further with the expression spec. I >>>> found the determinism problem to be deeper than it seemed at first glance, >>>> a lot of popular generation expressions in Delta, Spark are in fact >>>> dependent on engine configs like timezone, which makes data skipping a bit >>>> tricky, we've only recently started thinking about this issue. >>>> > >>>> > Thanks, >>>> > Szehon >>>> > >>>> > On Thu, Aug 6, 2026 at 9:07 AM Daniel Weeks <[email protected]> >>>> wrote: >>>> >> >>>> >> What Andrei captured is a great summary of related proposals and >>>> ongoing work. >>>> >> >>>> >> While there's no formal proposal, discussions have occurred >>>> regarding using expressions and the default value expressions concept to >>>> expose generated columns. >>>> >> >>>> >> We'd love for someone to create a formal proposal to expand upon >>>> these existing proposals/features as it's a pretty natural extension from >>>> what we have in progress already. >>>> >> >>>> >> I'd be happy to work with you on this if you would like to pick this >>>> up, >>>> >> -Dan >>>> >> >>>> >> On Thu, Aug 6, 2026 at 8:38 AM Andrei Tserakhau via dev < >>>> [email protected]> wrote: >>>> >>> >>>> >>> Thanks for bringing this to the list, Prashant. >>>> >>> >>>> >>> On your question: no, there's no active work on a format-level >>>> >>> derived/computed column concept in Iceberg, and as far as I can tell >>>> >>> from the archives none has been proposed before -- so you're not >>>> >>> duplicating anything. Some adjacent prior art that may be useful. >>>> >>> >>>> >>> Delta has had generated columns since OSS Delta 1.0.0 (May 2021), >>>> with >>>> >>> your exact motivation: a column materialized from an expression, >>>> used >>>> >>> as a partition column, lets the engine infer a filter on it from a >>>> >>> filter on the source and skip files. Two costs showed up in the five >>>> >>> years since. First, the expression is stored as a Spark SQL string >>>> >>> (delta.generationExpression), so anything that isn't Spark has to >>>> parse >>>> >>> Spark SQL to know what the column means -- which is roughly where >>>> >>> presto.derived-columns.spec.json would put Iceberg, one dialect >>>> over. >>>> >>> Second, Delta gates it behind a writer feature, so a writer that >>>> >>> doesn't understand the invariant is refused rather than allowed to >>>> >>> silently break it. Your RFC is candid that stale values produce >>>> >>> incorrect results once rewrite is enabled, and table properties are >>>> >>> defined as hints, so an unaware writer is within its rights to >>>> ignore >>>> >>> them. Worth deciding early which of those two you want. >>>> >>> >>>> >>> On both counts there's an existing pattern in Iceberg you'd probably >>>> >>> rather join than parallel. The expressions spec merged in June >>>> >>> (https://github.com/apache/iceberg/pull/16652) -- field-ID >>>> references >>>> >>> that survive renames, dialect-specific behavior pushed into UDF >>>> >>> references. Two features are already building on it: Huaxin's >>>> >>> constraint support ( >>>> https://lists.apache.org/thread/mv0gjm1o9v6wzx6dxkdw6yp5b9119w2d), >>>> >>> which stores CHECK expressions as Expression objects rather than SQL >>>> >>> strings, and Dan's default value expressions in V4 >>>> >>> (https://lists.apache.org/thread/w0xqrm0dpnsgvw0dyvy4r34y0dtzmn7f, >>>> PR >>>> >>> 16777). The constraint proposal also tracks validity per snapshot >>>> >>> rather than on the constraint, which might be a lighter answer to >>>> your >>>> >>> staleness problem than a format-version bump -- an unaware writer >>>> would >>>> >>> degrade you to "rewrite disabled" rather than "wrong answer." >>>> >>> >>>> >>> Two scope notes. Your item 3 (stats and a bloom filter with no >>>> stored >>>> >>> data) overlaps the secondary index work fairly directly -- Péter >>>> has a >>>> >>> draft spec (https://github.com/apache/iceberg/pull/16961) and >>>> there's >>>> >>> an active sync series. And it's worth stating explicitly that this >>>> >>> covers deterministic expressions only: Delta put deterministic >>>> >>> expression-derived columns and non-deterministic value generation >>>> >>> (identity/sequence) under one GENERATED ALWAYS AS umbrella, and >>>> >>> separating them afterwards was painful, since pushdown rewrite needs >>>> >>> reproducibility and value generation needs the opposite. Your RFC >>>> >>> already excludes non-deterministic functions, so this is just making >>>> >>> that explicit in the spec. >>>> >>> >>>> >>> Last thing: hidden-partition transforms already give metadata-level >>>> >>> skipping for monotonic cases with no extra stored column. lower(col) >>>> >>> genuinely isn't covered, so the motivation is fair -- but saying >>>> where >>>> >>> transforms already suffice and where a stored column is actually >>>> needed >>>> >>> would make the proposal harder to argue with. >>>> >>> >>>> >>> Best, >>>> >>> Andrei >>>> >>> >>>> >>> On Thu, Aug 6, 2026 at 12:28 PM Prashant Sharma < >>>> [email protected]> wrote: >>>> >>>> >>>> >>>> Hello All, >>>> >>>> >>>> >>>> Recently, I have been involved in bringing derived columns(aka >>>> generated columns) for the Iceberg connector in Presto Link: >>>> https://github.com/prestodb/rfcs/pull/61. We currently use table >>>> properties to store the extra metadata for derived columns. However, we >>>> cannot achieve the following: >>>> >>>> >>>> >>>> 1. Cross engine compatibility: How do we know if the derived >>>> columns metadata is in sync i.e. if other engines unaware of derived >>>> columns write to the table/alter the columns etc... >>>> >>>> 2. UDF portability and versioning: which is already addressed by >>>> (Iceberg UDF spec) >>>> >>>> 3. VIRTUAL columns: We store all the column metadata/metrics/stats >>>> including a bloom filter but not the actual data. A poor man's index. >>>> >>>> >>>> >>>> We are considering a Iceberg spec level support to address all of >>>> the above. Just wondering, is there work already happening in this >>>> direction elsewhere? >>>> >>>> >>>> >>>> If not, I have started an issue: >>>> https://github.com/apache/iceberg/issues/15923 , will be adding more >>>> details soon. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Prashant >>>> >>>
