HI Prasant, sure thanks! I added some brief context to the doc from our previous research:
1. Put the official SQL spec definition 2. Define 'Generated Column' 3. Define 'Virtual Column' 4. Clarify what we gain/lose with deterministic expressions. I didn't modify the proposal yet, just wanted to make sure we are on the same page on the definition of derived column as we go into the proposal. Hope it looks ok with you. Maybe we can also chat on slack channel if it's more convenient, up to you folks. Hi Stefan, it's a good point. Some database like SQLServer seem to also support a similar concept (computed column), but there's a lot that the column cannot do if it's non-deterministic. As in the points I put in the doc, these are thing like VIRTUAL column, partition column, index key, check constraint, even Update. So while its not so useful in the traditional database sense, it may make sense to let the engine have an option. Aside from determinism, another question is portability, ie have engines (at least different instance of same engine) understand each other's column expression. Sounds like we want to align with UDF/View spec, and upcoming Check Constraint and Default Value spec, as mentioned in Andrei's email? Thanks Szehon On Mon, Aug 10, 2026 at 5:23 AM Prashant Sharma <[email protected]> wrote: > I am fairly new to Iceberg. Thanks for the encouragement. Here is a first > draft (still WIP) > https://docs.google.com/document/d/1u93kyIrTc8bt9VSTB4O0GM9S7zuXekqixJR6Id2FZC4/edit?usp=sharing. > Please take a look ! > > https://github.com/apache/iceberg/issues/15923 > > @Andrei Tserakhau <[email protected]> > Your summary has helped a lot in catching up with parallel work and also > the history - Thanks a lot ! > > @[email protected] <[email protected]> and @[email protected] > <[email protected]> Happy to collaborate - was trying to add you as > editor - let me know your google account enabled id. > > In the proposal document above, I have also tried to address two problem > which are common to multiple v4 features > > 1. Expressions reuse > 2. Backward compatibility: V4-features need the ability to know data was > modified by an engine unaware of the feature and how does the engine tell > that? For example: derived columns feature on optimizer rewrite could give > incorrect results. > > On Fri, Aug 7, 2026 at 9:09 PM Stefan Grafberger via dev < > [email protected]> wrote: > >> Hi all, >> >> Thanks a lot for bringing up this topic! >> >> I think one important topic to consider here is multi-modal workloads. >> Both Lance and other databases like BigQuery use derived columns also >> for embedding generation, where materializing the output of >> expressions like embedding functions provides huge cost and compute >> savings. >> >> - Lance Embedding Generation: https://docs.lancedb.com/embedding >> - BigQuery Autonomous Embedding Generation: >> >> https://docs.cloud.google.com/bigquery/docs/autonomous-embedding-generation >> >> To support these multi-modal AI patterns in Iceberg, it would be >> beneficial not to broadly prohibit all non-deterministic expressions >> in the spec. >> >> Instead, it might be worth distinguishing between: >> 1. Purely deterministic expressions (e.g., LOWER(col)), >> 2. Purely non-deterministic expressions (e.g., RAND(), UUID()), and >> 3. Non-deterministic but reusable expressions (e.g., AI.EMBED() in >> BigQuery), where the value generation is AI based, so there's some >> non-determinism involved, but once generated, the materialized result >> acts as a reusable derived property. >> >> By making this distinction at the spec level, we could allow reusable >> non-deterministic expressions without sacrificing the predictability >> needed for query optimization. >> >> Curious to hear what others think about supporting such AI use cases. >> >> Best, >> Stefan >> >> On Thu, Aug 6, 2026 at 8:09 PM Szehon Ho <[email protected]> wrote: >> > >> > Hi, nice, I have also been looking at implementing this from the Spark >> DSV2 side, would love to collaborate on this as well. >> > >> > Yes it seems like we can go further with the expression spec. I found >> the determinism problem to be deeper than it seemed at first glance, a lot >> of popular generation expressions in Delta, Spark are in fact dependent on >> engine configs like timezone, which makes data skipping a bit tricky, we've >> only recently started thinking about this issue. >> > >> > Thanks, >> > Szehon >> > >> > On Thu, Aug 6, 2026 at 9:07 AM Daniel Weeks <[email protected]> wrote: >> >> >> >> What Andrei captured is a great summary of related proposals and >> ongoing work. >> >> >> >> While there's no formal proposal, discussions have occurred regarding >> using expressions and the default value expressions concept to expose >> generated columns. >> >> >> >> We'd love for someone to create a formal proposal to expand upon these >> existing proposals/features as it's a pretty natural extension from what we >> have in progress already. >> >> >> >> I'd be happy to work with you on this if you would like to pick this >> up, >> >> -Dan >> >> >> >> On Thu, Aug 6, 2026 at 8:38 AM Andrei Tserakhau via dev < >> [email protected]> wrote: >> >>> >> >>> Thanks for bringing this to the list, Prashant. >> >>> >> >>> On your question: no, there's no active work on a format-level >> >>> derived/computed column concept in Iceberg, and as far as I can tell >> >>> from the archives none has been proposed before -- so you're not >> >>> duplicating anything. Some adjacent prior art that may be useful. >> >>> >> >>> Delta has had generated columns since OSS Delta 1.0.0 (May 2021), with >> >>> your exact motivation: a column materialized from an expression, used >> >>> as a partition column, lets the engine infer a filter on it from a >> >>> filter on the source and skip files. Two costs showed up in the five >> >>> years since. First, the expression is stored as a Spark SQL string >> >>> (delta.generationExpression), so anything that isn't Spark has to >> parse >> >>> Spark SQL to know what the column means -- which is roughly where >> >>> presto.derived-columns.spec.json would put Iceberg, one dialect over. >> >>> Second, Delta gates it behind a writer feature, so a writer that >> >>> doesn't understand the invariant is refused rather than allowed to >> >>> silently break it. Your RFC is candid that stale values produce >> >>> incorrect results once rewrite is enabled, and table properties are >> >>> defined as hints, so an unaware writer is within its rights to ignore >> >>> them. Worth deciding early which of those two you want. >> >>> >> >>> On both counts there's an existing pattern in Iceberg you'd probably >> >>> rather join than parallel. The expressions spec merged in June >> >>> (https://github.com/apache/iceberg/pull/16652) -- field-ID references >> >>> that survive renames, dialect-specific behavior pushed into UDF >> >>> references. Two features are already building on it: Huaxin's >> >>> constraint support ( >> https://lists.apache.org/thread/mv0gjm1o9v6wzx6dxkdw6yp5b9119w2d), >> >>> which stores CHECK expressions as Expression objects rather than SQL >> >>> strings, and Dan's default value expressions in V4 >> >>> (https://lists.apache.org/thread/w0xqrm0dpnsgvw0dyvy4r34y0dtzmn7f, PR >> >>> 16777). The constraint proposal also tracks validity per snapshot >> >>> rather than on the constraint, which might be a lighter answer to your >> >>> staleness problem than a format-version bump -- an unaware writer >> would >> >>> degrade you to "rewrite disabled" rather than "wrong answer." >> >>> >> >>> Two scope notes. Your item 3 (stats and a bloom filter with no stored >> >>> data) overlaps the secondary index work fairly directly -- Péter has a >> >>> draft spec (https://github.com/apache/iceberg/pull/16961) and there's >> >>> an active sync series. And it's worth stating explicitly that this >> >>> covers deterministic expressions only: Delta put deterministic >> >>> expression-derived columns and non-deterministic value generation >> >>> (identity/sequence) under one GENERATED ALWAYS AS umbrella, and >> >>> separating them afterwards was painful, since pushdown rewrite needs >> >>> reproducibility and value generation needs the opposite. Your RFC >> >>> already excludes non-deterministic functions, so this is just making >> >>> that explicit in the spec. >> >>> >> >>> Last thing: hidden-partition transforms already give metadata-level >> >>> skipping for monotonic cases with no extra stored column. lower(col) >> >>> genuinely isn't covered, so the motivation is fair -- but saying where >> >>> transforms already suffice and where a stored column is actually >> needed >> >>> would make the proposal harder to argue with. >> >>> >> >>> Best, >> >>> Andrei >> >>> >> >>> On Thu, Aug 6, 2026 at 12:28 PM Prashant Sharma <[email protected]> >> wrote: >> >>>> >> >>>> Hello All, >> >>>> >> >>>> Recently, I have been involved in bringing derived columns(aka >> generated columns) for the Iceberg connector in Presto Link: >> https://github.com/prestodb/rfcs/pull/61. We currently use table >> properties to store the extra metadata for derived columns. However, we >> cannot achieve the following: >> >>>> >> >>>> 1. Cross engine compatibility: How do we know if the derived columns >> metadata is in sync i.e. if other engines unaware of derived columns write >> to the table/alter the columns etc... >> >>>> 2. UDF portability and versioning: which is already addressed by >> (Iceberg UDF spec) >> >>>> 3. VIRTUAL columns: We store all the column metadata/metrics/stats >> including a bloom filter but not the actual data. A poor man's index. >> >>>> >> >>>> We are considering a Iceberg spec level support to address all of >> the above. Just wondering, is there work already happening in this >> direction elsewhere? >> >>>> >> >>>> If not, I have started an issue: >> https://github.com/apache/iceberg/issues/15923 , will be adding more >> details soon. >> >>>> >> >>>> Thanks, >> >>>> Prashant >> >
