Hi Ryan,
Thanks for putting this together!
I had the same impression when I took a look at the derived columns
proposal, and we updated that.
Now, a schema would look like this:
schema {
1: id: required int
2: name: optional string
3: hilbert: optional long (generated: stats-only: hilbert(1, 2))
}
Or in JSON:
{
[...]
"fields": [
{
"id": 3,
"name": "id",
"required": false,
"type": "long",
"generated":
{
"strategy": "stats-only", // on-write, on-read(?)
"expression": [...]
}
},
[...]
]
}
This would help removing the extra rules around the id generation, stat
generation and we likely need the generated columns too.
Schema already contains expressions, elements that are not stored in the
files, and stats-only fields are optional so just return `null` when
queried.
WDYT?
Thanks,
Peter
On Fri, Oct 23, 2026, 00:56 Ryan Blue <[email protected]> wrote:
> Hi everyone,
>
> We've been working on the spec changes for non-materialized columns in v4,
> which was previously discussed here:
> https://lists.apache.org/thread/4h53r1nvvgxqymfvx7vcyhqgh1fk8180
>
> I think the spec is about ready. Please take a look here:
> https://github.com/apache/iceberg/pull/18405
>
> I ended up using "stats-only fields" since I think that's a more
> straightforward name than "non-materialized columns". The changes are what
> was discussed in the last thread.
>
> I also considered whether it would be a good idea to unify this with
> materialized columns that are generated by an expression, like default
> value expressions or generated columns. I decided not to do that for the
> same reasons we chose not to separate expressions from check constraints.
> That would require adding an expression and then adding a column that
> depends on it. That indirection isn't needed and I think generated columns
> should have an expression in the definition.
>
> Please take a look at the PR and comment. I'd like to start a vote soon if
> there is consensus. Thanks!
>
> Ryan
>
