Hi Ryan, +1 to this.
Collation support (#16972) would use this as the general form of the to_lower_case(name) example: one expr-value per collation version, like collate(ref(7), 'icu.en_US-ci', '<version>'). One question: could the rule for unsupported types also cover expressions, so readers ignore stats for an expr-value they cannot evaluate? That would handle collation directly. A reader on a different collator version would skip those stats, and the collation spec would not need its own reader rule for them. Thanks, Andrei On Fri, Oct 23, 2026 at 8:45 PM Péter Váry <[email protected]> wrote: > Thanks Ryan, > > I could argue with the other reasons, but the "should not be displayed as > part of the table schema" seals the discussion. > > +1 > > Peter > > > > On Fri, Oct 23, 2026, 18:29 Ryan Blue <[email protected]> wrote: > >> I don't think that we are trying to solve the problem of generated >> columns right now. I think that should be part of the schema so it is >> separate from this work. >> >> You might be suggesting that we use generated columns to store stats >> instead of the current proposal. I don't think that is a good idea because >> it adds extra fields to the schema with different behavior that breaks >> assumptions for schemas. >> >> For instance, stats-only fields are not fields in the schema and should >> not be displayed as part of the table schema. Generated columns are part of >> table data, which is why they belong in the schema. If we add stats-only >> fields to the schema, then anyone consuming the schema needs to know what >> to filter out before passing it along. >> >> Similarly, schema fields have specific types that change only when a user >> modifies them. Stats fields are derived and can change when schema changes >> affect them, so we would not be able to produce all field types without >> binding the schema to a subset of itself. >> >> This would also complicate composition. Right now, stats-only fields are >> not nested within the schema, they are independent. If we were allowing a >> field to be a stat-only field then we have to come up with rules around >> where they can be used (not in map keys?) and then check those rules. How >> would you handle a struct with only one field that is a stats-only field >> that is not actually present in the schema? >> >> The proposal is much simpler because we don't need to worry about many of >> these concerns. >> >> Ryan >> >> On Thu, Oct 22, 2026 at 10:15 PM Péter Váry <[email protected]> >> wrote: >> >>> Hi Ryan, >>> >>> Thanks for putting this together! >>> >>> I had the same impression when I took a look at the derived columns >>> proposal, and we updated that. >>> >>> Now, a schema would look like this: >>> >>> schema { >>> 1: id: required int >>> 2: name: optional string >>> 3: hilbert: optional long (generated: stats-only: hilbert(1, 2)) >>> } >>> >>> Or in JSON: >>> { >>> [...] >>> "fields": [ >>> { >>> "id": 3, >>> "name": "id", >>> "required": false, >>> "type": "long", >>> "generated": >>> { >>> "strategy": "stats-only", // on-write, on-read(?) >>> "expression": [...] >>> } >>> }, >>> [...] >>> ] >>> } >>> >>> This would help removing the extra rules around the id generation, stat >>> generation and we likely need the generated columns too. >>> >>> Schema already contains expressions, elements that are not stored in the >>> files, and stats-only fields are optional so just return `null` when >>> queried. >>> >>> WDYT? >>> >>> Thanks, >>> Peter >>> >>> >>> On Fri, Oct 23, 2026, 00:56 Ryan Blue <[email protected]> wrote: >>> >>>> Hi everyone, >>>> >>>> We've been working on the spec changes for non-materialized columns in >>>> v4, which was previously discussed here: >>>> https://lists.apache.org/thread/4h53r1nvvgxqymfvx7vcyhqgh1fk8180 >>>> >>>> I think the spec is about ready. Please take a look here: >>>> https://github.com/apache/iceberg/pull/18405 >>>> >>>> I ended up using "stats-only fields" since I think that's a more >>>> straightforward name than "non-materialized columns". The changes are what >>>> was discussed in the last thread. >>>> >>>> I also considered whether it would be a good idea to unify this with >>>> materialized columns that are generated by an expression, like default >>>> value expressions or generated columns. I decided not to do that for the >>>> same reasons we chose not to separate expressions from check constraints. >>>> That would require adding an expression and then adding a column that >>>> depends on it. That indirection isn't needed and I think generated columns >>>> should have an expression in the definition. >>>> >>>> Please take a look at the PR and comment. I'd like to start a vote soon >>>> if there is consensus. Thanks! >>>> >>>> Ryan >>>> >>>>
