bmorck opened a new pull request, #17452: URL: https://github.com/apache/iceberg/pull/17452
`ResolveViews` rebuilds a view's output from the stored schema, by position, wrapping each column in an `UpCast`. `UpCast` only widens, so a view whose stored type is narrower than what its SQL produces cannot be read at all: ``` [CANNOT_UP_CAST_DATATYPE] Cannot up cast id from "DOUBLE" to "BIGINT" ``` There is currently no way to relax that. Spark's `spark.sql.legacy.viewSchemaBindingMode` and `spark.sql.legacy.viewSchemaCompensation` do not help, since they are only consulted on the v1 `SessionCatalog` view path. This adds `spark.sql.iceberg.view.schema-binding-mode`, taking its mode names and coercions from Spark's `ViewSchemaMode`: | value | coercion | |---|---| | `BINDING` | `UpCast(col, storedType)` — the current behaviour, and the default | | `COMPENSATION` | `Cast(col, storedType, ansiEnabled = true)` — narrowing allowed | | `TYPE_EVOLUTION` | no cast, so the view reports the types its SQL produces | All three keep the stored column name and metadata; only the coercion differs. That mirrors `SessionCatalog.castColToType`, which ends with a single `Alias` for every mode. Two things worth flagging for review: 1. `TYPE_EVOLUTION` here is the read-side coercion only. Spark's `WITH SCHEMA TYPE EVOLUTION` also syncs the corrected schema back to the catalog, via `ViewSyncSchemaToMetaStore`. That seems wrong to do from a session-level conf, since any reader would then mutate a shared view definition; it would belong with a per-view property, as Spark stores the mode on the view itself. 2. When the conf is unset, Spark's two confs above are honored instead, reproducing how `castColToType` treats `SchemaUnsupported`. This gives one knob covering both view implementations, but it does mean Iceberg reading confs that are `.internal()` and documented for a DDL clause Iceberg views do not implement. Happy to drop that fallback if reviewers would rather not — the Iceberg conf alone is sufficient. Neither of Spark's confs can select `TYPE_EVOLUTION`, since there it is a per-view clause rather than a session default. The Iceberg conf takes precedence when both are set. The mode is read on each view resolution rather than cached on the rule, so `SET` applies within a session. Tests, in `TestViews` for each Spark version, over a view whose stored schema declares `BIGINT` for a column its SQL produces as `DOUBLE`: the default still fails; `COMPENSATION` reads the rows as `LongType`; `TYPE_EVOLUTION` reads them as `DoubleType` and keeps the stored column name; all nine states of Spark's two confs; the precedence between them; and an unrecognised mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
