szehon-ho opened a new pull request, #56164: URL: https://github.com/apache/spark/pull/56164
### What changes were proposed in this pull request? Extend the DSV2 `Column` API so that a generated column carries both its SQL text and (when translatable) a Spark-analyzed connector `Expression`, mirroring how `DefaultValue` already works. - Add `GenerationExpression` and a shared `SqlOrExpression` base class under `org.apache.spark.sql.connector.catalog`. `DefaultValue` is refactored to extend `SqlOrExpression` for symmetry. - `Column.columnGenerationExpression()` now returns a `GenerationExpression` with both the original SQL and an optional V2 `Expression`. A `Column.create(...)` overload accepts it. - Wire generation expression analysis into the create/replace table path. `GeneratedColumn.toGenerationExpression(sql, name, dataType, schema, statementType)` parses, analyzes, and verifies the expression once, then translates the analyzed Catalyst expression to V2 form via `V2ExpressionBuilder` (returning `null` for the V2 form when the expression is not translatable). - Replace `validateGeneratedColumns` with a single helper `GeneratedColumn.validateAndBuildV2Columns(columns, schema, catalog, ident, statementType)` that performs the catalog capability check and builds the V2 columns in one place; `DataSourceV2Strategy` calls it for both `CreateTable` and `ReplaceTable`. - `CatalogV2Util.structFieldToV2Column` now also analyzes generated columns when materializing V2 columns from a `StructType`, so the default `Table.columns()` implementation surfaces the Expression form on the read path too. - Add `equals`/`hashCode` to `Extract` for value-based comparison of V2 expressions in tests. ### Why are the changes needed? Today, DSV2 connectors only see the raw SQL string for a generated column and have to parse and translate it themselves. Many connectors prefer the Spark-analyzed `Expression` form so they can apply partitioning, statistics, or rewrite logic uniformly with how default values are already handled. Carrying the analyzed expression in the `Column` API lets connectors consume it directly, and lets Spark reuse the analysis it already performs at create-table time instead of discarding the result. ### Does this PR introduce _any_ user-facing change? Yes. `Column.columnGenerationExpression()` now returns a `GenerationExpression` with both SQL and (optionally) a connector `Expression`. The default `Column` builder still accepts SQL only, so existing connectors are source-compatible. `DefaultValue` now extends the shared `SqlOrExpression` base class; its public surface is unchanged. ### How was this patch tested? - Added `DataSourceV2DataFrameSuite` tests covering create/replace table with a translatable generation expression, a foldable generation expression, and a non-translatable generation expression. - Updated `V2TableUtilSuite` for the new column-generation-expression field. - Existing generated-column tests in `DataSourceV2SQLSuiteV1Filter` remain green. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor (Claude Opus 4.7) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
