szehon-ho opened a new pull request, #57481:
URL: https://github.com/apache/spark/pull/57481
### What changes were proposed in this pull request?
This PR moves the gate that decides whether Spark auto-fills and enforces
generated column values on a DSv2 write from a **catalog-level** capability to
a **table-level** capability.
Specifically:
- Adds `TableCapability.GENERATE_COLUMN_VALUES_ON_WRITE`, checked via
`Table.capabilities()`.
- Removes `TableCatalogCapability.SUPPORT_GENERATED_COLUMN_ON_WRITE` (added
by SPARK-57644 and still unreleased).
- `GeneratedColumn.supportsGeneratedColumnsOnWrite` now takes a `Table` and
checks the table's capabilities instead of the `TableCatalog`'s.
- Updates the four call sites to consult the table:
- `Analyzer` (exposing generation-expression metadata to
`TableOutputResolver` so missing values are auto-filled),
- `ResolveTableConstraints` (adding the generated-column `CheckInvariant`
constraints),
- `RewriteRowLevelCommand` (blocking MERGE/UPDATE),
- `ResolveWriteToStream` (blocking streaming writes).
- Test infra: the in-memory test table advertises the new capability, gated
by a `generate-column-values-on-write` table property (default `true`),
mirroring the existing `accept-any-schema` / `auto-schema-evolution` toggles.
### Why are the changes needed?
Whether Spark should generate/enforce generated column values is a property
of an individual **table**, not of the whole **catalog**. A single catalog can
expose tables from formats or protocol versions with differing generated-column
support, so a catalog-wide flag is too coarse.
This also aligns with how Delta Lake gates the feature: it is gated
per-table on the table's protocol.
`GeneratedColumn.satisfyGeneratedColumnProtocol(protocol)` returns
`protocol.isFeatureSupported(GeneratedColumnsTableFeature)` -- i.e. it checks
whether the `GeneratedColumnsTableFeature` is enabled in that specific table's
protocol. A per-table `TableCapability` is the natural DSv2 analogue and
matches the existing pattern for other per-table write behaviors
(`BATCH_WRITE`, `OVERWRITE_DYNAMIC`, `TRUNCATE`, ...).
### Does this PR introduce _any_ user-facing change?
No change relative to a released version. The catalog capability being
removed (`SUPPORT_GENERATED_COLUMN_ON_WRITE`) was introduced by SPARK-57644 and
has not shipped in any release, so this only reshapes an unreleased API within
the development branch. Connectors opt in by having their `Table` advertise
`TableCapability.GENERATE_COLUMN_VALUES_ON_WRITE`.
### How was this patch tested?
`GeneratedColumnWriteSuite` (69 tests) passes. The existing "connector
without the write capability does not auto-fill or enforce generated columns"
coverage is retained (renamed to "table without write capability ..."), now
driven by creating a table with `TBLPROPERTIES
('generate-column-values-on-write' = 'false')` instead of a dedicated
no-capability catalog.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
