anew opened a new pull request, #58784:
URL: https://github.com/apache/spark/pull/58784
### What changes were proposed in this pull request?
Adds partial-update ("ignore null updates") support to Auto CDC flows
defined over Spark Connect. Partial updates keep the existing target value for
a column when an incoming CDC upsert carries `NULL` for it, instead of
overwriting with `NULL`.
- Adds an `optional bool ignore_null_updates` field (field 16) to the
`PipelineCommand.DefineFlow.AutoCdcFlowDetails` message, and regenerates the
Python protobuf stubs.
- Wires all three ignore-null inputs through the server handler
(`PipelinesHandler.buildAutoCdcFlow`) into `ChangeArgs.ignoreNullSelection`,
mirroring the existing `track_history_*` handling:
- `ignore_null_updates` (all columns) ->
`ColumnSelection.ExcludeColumns(Seq.empty)`
- `ignore_null_updates_column_list` -> `ColumnSelection.IncludeColumns`
- `ignore_null_updates_except_column_list` ->
`ColumnSelection.ExcludeColumns`
- The three inputs are mutually exclusive; specifying more than one raises
the new error condition `AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS`.
This removes the `TODO(SPARK-57093)` in the handler, which had left the two
previously-declared column-list fields (14/15) unread.
### Why are the changes needed?
Auto CDC "ignore null updates" has three user-visible states: off, a column
subset, and all columns (Databricks pipelines' `ignore_null_updates=True`). The
engine already models these as `ChangeArgs.ignoreNullSelection:
Option[ColumnSelection]`, where "all columns" is `ExcludeColumns(Seq.empty)`,
but the Connect handler discarded the two existing column-list fields behind a
TODO, so none of it was reachable from a Connect client.
The two repeated fields can express the subset cases but not "all columns":
an empty repeated field is indistinguishable from an unset one on the wire, so
an empty except-list collapses to "unset" (off). Expressing "ignore nulls on
all columns" therefore requires an explicit scalar flag, named
`ignore_null_updates` to match the flag in Databricks pipelines.
### Does this PR introduce _any_ user-facing change?
No. Declarative pipeline flows are defined via Spark Connect; the Python API
surface that sets these fields is added in a follow-up. The engine behavior
itself is unchanged.
### How was this patch tested?
New tests in `SparkDeclarativePipelinesServerSuite` drive the raw Connect
stub and assert that the all-columns flag and an include list map to the
expected `ignoreNullSelection`, and that conflicting options are rejected
server-side. `SparkThrowableSuite` validates the new error condition.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
