Andreas Neumann created SPARK-58321:
---------------------------------------
Summary: Wire SCD2 AutoCDC streaming write and enable SCD2 end to
end
Key: SPARK-58321
URL: https://issues.apache.org/jira/browse/SPARK-58321
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 5.0.0
Reporter: Andreas Neumann
Add Scd2MergeStreamingWrite (mirroring Scd1MergeStreamingWrite in
FlowExecution.scala) that resolves the auxiliary-table identifier, constructs
Scd2ForeachBatchHandler with an Scd2BatchProcessor, and drives it via
.foreachBatch. Route ScdType.Type2 to it in FlowPlanner (currently throws
AUTOCDC_SCD2_NOT_SUPPORTED). Remove the AUTOCDC_SCD2_NOT_SUPPORTED gate in
Flow.scala and FlowPlanner.scala, and update the existing tests that assert the
gate. Add end-to-end pipeline tests that register an SCD2 flow through the
graph, run real microbatches, and assert target + auxiliary table contents.
This is the atomic 'flip the switch' change and depends on the SCD2 schema
derivation and SCD2 auxiliary-table spec subtasks. Sub-task of the SCD2
end-to-end enablement effort under SPARK-56249.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
