[
https://issues.apache.org/jira/browse/SPARK-58321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned SPARK-58321:
---------------------------------
Assignee: Andreas Neumann
> Wire SCD2 AutoCDC streaming write and enable SCD2 end to end
> ------------------------------------------------------------
>
> Key: SPARK-58321
> URL: https://issues.apache.org/jira/browse/SPARK-58321
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 5.0.0
> Reporter: Andreas Neumann
> Assignee: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
>
> Add Scd2MergeStreamingWrite (mirroring Scd1MergeStreamingWrite in
> FlowExecution.scala) that resolves the auxiliary-table identifier, constructs
> Scd2ForeachBatchHandler with an Scd2BatchProcessor, and drives it via
> .foreachBatch. Route ScdType.Type2 to it in FlowPlanner (currently throws
> AUTOCDC_SCD2_NOT_SUPPORTED). Remove the AUTOCDC_SCD2_NOT_SUPPORTED gate in
> Flow.scala and FlowPlanner.scala, and update the existing tests that assert
> the gate. Add end-to-end pipeline tests that register an SCD2 flow through
> the graph, run real microbatches, and assert target + auxiliary table
> contents. This is the atomic 'flip the switch' change and depends on the SCD2
> schema derivation and SCD2 auxiliary-table spec subtasks. Sub-task of the
> SCD2 end-to-end enablement effort under SPARK-56249.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
