[
https://issues.apache.org/jira/browse/SPARK-58313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Neumann updated SPARK-58313:
------------------------------------
Description:
For SCD 2 auto CDC to work properly, we must ensure certain invariants, for
example:
- all sequence columans are included in the target
- all tracking columns are included in the target
- no actual data columns collide with the autocdc metadata prefix
Most of these can only be validated at flow analysis time, because we usually
only know the data schema after analysis.
was:
For SCD 2 auto CDC to work properly, we must ensure certain invariants, for
example:
- all sequence columans are included in the target
- all tracking columns are included in the target
- no actual data columns collide with the autocdc metadata prefix
- more tbd
Most of these can only be validated at flow analysis time, because we usually
only know the data schema after analysis.
> Additional Validation at Graph Registration and Flow Analysis time
> ------------------------------------------------------------------
>
> Key: SPARK-58313
> URL: https://issues.apache.org/jira/browse/SPARK-58313
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.3.0
> Reporter: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
>
> For SCD 2 auto CDC to work properly, we must ensure certain invariants, for
> example:
> - all sequence columans are included in the target
> - all tracking columns are included in the target
> - no actual data columns collide with the autocdc metadata prefix
> Most of these can only be validated at flow analysis time, because we
> usually only know the data schema after analysis.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
