[
https://issues.apache.org/jira/browse/SPARK-58409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho resolved SPARK-58409.
-------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57689
[https://github.com/apache/spark/pull/57689]
> Add SCD2 end-to-end test suites to match SCD1 AutoCDC coverage
> --------------------------------------------------------------
>
> Key: SPARK-58409
> URL: https://issues.apache.org/jira/browse/SPARK-58409
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 5.0.0
> Reporter: Andreas Neumann
> Assignee: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> SCD2 AutoCDC end-to-end (graph-execution) test coverage is much thinner than
> SCD1's. SCD1 has eight E2E suites under
> sql/pipelines/src/test/scala/.../graph, but only two have SCD2 counterparts
> today.
> Have SCD2 equivalents:
> - AutoCdcScd1SinglePipelineSuite -> AutoCdcScd2SinglePipelineSuite
> - AutoCdcScd1AuxiliaryTableSpecSuite -> AutoCdcScd2AuxiliaryTableSpecSuite
> Also, the randomized order-invariance suite AutoCdcOutOfOrderConvergenceSuite
> (formerly AutoCdcScd1OutOfOrderConvergenceSuite) now runs for both SCD1 and
> SCD2 (SPARK-58321), and AutoCdcConfigDriftSuite covers SCD2 sequencing-type
> and track-history drift (SPARK-58391).
> Missing SCD2 counterparts (SCD1 suite -> gap):
> - AutoCdcScd1FullRefreshSuite -> no SCD2 full-refresh suite. Full refresh
> drops and recreates both the target and the (richer) SCD2 auxiliary table;
> worth dedicated coverage rather than assuming the SCD1 test generalizes.
> - AutoCdcScd1MultiPipelineSuite -> no SCD2 multi-pipeline suite (multiple
> pipelines / flows writing to the same SCD2 target).
> - AutoCdcScd1SchemaEvolutionSuite -> no SCD2 schema-evolution suite (evolving
> user columns on an SCD2 target alongside the framework columns,
> nested/struct/array evolution, etc.).
> - AutoCdcScd1TargetTableDurabilitySuite -> no SCD2 target-table durability
> suite.
> - AutoCdcScd1AuxiliaryTableDurabilitySuite -> no SCD2 auxiliary-table
> durability suite.
> - AutoCdcScd1KeyDriftSuite -> no SCD2 key-drift suite. (Key-drift validation
> is SCD-type-agnostic, exercised end-to-end only for SCD1; the SCD2 path
> should have equivalent coverage.)
> Proposed work: add the missing SCD2 E2E suites, reusing the shared
> AutoCdcGraphExecutionTestMixin helpers (scd2MetadataDdl,
> singleAutoCdcFlowPipeline/autoCdcFlow with scdType = Type2). Discovered while
> extending the out-of-order convergence test to SCD2 (SPARK-58321) and
> auditing AutoCDC test coverage.
> Note: this became actionable only once SCD2 flows are runnable end to end
> (SPARK-58321), which these suites depend on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
