AnishMahto opened a new pull request, #56208:
URL: https://github.com/apache/spark/pull/56208
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
**Preamble:**
The SCD type 2 flow is a foreachBatch streaming query on an input
change-data-feed, and is responsible for reconciling the incoming change data
onto some target table that follows SCD2 replication semantics.
SCD2 flows also maintain an "auxiliary" table to keep track of
early-arriving out-of-order received events state. Each microbatch will need to
reconcile against this auxiliary table as well, and update the auxiliary
table's state appropriately for future microbatches.
**Preprocess Microbatch**
For SCD2, preprocessing the microbatch is all about getting it in the right
shape, aligned with the shape of the target table the microbatch will be merged
into + the shape that SCD2 itself as a standard demands.
That is:
The microbatch must have a start-at and end-at columns projected as per the
SCD2 standard, to indicate that a historical/alive record was active between
those sequence stamps
The microbatch will have the operational CDC metadata column projected,
which is needed to reconcile late arriving events/bookkeeping
As per the Spark AutoCDC API, the microbatch should project down to just the
user-specified column selection
Implement the part of the core SCD2 microbatch processor that does this
microbatch preprocessing.
### Why are the changes needed?
To support AutoCDC SCD2 transformations, as per the approved SPIP:
https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
### Does this PR introduce _any_ user-facing change?
No. New feature.
### How was this patch tested?
`Scd2BatchProcessorSuite`
### Was this patch authored or co-authored using generative AI tooling?
Co-written with Claude Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
