AnishMahto opened a new pull request, #56160:
URL: https://github.com/apache/spark/pull/56160
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
- **Problem.** An AutoCDC flow's key columns determine how rows are matched
during the merge into the target. If a user changes the declared keys
between
pipeline runs without a full refresh, the merge silently mis-routes rows
(updates become inserts and vice versa) and corrupts the target. We need to
detect a key change between runs and fail fast.
- **Mechanism.** Record the resolved key columns into a reserved property on
the auxiliary table the first time it is created; on subsequent runs,
validate the flow's declared keys against this recorded value and fail with
`AUTOCDC_INVALID_STATE.KEY_SCHEMA_DRIFT` on mismatch.
- **Implementation notes.** Aux-table creation switched from `CREATE TABLE`
DDL to the programmatic V2 `TableCatalog.createTable` API, so the JSON
property reaches storage as raw bytes (no SQL-literal escape layer). Three
new structured sub-classes under `AUTOCDC_INVALID_STATE` cover
corrupted-metadata cases caused by user tampering, all recommending
full-refresh as remedy.
### Why are the changes needed?
To prevent users from incorrectly changing AutoCDC keys across runs without
a full refresh.
### Does this PR introduce _any_ user-facing change?
No, unreleased change.
### How was this patch tested?
Unit tests.
### Was this patch authored or co-authored using generative AI tooling?
Claude Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
