huan233usc opened a new pull request, #17953: URL: https://github.com/apache/iceberg/pull/17953
## Summary Expose Iceberg change data through the Spark 4.2 Data Source V2 `Changelog` API. - Implement `TableCatalog.loadChangelog` in the Iceberg Spark catalogs. - Support Spark `CHANGES` queries and `readStream().changes()` using Iceberg incremental changelog scans. - Map Iceberg snapshot sequence numbers and timestamps to Spark `_commit_version` and `_commit_timestamp`. - Use Iceberg v3 row lineage (`_row_id` and `_last_updated_sequence_number`) as Spark row identity and row version. - Advertise copy-on-write carry-over and delete/insert update semantics so Spark can drop unchanged rewrite rows and optionally produce update pre/post images. - Preserve the existing `table.changes` schema and behavior. This first PR intentionally supports copy-on-write tables with v3 row lineage. Merge-on-read delete-file planning is deferred to a follow-up. Related to #17949. ## Test Plan Added Spark 4.2 coverage for: - version-range `CHANGES` reads - default copy-on-write carry-over removal - raw copy-on-write changes with `deduplicationMode = none` - update pre/post images with `computeUpdates = true` - micro-batch `readStream().changes()` - compatibility with the existing `table.changes` relation Local execution of the focused Gradle test is blocked during project configuration because the managed host configuration resolves Maven Central to localhost. `git diff --check` and focused code review completed successfully; CI is expected to run the Spark 4.2 suite. --- **AI Disclosure** - Model: [unknown - human to fill in] - Platform/Tool: Cursor - Human Oversight: partially reviewed - Prompt Summary: Split the Spark 4.2 CDC prototype into a first copy-on-write PR based on merged upstream main, with correct v3 row-lineage carry-over and update semantics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
