rahulsmahadev opened a new pull request, #17955: URL: https://github.com/apache/iceberg/pull/17955
Persists the row-level operation metrics that Spark 4.2 reports via the new `InsertSummary`, `UpdateSummary`, and `DeleteSummary` write summaries into the Iceberg snapshot summary, mirroring the existing `MergeSummary` support added in #15014. Spark 4.2's `WriteToDataSourceV2Exec` passes an operation-specific `WriteSummary` to `BatchWrite.commit(messages, summary)` for INSERT (append), UPDATE, and DELETE. This change consumes them in Iceberg's commit path and records the following snapshot summary fields: - `spark.insert.num-inserted-rows` - `spark.update.num-updated-rows`, `spark.update.num-copied-rows` - `spark.delete.num-deleted-rows`, `spark.delete.num-copied-rows` `num-copied-rows` counts unmodified rows rewritten in copy-on-write mode and is `0` in merge-on-read. The append path (`BatchAppend`) now forwards the `InsertSummary` on commit; the copy-on-write and merge-on-read row-level paths already forward their summaries. Metric tests are added for copy-on-write and merge-on-read, and the new fields are documented in `spark-writes.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
