unikdahal opened a new issue, #6606:
URL: https://github.com/apache/datafusion-comet/issues/6606
### What is the problem?
Native `MergeRowsExec` support introduced in #5318 is currently limited to
Spark 3.5.x and 4.0.x.
Spark 4.1 introduced a writer-side `MergeSummary` contract. The V2 write
path derives semantic MERGE action counts from `MergeRowsExec` and passes them
through the summary-aware:
```scala
BatchWrite.commit(messages, summary)
```
Replacing Spark's `MergeRowsExec` with a native operator without preserving
those metrics would change the writer commit contract even when the produced
rows are otherwise identical.
For that reason, Spark 4.1+ currently keeps `MergeRowsExec` on the JVM.
### Proposed solution
Preserve the Spark 4.1+ MERGE summary contract end-to-end while allowing
`MergeRowsExec` itself to execute natively.
This requires:
- carrying MERGE action context through native instructions
- copy
- insert
- update
- delete
- exposing Spark-compatible semantic counters:
- `numTargetRowsCopied`
- `numTargetRowsInserted`
- `numTargetRowsDeleted`
- `numTargetRowsUpdated`
- `numTargetRowsMatchedUpdated`
- `numTargetRowsMatchedDeleted`
- `numTargetRowsNotMatchedBySourceUpdated`
- `numTargetRowsNotMatchedBySourceDeleted`
- preserving the Spark-version-specific metric semantics used by
`MergeSummary`
- making the V2 write path consume the native MergeRows metrics when
constructing the summary
- forwarding the resulting `MergeSummary` through the summary-aware
`BatchWrite.commit` path
- enabling native `MergeRowsExec` registration on Spark 4.1+ once the
contract is preserved
### Validation
Add differential coverage against Spark for:
- matched update
- matched delete
- not-matched insert
- not-matched-by-source update
- not-matched-by-source delete
- the eight semantic action counters
- final table contents
- the `MergeSummary` observed by the V2 writer
Spark 4.2 should also preserve its retry / last-attempt metric behavior.
### Scope
This issue is specifically about the Spark 4.1+ `MergeSummary` compatibility
boundary for native `MergeRowsExec`.
Native Iceberg merge-on-read / `WriteDeltaExec` support is separate and does
not need to block this work.
### Related
- #5318 — native `MergeRowsExec`
- #5122 — row-level MERGE / UPDATE / DELETE acceleration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
