unikdahal opened a new pull request, #6640: URL: https://github.com/apache/datafusion-comet/pull/6640
## Which issue does this PR close? Closes apache/datafusion-comet#6606. ## Rationale for this change Spark 4.1+ attaches eight semantic action counters to MERGE summaries. Native MergeRows must preserve those counters and use a writer that consumes them. Stock V2 writers discover Spark's concrete MergeRowsExec, so converting it can silently remove the summary, particularly when AQE replans the query without its enclosing writer. This follow-up depends on apache/datafusion-comet#5318 ## What changes are included in this PR? - Carry matched, not-matched, and target-only action context into native instructions and publish all eight counters for Keep, Discard, and Split. - Add version-specific metric access, including successful-task-attempt and latest-RDD handling for Spark 4.2. - Gate Spark 4.1+ conversion on eligibility applied to a cloned logical query by the Comet split Iceberg writer. Preserve that eligibility through AQE; keep stock V2 writers on Spark MergeRowsExec. - Forward native metrics through the summary-aware Iceberg commit and document the supported write path. - Retain the default-off flag and existing cardinality-memory compatibility warning; enforce parity for identical older metric shims. ## How are these changes tested? - 26 focused native MergeRows tests passed after building the debug JNI library. - Full-reactor Spark 4.1 run: 87 tests passed, zero failed, one Spark-4.2-only retry test cancelled. Suites: CometIcebergWriteActionSuite and CometMergeRowsSuite. - Full-reactor Spark 4.2 CometMergeRowsSuite: four tests passed, zero failed, zero cancelled, including task retries and latest-RDD metric selection. - Native/Spark counter parity covers Keep, Discard, Split, and all three action contexts. Native plan assertions and disabled broadcast thresholds ensure native execution is exercised. - Spark-only Iceberg siblings validate complete row results and all eight snapshot counters. CoW and MoR partitioned regressions run with AQE both on and off. - Stock V2 regression covers both AQE settings. Spark 4.2 retry coverage injects failed attempts and repeated cumulative snapshots, then runs a second RDD against the same metrics. - Versioned-source parity, suite-registration, and whitespace checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
