andygrove opened a new pull request, #6314: URL: https://github.com/apache/datafusion-comet/pull/6314
## Which issue does this PR close? Closes #6313. ## Rationale for this change A native block with no JVM input publishes its whole metric tree to the JVM after every output batch, ignoring `spark.comet.metrics.updateInterval`. Each publish walks the plan, aggregates every metric, encodes a protobuf and calls into the JVM. That cost 19–38 µs per batch in the measurements on the issue, or 40–78 ms for a task that returns 2,048 batches. ## What changes are included in this PR? The channel branch of `executePlan`, which serves native blocks with no JVM input, now calls `update_metrics_on_interval` instead of `update_metrics`. That's the check the `ScanExec` path already uses. `releasePlan` still publishes the final values. The task-completion listeners that read them (`reportScanInputMetrics`, `reportSpillMetrics`) are already registered to run after the iterator closes, since blocks with a JVM input rely on the same ordering. ## How are these changes tested? A new test in `CometTaskMetricsSuite` reads a 10,000-row Parquet file as ten batches with the interval disabled. The scan's `output_rows` must still be 0 after the first batch and reach 10,000 once the iterator closes. Without the fix the test fails with `output_rows` at 4,000 after one batch, because the producer runs ahead. I also ran `CometTaskMetricsSuite`, `CometExecIteratorLifecycleSuite`, the "Comet native metrics" tests in `CometExecSuite` and the metrics tests in `CometIcebergNativeSuite` locally against Spark 4.1. All 49 passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
