ranflarion opened a new issue, #5311: URL: https://github.com/apache/datafusion-comet/issues/5311
During the dicomed_objects_images investigation (2026-08-07, reducer-locality funnel), the executor that received 19/60 tasks of a 1.6 TiB shuffle stage showed tasks at 5413s wall with 1009s CPU and Spark task metrics reporting Disk/Memory Bytes Spilled = 0, which sent three separate investigations toward deadlock, data-skew, and OptimizeWrite theories. The per-operator SQL accumulators had the real story the whole time: ~10 GiB sort spill plus ~7.5 GiB shuffle-writer spill per task, in ~87 spill rounds per task (1653 spill events across 19 tasks), driven by the fair pool dividing 54 GiB across 19 tasks and their consumers. Shuffle Write Time was 2550s median on that executor vs 46s for the same task running solo, and ~1800s per task remains unattributed because sort-spill I/O has no time metric anywhere. The existing "memory pool time" metric read ~0.5s/task, so either blocking is genuinely negligible (making the spill storm the whole story) or the metric does not cover the block ing site. Native execution degradation is recorded only in per-plan-node SQL accumulators, never in Spark TaskMetrics, so stage summaries, the UI spill columns, event-log TaskEnd rows, analyze-job, and the telemetry backend all show a healthy-looking stage while the native side thrashes. Three additions close the gap: 1. Bridge native spill into TaskMetrics. At the existing metric-sync points (CometExecIterator close, CometNativeShuffleWriter), call taskMetrics().incDiskBytesSpilled / incMemoryBytesSpilled alongside the SQLMetric updates (spillMetrics defined in CometMetricNode.scala:172-178; the values already cross JNI). Dedupe against JVM-side paths that already increment TaskMetrics (JVM columnar shuffle sorter), and document the mapping (DataFusion reports uncompressed in-memory size and file bytes; Spark's memory/disk counters have their own conventions, and tools computing ratios will misread a sloppy mapping). 2. Add a sort-spill I/O time metric. Wrap the DataFusion spill write/read path (we own the pool and can wrap the DiskManager) with nano timing surfaced as a per-operator metric, so the ~1800s/task hole above becomes attributable. 3. Audit "memory pool time" coverage. Verify it covers the JNI acquire into Spark's TaskMemoryManager (which can sleep in ExecutionMemoryPool) and the try_grow retry loops in flarion_fair_pool.rs, or add a dedicated memory_wait_time. Fold in the fair-pool metrics/release issue found by the component-spec verification so this surface is only touched once. Acceptance: for a task like the ones above, wall time decomposes into cpu + fetch_wait + shuffle_write + spill_io + pool_wait + small residual using only stage-level/event-log data, and the stage summary shows the native spill volume. This is diagnosis-only (no runtime behavior change); expected cost is negligible counters/timers around already-expensive operations. Side effects to flag: Comet stages will start reporting nonzero spill, which shifts dashboard baselines and any spill-based alerting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
