retroaktive commented on issue #21719: URL: https://github.com/apache/datafusion/issues/21719#issuecomment-5006501256
A data point for this discussion, from a mixed-workload fairness measurement we ran on DataFusion 52.5's stock tokio-stream execution (multi-threaded runtime, operators unmodified). **Setup**: open-loop generator (arrivals don't wait for completions), 1,000 QPS of small partition-pruned point queries, first isolated, then mixed with 5 concurrent heavy full-scan aggregations, 30 s per phase, two back-to-back runs to check reproducibility (headline metrics repeated within ±5 %). Hardware: GCP c3-standard-22 (22 vCPU), OS background services quiesced, data on local NVMe. **Result**: the tail *shape* stays healthy under contention — p99/p95 ≈ 1.06 in both phases — but the whole small-query latency distribution shifts up when the heavy scans run: p95 inflates **1.38–1.40×** vs isolated (1.86–1.95 ms → 2.60–2.69 ms), consistent across both runs. No queries shed or failed in either phase. The failure mode looks like the one discussed here: with pull-based streams on a shared runtime, heavy operators get scheduling weight proportional to their poll appetite, and there is no mechanism to hold interactive latency steady against them short of running separate runtimes/pools per class. The absolute numbers are small (sub-3 ms), but for latency-SLO'd interactive workloads the ~1.4× multiplier is workload-structural, not noise — it tracks the heavy queries' presence, not load level. We had pre-registered ≤1.3× as our acceptable inflation bound before running (against the morsel-driven-scheduler literature's fairness claims), so for our use case this measurement tipped the decision toward building a morsel-style driver above DataFusion's operators (operators unchanged — only the stream *driver* replaced, roughly the seam the 2022 scheduler explored). Happy to share more methodology details if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
