adriangb commented on PR #23390: URL: https://github.com/apache/datafusion/pull/23390#issuecomment-4919778250
## ClickBench A/B regression: diagnosed and fixed (commit 3a03ec5b73) The earlier on/off A/B showed clickbench_partitioned +5.1% with the flag on (17 queries slower, up to 1.2x on string-heavy scans). `EXPLAIN ANALYZE` attribution on Q22: **7.96K of 12.3K scan batches hit the waste-compaction path** — parquet-decoded string batches routinely retain 2-4x their logical size (decode buffers shared across consecutive batches), tripping the flat 1MB absolute-waste floor on nearly every batch and paying a full copy each time for no benefit. Fix: the absolute-waste floor now scales to `max(1MiB, target_batch_size_bytes)` — pinned memory only matters at the scale of the byte target, and the shared-decode-buffer false positives sit well below it. After the fix Q22 shows 0 compacted / 12.2K passed through. Fresh GKE A/B on the fixed head (`2cf5402b58` both sides, flag on only for `changed`): | suite | off → on | before fix | |---|---|---| | tpch | 758.9 → 764.4ms (+0.7%), 22/22 no-change | +1.1%, 1 slower | | clickbench_partitioned | 19437 → 19790ms (**+1.8%**), 5 slower / 2 faster / 36 no-change | +5.1%, 17 slower / 2 faster | The former regressors (Q13/Q17/Q20/Q21/Q22/Q33/Q34) are all back to no-change. Residuals are Q23 at 1.07x (a wide-row `SELECT *` TopK where the byte target legitimately re-chunks output) and a few 1.06-1.14x entries with overlapping error bars. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UcPTREZVLXsSZDRCae33gm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
