Rich-T-kid commented on PR #26789: URL: https://github.com/apache/datafusion/pull/26789#issuecomment-6451190236
The follow-up comparison against the default main `Utf8View` path is complete. Both sides used `schema_force_view_types=true` and a 32 GB memory limit; the baseline used `enable_rle_to_dictionary=false`, while the PR used `enable_rle_to_dictionary=true` so the physical optimizer rule was exercised. Results from the distribution tables: - [ClickBench](https://github.com/apache/datafusion/pull/26789#issuecomment-6450975257): 19.52s baseline vs. 36.05s PR; 3 faster, 28 slower, 12 unchanged. - [TPC-H](https://github.com/apache/datafusion/pull/26789#issuecomment-6450829213): 858.38ms baseline vs. 917.94ms PR; 3 faster, 6 slower, 13 unchanged. - [TPC-DS](https://github.com/apache/datafusion/pull/26789#issuecomment-6450887793): 9.66s baseline vs. 9.80s PR; 14 faster, 25 slower, 60 unchanged. - [Wide schema](https://github.com/apache/datafusion/pull/26789#issuecomment-6451170025): 7 of 8 cases were numerically slower; Q03 narrow was 5% faster. So the current rule substantially improves dictionary-enabled execution, especially the expensive high-NDV group-bys, but it does not make globally enabled dictionary reads competitive with default main using `Utf8View`. The earlier zero-regression results only establish PR vs. dictionary-enabled main, not PR vs. the actual default representation. The remaining gap is broader than aggregation: decoding immediately before the partial aggregate avoids downstream dictionary aggregation/repartition costs, but the scan has already produced dictionary arrays and non-aggregate consumers still receive them. Reaching parity with the real baseline requires dictionary promotion itself to be consumer-aware, or retaining `Utf8View` whenever the plan cannot prove a dictionary-aware consumer will amortize the scan and conversion cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
