yinli-systems commented on PR #24227: URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5676104867
The new paired benchmark is useful because it compares the same `cb3a578` head with only `DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY` changed. Correctness CI is green, but flag-on performance needs a narrower target: TPC-H total 732.41 -> 810.78 ms (~11% slower), TPC-DS 9354.95 -> 9732.14 ms (~4% slower), and ClickBench partitioned 18999.43 -> 71162.33 ms (~3.75x slower). ClickBench Q33/Q34 are `GROUP BY URL`; they become 9.20x/8.93x slower, with peak MemoryPool reservation rising 8.1 -> 12.3 GiB and 8.2 -> 12.7 GiB respectively. A concrete hypothesis, not yet a proven attribution: `DFParquetMetadata::fetch_schema` currently promotes a top-level string/binary column if *any* row group has a dictionary page, without an NDV or repetition gate. That can select an expensive dictionary path for URL-like high-cardinality group keys. Could we capture flag-off/flag-on `EXPLAIN ANALYZE`, the actual scan schema for URL, and row-group dictionary cardinalities for Q33/Q34? Those would distinguish reader/materialization cost from grouping/hash cost before choosing a fix. The default-off flag keeps existing workloads unchanged, but these A/B results argue against enabling it broadly without a cost gate. For #24117 I am keeping the staged optimizer opt-in and fail-closed: exact NDV and row-count, physical page evidence, and a high repetition threshold are required before selectively requesting Dictionary on a direct group key. #25185 may help reused dictionary-value hashing, but its standalone benchmark is not yet end-to-end evidence that it offsets this regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
