Rich-T-kid commented on PR #12008: URL: https://github.com/apache/arrow-rs/pull/12008#issuecomment-6625508336
I ran the partitioned ClickBench suite twice with Parquet RLE-to-dictionary enabled: [run 1](https://github.com/apache/datafusion/pull/27077#issuecomment-6623071270), [run 2](https://github.com/apache/datafusion/pull/27077#issuecomment-6624182651). Both sides use DataFusion [#26816](https://github.com/apache/datafusion/pull/26816) at `747baf5f92`. The baseline pins arrow-rs `258b0d2031` (this PR's base); the changed side pins `31c0d5c510` (this PR's current head). The dependency and lockfile plumbing is otherwise identical. Configuration: `clickbench_partitioned`, 5 iterations per query, `DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY=true`, and a 32 GiB DataFusion memory limit. | Query | Run 1 | Run 2 | | --- | ---: | ---: | | Q16 | 2308.67 -> 1404.65 ms (1.64x) | 2352.52 -> 1464.47 ms (1.61x) | | Q17 | 2297.98 -> 1456.22 ms (1.58x) | 2438.17 -> 1432.44 ms (1.70x) | | Q33 | 7959.94 -> 4073.52 ms (1.95x) | 7893.80 -> 4356.48 ms (1.81x) | | Q34 | 7794.07 -> 4041.45 ms (1.93x) | 8315.91 -> 4342.55 ms (1.91x) | These are the runner's minimum times, which it uses for the comparison verdict. Total minimum time fell by 22.3% and 22.5% in the two runs. The first run classified 9 queries faster and 6 slower; the repeat classified 12 faster and none slower. The smaller changes were noisy, but the gains on these four queries repeated. One caveat: the first changed-side Q33 run had a single spill-heavy 21.6 s iteration, so its five-run mean was worse despite the 1.95x minimum-time result. That did not recur: Q33 was 4.36-4.70 s on the changed side in the repeat versus 7.89-8.16 s at baseline. The repeat still spilled 400 MiB and reported higher pool peaks for Q33/Q34, so the speedup is clear but the memory behavior is worth keeping an eye on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
