JonSnow1807 commented on issue #7073: URL: https://github.com/apache/datafusion-comet/issues/7073#issuecomment-6806832902
I re-ran the nested hash key cases of `CometShuffleBenchmark` (`--nested-hash-only`) on main (e59d9373), Spark 4.1.3, `local[5]`, Ryzen 9 5900X, twice. JVM shuffle time divided by native shuffle time, the worse of the two runs, 5 / 201 partitions: | Key | Now | #5788 | |---|---|---| | `struct<int, string>` | 1.80 / 1.96 | 3.02 / 7.13 | | `array<int>` | 1.37 / 1.83 | 2.23 / 5.19 | | `struct<array<int>, string>` | 1.47 / 1.88 | 3.14 / 3.56 | | `array<struct<int, string>>` | 1.61 / 1.71 | 0.72 / 0.61 | | `struct<map<string, int>, int>`, 1 entry | 1.79 / 2.01 | 2.40 / 2.43 | | same, 2-10 entries | 0.96 to 1.05 | 1.37 to 1.79 | | same, 2-50 entries | 0.96 to 1.00 | 0.86 to 1.39 | Since #5778, native is faster for struct and array keys in every case, while maps with several entries are about even. (#5788 ran on other hardware, so only which side wins carries over.) Would you take a PR that turns native hash partitioning on by default for struct and array keys and keeps maps behind the flag? Or would you rather flip the whole flag, since maps are even rather than slower? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
