andygrove commented on PR #7234: URL: https://github.com/apache/datafusion-comet/pull/7234#issuecomment-6444300243
@viirya I pushed c5ce2b151d, which drops the view-type guard and the unused `batch_size` plumbing, and updated the contributor guide row. I also reran `shuffle_writer_high_partition` against current main (6fe73d9be8, no `native/shuffle` changes since the branch point). Both sides were built with `lto=false` and `codegen-units=16`, and I alternated them for three rounds. These are the medians of the three Criterion estimates: | Partitions | main | This PR | Change | |---:|---:|---:|---:| | 200 | 3.66 ms | 3.07 ms | -16% | | 2000 | 13.32 ms | 8.06 ms | -39% | | 8000 | 30.96 ms | 17.88 ms | -42% | The gain is smaller than the old table, as you expected, since #5565 and #5916 already landed. The machine was shared (load 6 to 11), and one 8000-partition run on the PR came in at 35.9 ms against 15.6 and 17.9 ms for its other two, so treat the 8000 row as the roughest. The description has the same table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
