sunchao commented on code in PR #7199:
URL: https://github.com/apache/datafusion-comet/pull/7199#discussion_r4417630504
##########
native/shuffle/benches/shuffle_writer.rs:
##########
@@ -162,28 +162,36 @@ fn criterion_benchmark(c: &mut Criterion) {
group.finish();
// High partition counts stress the per-partition write path (one
short-lived
- // buffered writer per partition), which low counts barely exercise;
compression
- // is disabled to isolate it. Few samples: each iteration is a full
end-to-end
- // write across thousands of partitions.
+ // buffered writer per partition), which low counts barely exercise. The
None rows
+ // isolate that path. The Lz4Frame rows, what a wide shuffle runs by
default, add the
+ // per-block codec cost on top: at 8000 partitions each partition gets
about ten rows,
+ // so every block is tiny and still pays for a fresh LZ4 frame encoder,
its header and
Review Comment:
Could the comment and results describe the repeated input behind the codec
crossover? `create_batches` clones the same 8,192 rows ten times. At 8,000
partitions, the average of roughly ten rows includes 2,844 empty partitions,
and 2,996 partitions contain ten copies of a single row. This affects how many
frames LZ4 writes as well as the payload it compresses.
I parsed the writer output for this case and counted 5,156 LZ4 blocks
(2,822,900 bytes). A scratch control that changed only the fixture to generate
fresh row values for each batch wrote 7,999 blocks (5,548,973 bytes). The
timings are a valid comparison for the repeated fixture. Please scope the
crossover claim to that input, or add a matched None/LZ4 pair using distinct
batches under new IDs if the goal is to cover that workload too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
