HarshilMital opened a new pull request, #7200: URL: https://github.com/apache/datafusion-comet/pull/7200
## Which issue does this PR close? Part of #7080. ## Rationale for this change Hash placement scatters rows, so the flush gathers each output partition back out a row at a time with `interleave_record_batch`. For list and map columns that means rebuilding offsets and recursing into child arrays, but the existing gather fixtures have no list or map columns at all. ## What changes are included in this PR? `shuffle_gather_nested`: the existing `BenchRepartitioner` `place` and `place+gather` harness, under `CometPartitioning::Hash` (50 partitions, 8 × 8192 rows), over four shapes: `List<Int64>`, `List<Struct<Int64, Utf8>>`, `Map<Utf8, Int64>` and `Struct<Int64, Utf8, Float64>`. Each schema is an Int64 key plus four columns of the shape. List lengths vary 0–8 and map lengths 1–7 by row; every value is per-row and every batch is built separately. The hash reads only the key and the discarding writer keeps IPC and disk out, so `place+gather` minus `place` is the nested gather. 8 bench IDs. ### Results **Machine:** 8 vCPU Arm Neoverse-V2 (aarch64), 31 GiB RAM, rustc 1.99.1 **Commit:** `d53f78a`. Median with Criterion's 95% interval. Reproduce from `native/`: ```shell cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- 'shuffle_gather_nested' --save-baseline main ``` | Shape | `place` | `place+gather` | Gather alone | |---|---:|---:|---:| | list_int64 | 809.42 µs | 7.01 ms | 6.21 ms | | list_struct | 822.17 µs | 31.68 ms | 30.85 ms | | map_utf8_int64 | 827.69 µs | 29.51 ms | 28.68 ms | | struct | 831.65 µs | 6.71 ms | 5.88 ms | Gathering `List<Struct>` and `Map<Utf8, Int64>` costs about 5× what `List<Int64>` or a flat struct does. All intervals are within ±1%. ## How are these changes tested? This PR only adds benchmark code under `native/shuffle/benches/`; no production code changes. - `cargo fmt --all -- --check` and `cargo clippy -p datafusion-comet-shuffle --all-targets -- -D warnings` pass. - `cargo bench -p datafusion-comet-shuffle --bench shuffle_writer --no-run` builds. - Every new case runs once in Criterion's test mode: `cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- --test 'shuffle_gather_nested'`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
