peterxcli opened a new pull request, #5916: URL: https://github.com/apache/datafusion-comet/pull/5916
## Which issue does this PR close? Closes #3859. Stacked on #5807; only the last three commits belong to this PR. ## Rationale for this change The native shuffle writer spilled each output partition to its own file and kept every file open until the merge. With many partitions a task creates, opens, closes and unlinks thousands of files, and under the default 1024 soft `nofile` limit a task spilling 1000 partitions fails with `Too many open files`. ## What changes are included in this PR? - `PartitionedSpill` replaces the per-partition `SpillWriter`s: one spill file per task and the byte ranges each partition's blocks occupy. A failed write makes the spill unusable. - `finish_partition` copies a partition's ranges into the output in write order. A range that fits in the write buffer is read with one `read_exact_at`; longer ranges keep `io::copy`. A spill file shorter than its ranges fails the task. - Spill writes share one `BufWriter` across partitions, flushed before the merge reads the file. ## How are these changes tested? New unit tests: partitions interleaved across spill rounds read back in write order through both copy paths, one spill file for any partition count, truncated spill file, spill unusable after a failed write, writes buffered until flush. `datafusion-comet-shuffle` 132 passed, clippy clean. `shuffle_bench` output is byte-identical to #5807's at 64, 1000 and 4000 partitions with 8 spills and at 200 partitions with 49 spills. ### Benchmarks `shuffle_bench`, 8M rows (5 numeric columns), lz4, 16-core Linux host with a rotational disk, 3 rounds in rotating order. Mean time change against #5807: | partitions | `--max-buffer-bytes` (spills) | change | | --- | --- | --- | | any | no spill | within ±3% | | 200 | 128 MiB / 32 MiB (2 / 11) | +4.3% / +2.0% | | 200 | 4 MiB (88) | -5.8% | | 1000 | 128 MiB / 32 MiB (2 / 11) | -1.6% / -1.2% | | 1000 | 4 MiB (88) | -7.7% | | 4000 | 128 MiB / 32 MiB (2 / 11) | -6.6% / -10.2% | | 16000 | 128 MiB / 32 MiB (2 / 11) | -17.5% / -19.7% | At 200 partitions with 49 spills, syscalls drop from 12,152 to 10,375 (`write` 9,846 to 263, no per-partition `openat`/`unlink`). TPC-H SF1/SF10/SF100 and ClickBench (Spark 4.1, 2 executors x 8 cores) never spill at default settings and show no difference. These end-to-end runs used the first commit only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
