dwsmith1983 commented on PR #6336: URL: https://github.com/apache/datafusion-comet/pull/6336#issuecomment-6073924290
> Could `spill()` call `write_burst_complete` after `self.reservation.free()`, and have the local writer retry the full-size spill buffer there? Done in adb9daa02. `spill()` now calls `write_burst_complete` after `reservation.free()`, and `PartitionedSpill::restore_buffer` grows the reservation by the difference with `try_grow`, takes the writer back with `BufWriter::into_parts`, and wraps it in a full-size `BufWriter` that takes over the buffered bytes. It moves those bytes instead of flushing them: they always fit the larger empty buffer, so the swap writes nothing and cannot fail, which matters because `write_burst_complete` returns nothing. A refusal leaves the 8 KiB buffer in place and the next spill asks again. A spill file that already failed a write is not grown. The data file buffer is sized when the writer attaches, before the consumer holds anything, so the writer's own batches don't shrink it. The copy buffer is sized during the final write while the last batches are still charged and is not grown back; a range larger than it goes through `io::copy`. `pressure_spill_restores_the_spill_buffer` spills with no buffer limit in 1, 4 and 16 MiB pools and checks that the spill buffer is back to full size after every spill, that the reservation equals the bytes held, and that every row is written once. `shuffle_bench` on `lineitem` at the SF1 row count, 200 partitions, lz4, no buffer limit, `ci` profile, 7 rounds with the build order rotated, in a Linux container. Median shuffle write time: | Pool | main | before | after | |---|---|---|---| | 16 MiB | 0.128 s | 0.130 s | 0.130 s | | 32 MiB | 0.127 s | 0.146 s | 0.129 s | | 64 MiB | 0.126 s | 0.130 s | 0.120 s | On this data the 8 KiB buffer showed up at 32 MiB rather than 16. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
