andygrove opened a new issue, #2651: URL: https://github.com/apache/datafusion-ballista/issues/2651
> Disclaimer: this issue was drafted by an LLM (Claude Code) at my request, against `b89a4e46`. Links and quotes were checked, but please push back if anything is wrong. The "In-memory shuffle" bullet in the shuffle design page ([shuffle.md#L263-L271](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/docs/source/contributors-guide/shuffle.md#L263-L271)) says: > This looks like the largest single win available: simply pointing `work_dir` at a RAM disk large enough to hold the whole shuffle already recovers a large share of the TPC-H gap against a pipelined engine, which says the cost being paid there is the filesystem round trip rather than the barrier itself ([#2318]). #2318 opens with the same measurement: about 40% faster TPC-H with a 4 GiB RAM disk. That measurement was retracted on #2308, 35 minutes after the doc merged ([comment](https://github.com/apache/datafusion-ballista/pull/2308#issuecomment-5308605130)). The queries had been failing after the first iteration, once the RAM disk filled up. The description of #2317 quotes the same RAM-disk numbers, because it was written before the retraction. The result that held up is #2317's own experiment, which kept sort-shuffle output in an in-memory store. It ran TPC-H SF10 on 2 executors with 4 vcores each: | in-memory store budget | median | | ---------------------- | ------ | | off | 22.12 s | | 1 GiB (default) | 22.54 s | | 12 GiB | 20.98 s | That is neutral at the default budget and about 5% faster when the store held almost everything. The passthrough writer, including every query's final stage, still wrote files in that experiment, so this doesn't settle the question either way. But it no longer supports "largest single win", or the conclusion that the filesystem round trip, rather than the barrier, is the cost. **Proposed change** - Rewrite the bullet without the RAM-disk claim. Keep in-memory shuffle as an open direction, with #2317's numbers as the current evidence. - Update the motivation in #2318 the same way. Suggested wording for the bullet: > - **In-memory shuffle.** Small intermediates are written and read back through the filesystem regardless of size. Keeping them in a bounded in-memory buffer, spilling to files only once the buffer is exceeded, would remove that write amplification for short queries while leaving the durable path in place for the ones that need it. How much it buys is still open. A prototype that kept sort-shuffle output in memory measured neutral at a 1 GiB budget and about 5% faster at 12 GiB on TPC-H SF10, but the passthrough writer, including every query's final stage, still went through files ([#2317], [#2318]). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
