comphead opened a new pull request, #7308: URL: https://github.com/apache/datafusion-comet/pull/7308
## Which issue does this PR close? None. This documents an existing setting that the user guide did not explain. ## Rationale for this change `spark.comet.shuffle.native.maxBufferBytes` is only described in the generated configuration reference and the contributor guide, so users whose tasks end in a native shuffle have no guidance on when to set it or what it costs. By default the writer buffers until the memory pool refuses it. In a wide group-by we ran (87 sort keys, 12 tasks per executor), that was about 2.6 GB per map task, and executors were killed for exceeding their containers. With the limit at `256m`, far fewer executors were killed, and a local reproduction of the same job showed no slowdown and a 3% larger shuffle. At `128m` and below, the stage reading the shuffle got slower. ## What changes are included in this PR? - A "Native Shuffle Writer Memory" section in the shuffle tuning page: what the setting does, why to set it, and what it costs. - The tuning guide's index mentions the new section. ## How are these changes tested? Documentation only. prettier, which CI runs on Markdown, makes no changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
