Abhishek-Mohan opened a new pull request, #7332: URL: https://github.com/apache/datafusion-comet/pull/7332
## Which issue does this PR close? Closes #6032. ## Rationale for this change The memory tuning page explains overhead sizing, but still lacks the executor container sum and a way to distinguish container kills, heap exhaustion, and refused memory reservations. Readers can therefore increase the wrong setting after a memory failure. The issue's introduction and old pool/fraction concerns have already been addressed on main. This completes the remaining documentation gaps. ## What changes are included in this PR? - Add Spark's four-term container-sizing formula with its Kubernetes/YARN, off-heap, Python, and resource-profile qualifications. - Explain how off-heap size and overhead affect the container allocation and reservation budget differently. - Add a diagnostic table with symptoms and relevant tuning guidance, and replace the blanket exit-137 recommendation. Include YARN's JVM OOM handler, which can terminate an executor before Spark's exit-52 handler. - Link to the contributor memory-management guide. Only `docs/source/user-guide/latest/tuning/memory.md` changes. ## How are these changes tested? - Prettier 3.9.9 checking and `git diff --check` pass. - Direct Sphinx 7.4.7 builds of baseline and changed documentation succeed with isolated output and doctree directories. The changed build adds no warnings. Existing warnings concern missing generated/released documentation, existing references/lexing, and the unavailable Mermaid CLI; this is a targeted rendering check, not the Maven-backed publishing build. - Inspected generated HTML: the three diagnostic rows, new section anchor, YARN paragraph, and in-page/contributor links render and resolve correctly. - Checked sizing and error handling against Spark 4.1.3, current Comet code, and DataFusion 55.2.0's reservation-error wrapper. Documentation only; no runtime code changes or native/JVM test runs. ## Council review Used the t3dotgg/tstack council workflow. Round 1 identified the YARN OOM-handler exception to exit-code guidance; round 2 reviewed the correction. All three reviewers are satisfied. An initial interrupted dispatch did not complete a review round. Rounds: 2 | Delegate | Round 1 | Round 2 | | --- | --- | --- | | Claude Opus 5.5 | 8/10 | 9/10 | | Gemini 4 Argon | 10/10 | 10/10 | | GPT-6 Astra | 10/10 | 10/10 | AI assistance: GPT-6 Astra orchestrated investigation and validation in OpenCode/T3 Code; Claude Opus 5.5 completed the implementation, with source-checked corrections by Astra. The models above provided independent implementation reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
