920032148 opened a new pull request, #60023:
URL: https://github.com/apache/spark/pull/60023
### What changes were proposed in this pull request?
This PR adds a small cgroup v1/v2 memory-limit reader and checks the
configured managed off-heap size when UnifiedMemoryManager is created.
If Runtime.maxMemory() plus spark.memory.offHeap.size exceeds the cgroup
memory limit, Spark logs a warning with suggested configuration changes. The
patch does not change memory admission, spilling, or eviction behavior.
This is a focused first step for the problem discussed in #57530.
### Why are the changes needed?
spark.memory.offHeap.size is an upper bound in Spark's memory manager, but
it is independent of the process cgroup limit. A configuration can therefore
allow the JVM heap and managed off-heap memory to exceed the container limit.
In that case the executor may be killed by the container before Spark can
spill, and the configuration problem is otherwise difficult to diagnose.
### Does this PR introduce _any_ user-facing change?
Yes. When managed off-heap memory is enabled and the configured JVM heap
plus managed off-heap size exceeds a bounded cgroup v1/v2 limit, Spark emits a
startup warning. No allocation behavior or configuration defaults change.
### How was this patch tested?
Added CgroupMemoryLimitSuite covering:
- bounded and unlimited cgroup values;
- cgroup v1 and v2 process paths;
- configurations that do and do not exceed the cgroup limit.
Ran:
build/mvn -pl core -am
-DwildcardSuites=org.apache.spark.memory.CgroupMemoryLimitSuite test
The Maven reactor completed successfully and all 3 tests in the new suite
passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
