Yifeng Wang created SPARK-54449:
-----------------------------------
Summary: Spark UI display unexpected "Storage Memory" capacity
when spark.memory.offHeap.enabled is false
Key: SPARK-54449
URL: https://issues.apache.org/jira/browse/SPARK-54449
Project: Spark
Issue Type: Bug
Components: Block Manager
Affects Versions: 3.5.2
Environment: Tested with Spark v3.5.2.
*Steps to Reproduce:*
# Configure a Spark application with the following settings:
** {{spark.memory.offHeap.enabled=false}}
** {{spark.memory.offHeap.size=10g}} (or any non-zero value)
*Screenshots:*
1. Comment out configs
!image-2025-11-21-21-42-39-153.png!
Got: (Storage Memory =2 GiB)
!image-2025-11-21-21-43-07-597.png|width=289,height=273!
2. Enable configs
!image-2025-11-21-21-44-10-525.png|width=527,height=26!
Got: (Storage Memory =12 GiB)
!image-2025-11-21-21-44-43-226.png|width=220,height=246!
Reporter: Yifeng Wang
Dear Spark Community:
*Current Behavior:* In the Spark UI (Executors tab) and Spark History Server,
the "Storage Memory" column displays the total capacity as the sum of On-Heap
Storage Memory + Off-Heap Storage Memory. Even when the configuration
{{spark.memory.offHeap.enabled}} is explicitly set to {{{}false{}}}, the UI
still adds the value of {{spark.memory.offHeap.size}} to the total displayed
capacity.
*Expected Behavior:* Perhaps when {{spark.memory.offHeap.enabled}} is set to
{{{}false{}}}, the "Storage Memory" total in the UI should *only* reflect the
On-Heap Storage Memory? The {{spark.memory.offHeap.size}} configuration should
be ignored in the UI display calculation, similar to how it is ignored in YARN
resource allocation logic.
*Personal Understandings:* The issue stems from how {{BlockManager}} reports
memory to the {{{}BlockManagerMaster{}}}. {{UnifiedMemoryManager}} initializes
{{maxOffHeapMemory}} based on the configuration {{spark.memory.offHeap.size}}
regardless of the {{enabled}} flag. Then {{BlockManager}} reads this value via
{{memoryManager.maxOffHeapStorageMemory}} and passes it to the
{{registerBlockManager}} RPC call. {{At register}} method, the total memory is
calculated by simply adding on-heap and off-heap values without checking if
off-heap is enabled.
At ther parts of the codebase, such as {{Client.scala}} (YARN) and
{{{}ResourceProfile.scala{}}}, which correctly utilize
{{Utils.checkOffHeapEnabled}} to ensure off-heap memory is treated as 0 when
the feature is disabled.
*Proposed Fix:* Perhaps we need enforce the {{checkOffHeapEnabled}} logic
before aggregating the total memory for the UI?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
