[
https://issues.apache.org/jira/browse/SPARK-36473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angerszhu updated SPARK-36473:
------------------------------
Description:
Spark split memory to managed memory (storage + execution) and un-managed
memory. And use `spark.memory.fraction` to adjust the fraction of this to
memory.
But we have no metrics to know how much we used for unmanaged memory and we
have no data evidence to optimize configuration of `spark.memory.fraction`.
Maybe we should add it
> Add non-unified memory metrics
> ------------------------------
>
> Key: SPARK-36473
> URL: https://issues.apache.org/jira/browse/SPARK-36473
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: angerszhu
> Priority: Major
>
> Spark split memory to managed memory (storage + execution) and un-managed
> memory. And use `spark.memory.fraction` to adjust the fraction of this to
> memory.
> But we have no metrics to know how much we used for unmanaged memory and we
> have no data evidence to optimize configuration of `spark.memory.fraction`.
> Maybe we should add it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
