[
https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181905#comment-16181905
]
Saisai Shao commented on SPARK-9103:
------------------------------------
Hi [~irashid], thanks a lot for your response.
I agree that your concern is very valid, especially on how to correlate the
whole memory usage to the task execution. But somehow it is hard to do from the
task level based on current Spark's design, in which some memory usage is
shared between tasks, like Netty memory, storage and execution memory. Also
about user memory, I think it is a missing part in the current Spark, but to
know this part of memory seems quite expensive, since we cannot expect what
will user do in the task, like memory used by 3rd party lib.
So let me think a bit on how to further extend this feature (though looks a
little difficult to do) :).
> Tracking spark's memory usage
> -----------------------------
>
> Key: SPARK-9103
> URL: https://issues.apache.org/jira/browse/SPARK-9103
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Reporter: Zhang, Liye
> Attachments: Tracking Spark Memory Usage - Phase 1.pdf
>
>
> Currently spark only provides little memory usage information (RDD cache on
> webUI) for the executors. User have no idea on what is the memory consumption
> when they are running spark applications with a lot of memory used in spark
> executors. Especially when they encounter the OOM, it’s really hard to know
> what is the cause of the problem. So it would be helpful to give out the
> detail memory consumption information for each part of spark, so that user
> can clearly have a picture of where the memory is exactly used.
> The memory usage info to expose should include but not limited to shuffle,
> cache, network, serializer, etc.
> User can optionally choose to open this functionality since this is mainly
> for debugging and tuning.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
