[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340449#comment-16340449 ]
Edwina Lu edited comment on SPARK-9103 at 1/26/18 2:08 AM: ----------------------------------------------------------- We (at LinkedIn) are interested in gathering more memory metrics as well. The pull requests for exposing netty and java.nio bufferedPool metrics via Metrics System have been merged. Are the changes for adding the metrics to the heartbeat and exposing it via the web UI still being worked on? From pull request 17762, it sounds like this may have been replaced by SPARK-21157. Having both total memory and netty memory information would be useful. SPARK-23206 proposes adding executor level metrics for JVM used memory, storage memory, and execution memory. It is also using the heartbeat to send executor metrics, and exposing the metrics via the web UI, and could share some of the same infrastructure. was (Author: elu): We (at LinkedIn) are interested in gathering more memory metrics as well. The pull requests for exposing netty and java.nio bufferedPool metrics via Metrics System have been merged. Are the changes for adding the metrics to the heartbeat and exposing it via the web UI still being worked on? From pull request 17762, it sounds like this may have been replaced by SPARK-21157. Having both total memory and netty memory information would be useful. [LIHADOOP-34243|https://jira01.corp.linkedin.com:8443/browse/LIHADOOP-34243] proposes adding executor level metrics for JVM used memory, storage memory, and execution memory. It is also using the heartbeat to send executor metrics, and exposing the metrics via the web UI, and could share some of the same infrastructure. > Tracking spark's memory usage > ----------------------------- > > Key: SPARK-9103 > URL: https://issues.apache.org/jira/browse/SPARK-9103 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI > Reporter: Zhang, Liye > Priority: Major > Attachments: Tracking Spark Memory Usage - Phase 1.pdf > > > Currently spark only provides little memory usage information (RDD cache on > webUI) for the executors. User have no idea on what is the memory consumption > when they are running spark applications with a lot of memory used in spark > executors. Especially when they encounter the OOM, it’s really hard to know > what is the cause of the problem. So it would be helpful to give out the > detail memory consumption information for each part of spark, so that user > can clearly have a picture of where the memory is exactly used. > The memory usage info to expose should include but not limited to shuffle, > cache, network, serializer, etc. > User can optionally choose to open this functionality since this is mainly > for debugging and tuning. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org