[
https://issues.apache.org/jira/browse/SPARK-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320293#comment-16320293
]
assia ydroudj commented on SPARK-21157:
---------------------------------------
I m beginner in apache spark and have installed a prebuilt distribution of
apache spark with hadoop. I look to get the consumption or the usage of memory
while running the example PageRank implemented within spark. I have my cluster
standalone mode with 1 maser and 4 workers (Virtual machines)
I have tried external tools like ganglia and graphite but they give the memory
usage at resource or system level (more general) but what i need exactly is "to
track the behavior of the memory (Storage, execution) while running the
algorithm does it means, memory usage for a spark application-ID ". Is there
anyway to get it into text-file for further exploitation? Please help me on
this, Thanks
> Report Total Memory Used by Spark Executors
> -------------------------------------------
>
> Key: SPARK-21157
> URL: https://issues.apache.org/jira/browse/SPARK-21157
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 2.1.1
> Reporter: Jose Soltren
> Attachments: TotalMemoryReportingDesignDoc.pdf
>
>
> Building on some of the core ideas of SPARK-9103, this JIRA proposes tracking
> total memory used by Spark executors, and a means of broadcasting,
> aggregating, and reporting memory usage data in the Spark UI.
> Here, "total memory used" refers to memory usage that is visible outside of
> Spark, to an external observer such as YARN, Mesos, or the operating system.
> The goal of this enhancement is to give Spark users more information about
> how Spark clusters are using memory. Total memory will include non-Spark JVM
> memory and all off-heap memory.
> Please consult the attached design document for further details.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
