Hello,a good tool to manage the cluster usage for each mode is using Grafana
which use Prometheus in the background to collect metrics.
-Iacovos
Εστάλη από τη συσκευή μου Samsung
-------- Αρχικό μήνυμα --------
Από: "Jan Brabec (janbrabe)" <[email protected]>
Ημερομηνία: 24/9/18 08:57 (GMT+01:00)
Προς: Muhib Khan <[email protected]>, "Liu, Jialin" <[email protected]>
Κοιν.: [email protected]
Θέμα: Re: Question about Spark cluster memory usage monitoring
Hello,
Maybe Ganglia http://ganglia.info/ might be useful for you? I have only shallow
experience with it, but it might be what you are looking for.
Best,
Jan
From: Muhib Khan <[email protected]>
Date: Thursday 20 September 2018 at 23:39
To: "Liu, Jialin" <[email protected]>
Cc: "[email protected]" <[email protected]>
Subject: Re: Question about Spark cluster memory usage monitoring
Hello,
As far as I know, there is no API provided for tracking the execution memory of
a Spark Worker node. For tracking the execution memory you will probably need
to access the MemoryManager's onHeapExecutionMemoryPool and
offHeapExecutionMemoryPool
objects that track the memory allocated to tasks for execution memory and
write to a log the amount for parsing it later or send it to the driver in a
periodic interval to collect the memory usage in a central location. Either
way, I think tracking execution
memory may require code changes in Apache Spark. Hope this helps.
Regards,
Muhib
Ph.D. Student, FSU
On Thu, Sep 20, 2018 at 4:46 PM Liu, Jialin <[email protected]> wrote:
Hi there,
I am currently using Spark cluster to run jobs but I really need to collect the
history of actually memory usage(that’s execution memory + storage memory) of
the job in the whole cluster. I know we can get the storage memory usage
through either Spark UI Executor
page or SparkContext.getMemoryExecutorStatus() API, but I could not get the
real time execution memory usage.
Is there anyway I can collect total memory usage? Thank you so much!
Best,
Jialin Liu
---------------------------------------------------------------------
To unsubscribe e-mail:
[email protected]
