I am not sure what column/properties you are referring to. But the
event log in Spark deals with application level "events', not JVM-level
metrics. To retrieve the JVM metrics, you need to use the REST API
provided in Spark. Please see
https://spark.apache.org/docs/latest/monitoring.html for details..
On 3/9/22 10:21 AM, Prasad Bhalerao wrote:
Hi,
I am trying to calculate CPU utilization of an Executor(JVM level CPU
usage) using Event log. Can someone please help me with this?
1) Which column/properties to select
2) the correct formula to derive cpu usage
Has anyone done anything similar to this?
We have many pipelines and those are using very huge EMR clusters. I
am trying to find out the cpu utilization and memory utilization of
the nodes. This will help me find out if the clusters are under
utilized and reduce the nodes,
Is there a better way to get these stats without changing the code?
Thanks,
Prasad
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
