Shubham Gupta created SPARK-21352:
-------------------------------------
Summary: Memory Usage in Spark Streaming
Key: SPARK-21352
URL: https://issues.apache.org/jira/browse/SPARK-21352
Project: Spark
Issue Type: Bug
Components: DStreams, Spark Submit, YARN
Affects Versions: 2.1.1
Reporter: Shubham Gupta
I am trying to figure out the memory used by executors for a Spark Streaming
job. For data I am using the rest endpoint for Spark AllExecutors and just
summing up the metrics totalDuration * spark.executor.memory for every executor
and then emitting the final sum as the memory usage.
But this is coming out to be very small for application which ran whole day ,
is something wrong with the logic.Also I am using dynamic allocation and
executorIdleTimeout is 5 seconds.
Also I am also assuming that if some executor was removed for due to idle
timeout and then was allocated to some other task then its totalDuration will
be increased by the amount of time took by the executor to execute this new
task.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
