[
https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Gupta reopened SPARK-21352:
-----------------------------------
There is no solution provided for the problem and neither stack overflow helping
> Memory Usage in Spark Streaming
> -------------------------------
>
> Key: SPARK-21352
> URL: https://issues.apache.org/jira/browse/SPARK-21352
> Project: Spark
> Issue Type: Improvement
> Components: DStreams, Spark Submit, YARN
> Affects Versions: 2.1.1
> Reporter: Shubham Gupta
> Labels: newbie
>
> I am trying to figure out the memory used by executors for a Spark Streaming
> job. For data I am using the rest endpoint for Spark AllExecutors and just
> summing up the metrics totalDuration * spark.executor.memory for every
> executor and then emitting the final sum as the memory usage.
> But this is coming out to be very small for application which ran whole day ,
> is something wrong with the logic.Also I am using dynamic allocation and
> executorIdleTimeout is 5 seconds.
> Also I am also assuming that if some executor was removed for due to idle
> timeout and then was allocated to some other task then its totalDuration will
> be increased by the amount of time took by the executor to execute this new
> task.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
