[
https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Gupta updated SPARK-21352:
----------------------------------
Description:
I am trying to figure out the memory used by executors for a Spark Streaming
job. For data I am using the rest endpoint for Spark AllExecutors and just
summing up the metrics totalDuration * spark.executor.memory for every executor
and then emitting the final sum as the memory usage.
But this is coming out to be very small for application which ran whole day ,
is something wrong with the logic.Also I am using dynamic allocation and
executorIdleTimeout is 5 seconds.
Also I am also assuming that if some executor was removed for due to idle
timeout and then was allocated to some other task then its totalDuration will
be increased by the amount of time took by the executor to execute this new
task.
https://stackoverflow.com/questions/44995212/spark-streaming-memory-usage-doubts
was:
I am trying to figure out the memory used by executors for a Spark Streaming
job. For data I am using the rest endpoint for Spark AllExecutors and just
summing up the metrics totalDuration * spark.executor.memory for every executor
and then emitting the final sum as the memory usage.
But this is coming out to be very small for application which ran whole day ,
is something wrong with the logic.Also I am using dynamic allocation and
executorIdleTimeout is 5 seconds.
Also I am also assuming that if some executor was removed for due to idle
timeout and then was allocated to some other task then its totalDuration will
be increased by the amount of time took by the executor to execute this new
task.
> Memory Usage in Spark Streaming
> -------------------------------
>
> Key: SPARK-21352
> URL: https://issues.apache.org/jira/browse/SPARK-21352
> Project: Spark
> Issue Type: Improvement
> Components: DStreams, Spark Submit, YARN
> Affects Versions: 2.1.1
> Reporter: Shubham Gupta
> Labels: newbie
>
> I am trying to figure out the memory used by executors for a Spark Streaming
> job. For data I am using the rest endpoint for Spark AllExecutors and just
> summing up the metrics totalDuration * spark.executor.memory for every
> executor and then emitting the final sum as the memory usage.
> But this is coming out to be very small for application which ran whole day ,
> is something wrong with the logic.Also I am using dynamic allocation and
> executorIdleTimeout is 5 seconds.
> Also I am also assuming that if some executor was removed for due to idle
> timeout and then was allocated to some other task then its totalDuration will
> be increased by the amount of time took by the executor to execute this new
> task.
> https://stackoverflow.com/questions/44995212/spark-streaming-memory-usage-doubts
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
