[
https://issues.apache.org/jira/browse/SPARK-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269738#comment-15269738
]
Alex Bozarth commented on SPARK-13269:
--
Hey [~andrewor14], I was interested in this and took a look at the two examples
you gave and am a bit confused at what exactly you actually want. You can
currently see the used memory, used disk space, and active task count for each
executor by calling /applications/[app-id]/executors or (in code) getting the
ExecutorSummary class for each executor and checking activeTask, memoryUsed,
and diskUsed. Are these numbers different from what you were interested in
surfacing? I was unsure of how those example related to JobProgressListener as
well.
> Expose more executor stats in stable status API
> ---
>
> Key: SPARK-13269
> URL: https://issues.apache.org/jira/browse/SPARK-13269
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Reporter: Andrew Or
>
> Currently the stable status API is quite limited; it exposes only a small
> subset of the things exposed by JobProgressListener. It is useful for very
> high level querying but falls short when the developer wants to build an
> application on top of Spark with more integration.
> In this issue I propose that we expose at least two things:
> - Which executors are running tasks, and
> - Which executors cached how much in memory and on disk
> The goal is not to expose exactly these two things, but to expose something
> that would allow the developer to learn about them. These concepts are very
> much fundamental in Spark's design so there's almost no chance that they will
> go away in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
