[
https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143425#comment-14143425
]
Josh Rosen commented on SPARK-2321:
-----------------------------------
{quote}
... maybe we should redesign the SparkListener event API, and add job id info
into Stage/Task event in Scheduler before post it to listener bus.
{quote}
A stage may be used by multiple jobs, so we'd have to think carefully about how
the API should reflect this. It looks like DAGScheduler's internal {{Stage}}
class tracks the id of the job that first submitted the stage, and
{{activeJobForStage}} finds "the earliest-created active job that needs the
stage." It might make sense to associate Stage/Task start events with the list
of active jobs that depend on them.
> Design a proper progress reporting & event listener API
> -------------------------------------------------------
>
> Key: SPARK-2321
> URL: https://issues.apache.org/jira/browse/SPARK-2321
> Project: Spark
> Issue Type: Improvement
> Components: Java API, Spark Core
> Affects Versions: 1.0.0
> Reporter: Reynold Xin
> Assignee: Josh Rosen
> Priority: Critical
>
> This is a ticket to track progress on redesigning the SparkListener and
> JobProgressListener API.
> There are multiple problems with the current design, including:
> 0. I'm not sure if the API is usable in Java (there are at least some enums
> we used in Scala and a bunch of case classes that might complicate things).
> 1. The whole API is marked as DeveloperApi, because we haven't paid a lot of
> attention to it yet. Something as important as progress reporting deserves a
> more stable API.
> 2. There is no easy way to connect jobs with stages. Similarly, there is no
> easy way to connect job groups with jobs / stages.
> 3. JobProgressListener itself has no encapsulation at all. States can be
> arbitrarily mutated by external programs. Variable names are sort of randomly
> decided and inconsistent.
> We should just revisit these and propose a new, concrete design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
